Should a student be penalized for using a theorem outside of the curriculum?
This depends on what is announced in the syllabus and on the kind of test. 

If the syllabus explicitly says that tests should be solved using the course material only, then, yes, any answer using anything else than course material is not totally correct and should get deductions. If the syllabus does not say so, the answer is not so clear anymore.

If the test is kind of a final exam on the whole course where the students are tested on the whole subject, than I (and this is a personal opinion) would not deduct anything if the proof is technically correct, whatever tools have been used (unless the problem says "solve this with this method"). However, in an intermediate test it may well be that the instructor wants the students to show that a certain technique can be applied or a certain concept can be used. Going beyond the course material spoils this idea and (again, personal opinion) one may deduct points, although it would be much better if these policy has been stated beforehand.

The above should answer the question "Why could points be deducted?" but not the "Should points be deducted?". I think the answer to the latter question is really opinion based (and may be a question for matheducators.stackexchange.com).

As a side remark on "How would someone be able to use a theorem correctly if he doesn't know its proof?" I (and I guess most other working mathematicians) do that all the time. I would not get anywhere without using theorems of which I haven't even dare to read to proof. Some examples from outside my field: People use Fermat's last theorem, the "Kepler conjecture" (also a theorem now), the four-color theorem, the "10,000 pages theorem" on the classification of finite simple groups, the Poincare conjecture (proven as special case of the geometrization theorem)…