Negative marks for mistakes
I've seen such grading used for true-false and multiple-choice questions in an exam, where a right answer gained you points and a wrong answer cost you points, but I've never heard of a problem worth "nothing" that cost you points if you got it wrong.

Personally, I dislike such systems—I'd rather give credit than subtract for mistakes—but instructors often have broad latitude in the design and scoring of exams. Some schools do have rules with regards to specific situations (if I remember, there were some weird regulations regarding all multiple-choice exams at my old institution), but in the absence of those, professors are free to set grading criteria as they choose, so long as they are applied consistently.