What does ATC do when a VFR flight in the mode C veil has a malfunctioning altitude encoder or transponder?
A lot of this has to do with whether or not you're talking to ATC.

If you're talking to ATC and your Mode C disappears or goes insane they'll ask you to report your altitude (and if your Mode C altitude reporting is wildly off they'll probably tell you something like "Stop altitude squawk - altitude differs by 500 feet". This is your cue to take the plane to an avionics shop and have the system checked/serviced, and not to attempt to enter B or C airspace, or fly above 10,000MSL, until its fixed.

In most cases ATC will continue to work you in the system until you are clear of the Mode C veil (possibly expediting your exit if having no altitude readout on your target is messing up their day) because it's better to be talking to you than not.



If you're not talking to ATC things get a bit more complex:
If your Mode C altitude readout is missing ATC will generally report you as "traffic, altitude unknown" to other aircraft and apply lateral separation exclusively (nobody is allowed within the minimum lateral separation distance of your target unless they have you in sight and can maintain visual separation). If/When other pilots spot you ATC will usually ask them for your type and altitude so they have some idea of who you are and what you're doing.

If your encoder or transponder is malfunctioning and reporting an incorrect altitude that's substantially more dangerous: ATC has no way of verifying your altitude other than asking you (or another pilot), so they'll be maintaining separation based on an incorrect altitude. This is why even VFR aircraft equipped with Mode C need an encoder check.
In this case ATC will find out about a malfunction when you're called as traffic to someone and they respond with "Is that my traffic above/below me?"