Are there any major drawbacks to using a track frame as a frame for a fixed-gear or single-speed commuter?
Real track bikes don't have brakes. You need brakes for commuting, especially in a city, and at least 2 independent braking systems. Not all frames and forks have brake mounts. You also won't have fender or rack mounts which are nice in wet weather. 

Ergonomics and geometry can be an issue, given that track bikes tend to be a bit twitchy and aggressively positioned.

A lot of people use them in an urban environment, probably because of fashion and some sort of bike messenger fantasy.