An F grade (in a labor studies class) on the transcript will hurt my admission to mathematics grad school?
This could possibly hurt you. 

In grad school, and even more so as a working academic, you will be asked to do various tasks that you might disagree with. For example you might be asked to teach precalculus or college algebra. Occasionally grad students don't like how they are asked to teach these courses and they either raise a big stink, or else simply don't follow instructions. In either case, this can cause huge headaches for the person supervising the TA -- even if the TA's complaints have some merit. Whether fairly or not, the prospect that you may be unwilling to follow instructions might give some admissions committees pause.

Doing tasks which you disagree with and that were assigned by people whom you may or may not respect (and doing them quickly, so that you can get on to the work you actually care about) is a vital survival skill in academia. I recommend that you learn this skill immediately and do what you can to pass the course.