Ok, people often forget that you work to live and you do not live for work. Your family is the most important thing. You do not go to work to be "fair" to your coworkers. What this man is doing is providing a hostile work environment. You made an arrangement with the management for special circumstances and this has nothing to do with your coworkers. Report this man's behavior to your management, they must then handle his behavior. Document the fact that he is trying to override the arrangement that was approved by your supervisor. At that point if your coworker tries to retaliate then the company, government position or not, is liable.