Is mixing coffee with milk directly unhealthy?
Not really, but there haven't been that many studies in to the issue. An abstract of the most significant one can be found here.

To summarise, the study tested people who had drunk black coffee, coffee with milk, and coffee with non-dairy creamer to see how much of the phenolic acid (one constituent of coffee that is supposed to have health benefits) entered the bloodstream in each case.

The study found that adding milk had no effect, but adding creamer affected the maximum concentration of the phenols in blood plasma, and the time it took to achieve that concentration.

This kind of toing and froing over the health of certain foods is rife today. One minute coffee is bad for you, the next it's good for you, and so on. For example, the study above assumes that the phenolic acids involved, being antioxidants, are good for you because they reduce free radicals in the body. 

Free radicals in too high a concentration can indeed be harmful. However, they are also an essential part of the immune system - when a harmful bacteria is found in the body, phagocyte cells will surround it and destroy it with free radicals, for example. Things like this mean that there is rarely a simple answer to questions of food health.

Likewise, it has been suggested that people with a certain set of genes may be adversely affected by drinking too much caffeine, which may cause bone loss, unless they also have a source of calcium, like milk, in their diet as well, which would rather suggest that milk is definitely a good thing to have with your coffee!