Are the particulates that settle in a veal stock impurities?
Yes, they are most likely to be denatured protein. It is feasible that they also contain some other matter like stray pieces of vegetable, but the bulk should be the protein.

When the  particles are small enough, they are agitated in the boiling water and become a suspension. When you let them rest, they slowly find their way to the bottom, due to gravity. 

So, the answer to your other questions is yes and yes. Are you approaching consomme when you filter them out? Yes, you do. Are you removing flavor? Also yes. After all, it's not the water that flavors the stock, it's what remains in it from the veal. While the pure protein doesn't have much flavor by itself, enough stuff tends to stick to it to make it flavorful. 

Consomme is made clear for presentation purposes and for texture, not for flavor. The loss is slight, but it is a loss, not a gain. It is up to you to decide which stock you prefer, the filtered or unfiltered version, there is no single "best" answer.