Is it advisable to drop out of a PhD and look for a new one?
[I] am currently a over a year into a PhD in maths. However...I told my supervisor right from the beginning that my interests were in physics (and very specific areas of physics), yet I have spent a year without doing any physics.


Seems like a rather obvious mismatch! Why are you in math if you want to do physics? 


  My supervisor has no clue how to solve the problem and suggestions he gives are never useful.


Another bad sign. Sometimes it's worth accepting a less-interesting area in exchange for an awesome advisor. But you don't seem to like your advisor either.


  All of these things make me...want to drop out and start again somewhere else...I don't want to waste 4 years of my life simply out of a desire to "see it through." 


Makes perfect sense. Also consider the quality of your work...once you have a PhD, people will care about your publications and letters -- slogging through a PhD you don't like is unlikely to produce nice publications or letters of recommendation.


  If I drop out and apply to another PhD program is the fact that you've already dropped out once damning?


Nope, changing departments is a reasonable reason to drop out. But, it would be nice if you could declare victory -- "I was a brilliant math student who got awesome grades and interesting results, but I realized I like physics better" sounds better than "I got my a** kicked at math and physics sounds easier." Consider what you can do over the next few months to salvage your year (publishing, securing letters of recommendation, getting good grades, etc.)