How did Homura able to return to a past where Madoka isn't already a Magical Girl?
This is because the date in which Homura returns to is actually before Madoka contracts with Kyubey

Looking at the Timelines Wiki page we see that it's estimated that Madoka makes the contract a couple of days after Homura would wake up in the hospital.

while the wiki page says that the dates shouldn't be taken as canon we can make the assumption that this is close to being correct as we see in the Hospital on Homura's Calander. 



according to the wiki


  The calendar we see in Homura's hospital room reads "leaving hospital" on Wednesday the 16th and "going to school" on Friday the 25th. March has 31 days with the 25th falling on a Friday in 2011, and is the only such month in 2011.


In the 4th Timeline shown in Episode 10 (the timeline after the one Homura shoots Madoka's soul gem at her request) we see Homura appear in Madoka's window warning her while holding a dead Incubator. 

at the end of the 3rd Timeline we also get an indication of this in what Madoka asks of Homura


  Madoka: Sorry, I lied. I guess I still have one left.
  Homura: But why?! You should've used it!
  Madoka: It's better this way. I want you to do something that I couldn't do. You can go back in time, right, Homura? You can go back and change everything... so that we don't end up like this...
  Homura: Okay...? 
  Madoka: Then save me from being stupid, from getting tricked. Don't let Kyubey fool me again.
  Homura: I swear I'll save you! I'll do whatever it takes to keep you safe! I'll come back again and again and again! I'll save you! I swear!


this would indicate that at some point in the third Timeline Homura had told the other girls (or at least Madoka) that she could go back in time, not just freeze it like they would have seen. Madoka also seems to imply that she knows Homura can travel back to a point in time before she made the contract which i would see coming up in the conversation.

Since Homura's return point in the past is always the same, this would indicate that between Homura's release from hospital (when she wakes up) and when she'll transfer to school is when Madoka makes the contract as such in the 4th Timeline and beyond that Homura would intercept any Incubator who would try to trick Madoka