Sounds during meditation
Distraction one of the two main obstacles in meditation (the other is dozing off or even falling asleep). Many people get distracted by sound. 

It's normal and as long as you keep getting distracted you have two choices: 
1) keep sitting there, and keep bringing your attention back to your meditation object
2) quit meditating. 
3) change your meditation. (headphones, change in meditation object etc. see below)

There is no expected time limit within which you are supposed to stop being distracted. Distraction happens. 

From the way you describe it - you don't just note the distraction and then get back to meditating. You are also getting annoyed at yourself. So note that too. 

Patience. Patience. Patience. Remember that this is all part of the course &amp; that as you are not living on a mountain somewhere, your conditions are not ideal. So be patient with yourself. I'll repeat myself again: be patient with yourself.

One alternative is that when you don't meditate with plugs (which I do think an excellent idea, btw), is to meditate on sound instead. That way the distraction has become the object. This may change the meditation dynamic. 

Good luck. And remember: the fact that you are still regularly sitting there is in itself an achievement. Don't be too hard on yourself. And perhaps read something by Pema Chodron? She's very good at reminding us of being kind to ourselves.