Truing an 'out of round' wheel
Whenever I get stuck wheelbuilding, I loosen every spoke one turn, walk away and come back in a few hours or the next day. 

The worst thing you can do is keep doing something that is not providing the results you expect. It is completely possible to overtighten spokes enough that they stretch significantly under the tension. You can even break them with a bow and arrow like result that can be slightly terrifying. ( Fortunately the spoke was pointed at the ceiling and not me when that happened. )

Part of wheelbuilding is getting a feel for how much tension the spokes will support. There is a point at which tightening them further just does no good and actually makes the wheel worse, not better. In the past when rims where just boxes made as light as possible and spokes were more burly, you'd get wonky things happening to the true to let you know when you'd gone too far. Now most rims have enough structual shape that they won't deform that way and the spokes are much lighter. 

In my experience out of round truing almost always requires adjusting all 4 quarters of the wheel. ( i.e. the trough as well as the bulge. ) You can't fix
it by just working on one section. 

If tightening stops working, it's time to take some tension out of the entire build. Start by loosening the spokes you've been working on a half turn at least and then start loosening other spokes to achieve the affect you want. If that still doesn't work, loosen all the spokes and start over.