How far up do you have to go before most of the atmosphere is below you?
Roughly 5.5 km, although the exact value depends on the weather.  All you need to do is to solve the hydrostatic equation and find the height at which the pressure is 50% of the height at sea level (or perhaps average elevation of Earth surface instead of sea level).  Any answer actually performing the calculation deserves upvotes more than I do ;-)

See, for example, Thickness of Earth's Atmosphere:


  Using boundary layer theory as an example, we can define the thickness of the atmosphere to be the altitude that encloses 99 percent of the total mass of the atmosphere. Looking at the chart, we can see that this seems to be about 31 kilometers. The halfway point, where half the mass of the atmosphere is below and half above occurs at 5.5 kilometers. Another interesting fact is that when you are cruising in a modern jet transport at 11 kilometers, you are above 77.5 percent of the atmosphere. The total mass of the atmosphere turns out to be 5.3 zettagrams (5.3 Zg). 


I'm not entirely sure how the calculations in the linked article are performed, but I recall calculating this as a student and ending up somewhere around 5 km, so it sounds right.  Simple calculations often assume an isothermal atmosphere, which is of course not accurate, but good enough if you don't need to know the answer more precisely than within several hundred metre.

This page at Stanford reaches the same number.