How do airplanes maintain separation over large bodies of water?
To expand on DeltaLima's answer:
They use routes called the NATs (North Atlantic Tracks), which can be different each day depending on things like weather and wind speeds. Each track is a set of coordinates. A pilot will do a position report at each coordinate or every 10 degrees of longitude. The aircraft has to get special clearance to fly oceanic about 30 minutes prior to crossing the entry waypoint for the NAT. Each NAT is 60 NM apart.
To clarify a bit more, HF radio is used to communicate, since it has a longer range than VHF, but is much noisier. This is where systems like SELCAL come in, but I won't talk about that here. They can also do position reports over the radio or over text (CPDLC/ACARS), taking advantage of ADS B/C, which I also won't talk about here. These communication systems are pretty complicated, so I recommend you search it on the web or look at the sites I linked below.
Aircraft are required to be 1,000 feet apart vertically. Eastward flying aircraft have odd altitudes such as 37,000, 39,000, 41,000, etc, while westerly flights will be the opposite. They are also required to be about 10 minutes apart from each other.
Gander Radio (notice not &quot;radar&quot;!) controls much of the North Atlantic airspace from Gander, Newfoundland, Canada. Shanwick is another major Radio station. New York controls further south, along with Santa Maria having relatively small airspace.
There is a similar system over the Pacific ocean, with stations in Anchorage, Alaska, and Oakland.
Links:
I highly recommend you read this article about North Atlantic Tracks, and this one too.
You should at least watch this video, it's only 3 minutes, and it's summarized: 
                
            
More about ADS-B
More about SELCAL
More about position reports
If you want to learn more about ACARS: