Is the ocean guaranteed to warm in a warming climate, regardless of whether it absorbs LW radiation?
My naive understanding was that our increasing the magnitude of the
  greenhouse effect meant that there was less radiation escapting to
  space


That's 100% correct.   If we assume constant solar input, there's only two ways the Earth's surface temperature can change.   Albedo and Thermal radiation.   Greenhouse gas driven climate change means exactly what you said - less heat radiates off Earth into space.    Most of this heat goes into the Ocean.


  And if it is, does it not imply that the ocean must also get warmer,
  simply because it is a thermally coupled part of the entire warming
  system? I had someone tell me that the a warming atmosphere could not
  warm the ocean beyond an infinitesimal amount, because the atmosphere
  has a much lower heat capacity.


Your friend is half right, and half wrong.   Air has a bit over 1/4th the heat capacity compared to water and it's about 800 times less dense, so he's correct, but it's not that simple.

Sunlight - somewhat counter-intuitively, isn't great at warming oceans because the photons from sunlight are energetic enough to evaporate water molecules into gas molecules.  Oceans have low albedo which means they absorb most of the energy from the sunlight, but much of that heat is lost in evaporation by visible light photons.   While that has nothing to do with your question, it's worth pointing out that sunlight isn't as good at warming oceans as one might think.   (If anyone has one of those solar mirror ovens, I'd be curious to see how well they work on pure water . . . just out of curiosity, evaporation loss vs rate of warming).

The back-radiation from the atmosphere is comparatively much less total solar energy, but oceans are good at absorbing and storing thermal back-radiation reflected back off the greenhouse gas rich atmosphere into the ocean.   This is a tiny amount of the total heat Earth gets from sunlight, and the increase of this radiation due to greenhouse gas is a fraction of one percent of solar energy, but it adds up.  

One way to explain this is that 85 degree air will warm 80 degree water. That's a thermodynamic law.  It just takes a while and because the heat capacity and density of water is much greater, it takes about 4 liters of air to give 1 degree back to warm 1 cc of water 1 degree.  But despite the inefficiency, warmer air still transfers heat into colder water.  It takes many decades, perhaps centuries, for the oceans to catch up to the warming air, but air, however inefficiently, does warm the oceans.

The quirk in this, is that the oceans, despite warming much more slowly than the air, are still absorbing over 90% of the trapped heat added by the increase in greenhouse gas.   It's a matter of scale.  Oceans warm slowly because they're enormous and always circulating and it takes much more energy to war them, but they also absorb most of the heat, for the same reason, they're very good at holding heat.  Slow to heat up, but also, slow to cool down.  That's why bodies of water often feel warm when you go swimming at night.

I should probably add something about increased evaporation in higher air temperature, which effectively cools the oceans and surface air, but also increases the greenhouse effect with an increase in water vapor, but running those numbers is a bit over my pay-grade.   The greater efect is the one mentioned above.