What is the strange object on 737-800 exterior, on the aft right side?
It's the air inlet for the Auxiliary Power Unit (APU).

Slide 18 here has an image labelled "APU Air Inlet door with Vortex Generator". The inset image on the slide looks almost exactly like your posted image.

I found a much better image here, captioned "Boeing 737-800 with APU inlet".

There is some discussion about why the 737 has vortex generators for the APU on this thread on PPRuNe.