What does the COM do on a DC/DC converter?
I got -VIN at (-5V), +VIN at +5V. 




Figure 1. DC-DC converter wiring.

Ensure that you have only 5 V between the input terminals. Your post makes it sound as though you have +/- 5 V (= 10 V) between the terminals and this would explain the heat.


  What's COM and where do I connect it?


COM is the common or mid-point on the output. Connect it to the GND or 0 V of your isolated circuit.


  My "RGZ" model seems to be outputting -50/50 V and getting toasty.




Figure 2. The output voltage regulation.

The open-circuit output voltage could be up to 25% higher than the nominal. If you are getting double then it is probably due to incorrect input voltage as explained above.

Decoupling capacitors:

Don't try omitting the decoupling capacitors. They will be required.




  Whats a good choice for the decoupling caps?


The datasheet doesn't say other than the maximum capacitive load table on page 1. 


I'd reckon that you stabilise the input with 100 - 470 µF in parallel with 100n. 
On the output I would aim for half the max capacitive load including any capacitors across the supply rails in the device you are powering. 
The 100 nF capacitors are better at handling high-frequency noise than the electrolytics are.