USB lights up simple circuit but not coin-cell
Coin cell batteries generally have a voltage of 3V.  Your Gemma takes in 5V.  Even though your LEDs are powering on, your Gemma doesn't have enough voltage to actually run.  

The voltage regulator doesn't take whatever input you give it and make the proper output, it needs either a lower OR a higher voltage.  In this case, and this is the most often case, you'll need a higher voltage than what your voltage regulator will output

I would caution you regarding putting 2 CR2032 cells together because this will result in 6V.  If the Gemma has a regulator, then that will be safe but your leds might not have the proper resistor for a 6V source.