Sewable LED Ribbon - 1m, 50 LEDs (Blue)
Yes, you can. If you look at the sparkfun video on that page, they use 2 cr2032 3v batteries directly.. That's what they intended. 

They also show the datasheet, where the strip is just leds in parallel and the wire. 

It works without a resistor because the wire is thin, the current is low, and the voltage is close to what the leds want. Additionally, coin cells have a high Equivalent series resistance, so they act like a resistor. The higher the load, the more they resist so the lower the voltage and current goes through it. I would not connect this to a 5v usb power bank without current limiting. The 4.5V is really the absolute maximum voltage, not the intended actual voltage.

That said, coin cell batteries are not ideal. They have a low capacity and the capacity depends on how hard you drive the load. They won't last too long before the leds are very dim. Less than a day, maybe a few hours. The more leds you use, the worse the problem gets.