Most compact way to power 5V Arduino boards?
I made a torch locator which looks like this:



That was powered by a single button cell. The actual processor is an ATtiny85. As you can see it isn't much bigger or heavier than the coin cell.

You can run the Atmega328P (and the ATtiny) at 3V (as from that battery) provided you slow them down to 8 MHz. That is quite easy to achieve, and then you don't need an external crystal as the Atmega328P has an inbuilt oscillator which can drive it at that frequency (as does the ATtiny).

I have a page about power saving which might help you with your design.

A possible alternative is a single LiPo cell (nominally 3.7V) which, if you get a small one, should be quite light.