Smallest Primary Cell
You might consider using hearing aid batteries. Size 10 and size 312 may meet your requirements:



Generally hearing aid batteries hover around 1.4V, so depending on your application you may need to stack them. Not sure of the effect stacking them has on long-term reliability though.