What is the typical RMS noise of a lithium battery?
Apparently you are asking about "ECN", Electrochemical Noise, which is the source of RMS noise from Li-Ion primary cells, and all other chemistries. Quick Google search and some browsing of search results leads to the following scholarly article, "A Method for Voltage Noise Measurement and Its Application to Primary Batteries". 

They did several measurements, including CR2032 cells. The resulting noise is very small and needs special precautions to even measure it, a Faraday cage is a must, and uV-level special instrumentation was used to capture data. Here is one of their results, a typical waveform of noise, and its power spectrum:



By my eye-ball estimation, the RMS of noise of unloaded CR2032 cell is about 2-3 uV in 10^-2 to 5Hz range.