Is it normal for a lithium ion battery to read with a very low voltage on a multimeter?
Almost any charging circuit for Lithium batteries has a low voltage protection built in. This will protect against charging dead batteries (which might overheat) and this also inhibits enabling the charger when no battery is present.

I have been faced with such a situation myself several times: the voltage of the battery (NiMH, LiIon, etc) was too low for the charger to work.  I have succeeded in most cases to jump-start the system - in some cases the battery was really dead.

What I do is to charge the battery using a current limited voltage supply.  I limit the current to less than 1/10th of the rated capacity of the battery (about 200mA in your case), and I charge about 10 minutes and monitor the voltage using a voltmeter. If the voltage is not high enough after 10 minutes, I'll continue to charge as long as the measured voltage at rest is increasing.
I'll set the voltage to something low to start (1V) and limit the current as indicated.

Once the voltage is high enough, I put it back in the original system and I check that it is charging. If not, I may continue to charge it outside the system until the voltage is higher. For Lithium batteries, 2.5V should be sufficient for the original charger to be happy (which is of course a guess), usually it is lower.

I also have a charger which will charge any type of battery according to the charger settings. This allows me to charge the battery in a controlled manner. It also has a low battery safety protection so I still need to pre-charge the empty battery before.  But it starts charging at a pretty low voltage and it clearly reports the charging error on its LCD which is convenient. Such a charger allows you to charge a battery completely before putting it back into the original system.

As an example with some "side effects" that show that you must be patient and/or measure:

With a Samsung Galaxy Tab3 that I fixed this way, I noticed that it required:


The "official charger" (AC to USB converter) [Not sure, but I had this impression];
Some extra time before it showed that it was charging.
I initially thought that it did not work, but the voltage it was presenting to the battery was higher than at rest. It just required extra charge before the screen lit up - charge that the internal charger was able to build up.


WARNINGS

You must monitor the charging while you are doing this. If the battery heats up or is building up gas internally: stop trying to recover it. If there is a risk that you forget monitoring it: use a timed wall plug, set multiple alarms (phone, PC, ...) to make sure that you check on your system.
When possible I put the battery in a box and I put a transparent lid on it.  I did not have any problem yet, but in case it leaks or "explodes", there is at least some protection.  If it does leak for any reason -protect yourself from the chemicals and dispose of the battery in a controlled manner.

DO NOT DO THIS WITH BATTERIES THAT ARE OBVIOUSLY IN BAD SHAPE (leaking, built-up gas, damaged)

I am talking about small batteries - that is where I have experience.

If the battery voltage is not increasing rapidly, the battery is dead.  Which means it should go from close to 0V to 0.5/0.8V in just a few minutes (measured when it is not charging).

I recommend not doing this if you do not know what current limiting is or if you do not have any means to achieve and verify the actual current limit

And of course, this is entirely at your risk!  Be carefull and stop when in doubt - the battery or anything else should not heat, leak or smoke.  The slightest sign of that: STOP the process!