Li-Ion Battery Undervoltage Protection
I would go with the second approach, as you don't want to cut off for momentary spikes in current. However, towards the end of discharge, Li-ion battery tends to drop the voltage quickly. So you cannot wait for long, a delay of 2 seconds should be good enough.

The threshold voltage will depend on the chemistry and the undervoltage protection built into the battery. 

The battery terminal voltage is a function of current as well. If the current is higher, there will be higher drop in battery voltage. If you are expecting no load on the battery for some time, you can consider 2 different threshold voltages, one for no load and one for full load.