Is it okay to connect 2 lithium ion battery in parallel to a charger that can maintenance 4.2V? (more detail in post)
The Voltage you want to measure is the open circuit voltage (OCV), which is the voltage of the battery, when no current is flowing trough it. So you would need to disconnect the charger to measure the exact OCV. 

The OCV is then related to the state of charge (SOC) according to the graph in the datasheet.

Charging the cells in parallel is an easy method to balance the cells, but as ddurgin said,  your current limit influences the charge time. If your charger is set to charge at 1A and you charge 4 cells in parallel, this will result in .25A per cell (on average), so it would take four times as long.