Undervoltage from solar panels
You have selected a number of things that don't go well together. It is possible to use relays or mosfets to switch between batteries, and the Arduino could be used to measure a voltage. Or you can make a circuit with a few electronic components (e.g. a comparator). However, no one does it like that. It would be unreliable.

Every solar power charging module can handle darkness as well. Some modules will also create 5V.

This Adafruit module has no 5V output, it only contains the charging part: Adafruit:USB / DC / Solar Lithium Ion/Polymer charger - v2
It is out of stock at the moment, and you need solar cells, battery, a cable, and a DC-DC converter to make 5V.
When it is dark and the battery is empty, I assume the output drops and the Arduino stops working. Adafruit made a tutorial for it that shows the DC-DC converter and also Collin shows a DC-DC converter: Youtube, Collin's Lab: Solar

There are other modules like that. For example the Sparkfun Sunny Buddy.

Take care that the batteries don't get too hot.
It is possible to run some Arduino boards directly from a battery (3.1 to 4.2V). Then you would not need the DC-DC converter.
Buy solar cells that are the best in low light conditions. With enough sunlight, the power is no problem. It is the dark winter days that could be a problem.