Layering (lower half) for long chilly rides
For the kind of riding you describe, the best choice are windproof tights. You should look for the kind that has windproof membrane in the front, breathable fabric in the back and most importantly no padding. Underneath them you wear shorts or bib tights with chamois. 

These are available from many cycling-specific brands, but the ones intended for nordic skiing work too.

If the membrane gets too warm, the next thing are winter running tights made of brushed lycra. Running-specific because they come without padding.