Can I drill into a wall's baseboard?
Perfectly fine to drill through a baseboard.

I would drill a pilot hole from the side that still has the shorter-baseboard. That gets you lined up without worrying about splitting the new baseboard or pushing it away from the wall. Then drill the full-size hole from the taller-baseboard side, centered on the pilot hole.