How can I fix slight imperfections that mess up the shadows?
You can make the surface planar by making a custom transform orientation from a selected face that is flat on the right surface plane pressing Ctrl+Alt+Space, then you can select linked flat faces with Ctrl+Alt+Shift+F ant adjust the Sharpness property in the Redo Last panel so that it includes the out of place faces making sure the first(active) face you selected is on the right plane and then if you set the Active Element as Pivot Center you can scale all the selected faces to 0 in the custom transform orientation's z axis. This will make the surface planar to the selected face.



You could also just use Normal transform orientation and it would scale everything according the active face's normal just fine, but I like setting it to custom because it then works in all situations for example even if I need to make the plane at a floating single vertex.