How can I clean up Boolean operations?
I would do it this way:


First apply the Scale on your object because if you've scaled your object in Object mode the bevels won't give anything good.
Only keep the front face.
Use the knife to create quads all over your plane: K to activate, Z to cut through, C to cut perpendicularly.
Make sure you don't have vertices very close to each other, if so use merge, remove doubles.
Extrude.
Give your object a Subsurf modifier and smooth it.
Bevel the drawings edges.
Bevel the border edges.
Cut an edge loop with the knife along the background face (or mirror the front face so that you don't have to work the back).
Clean all the corners.