How can I best combine different objects in a way that reduces vertices and edges?
The Boolean modifier should work perfectly for this purpose.

First, split the separate parts into individual objects (P with everything selected in edit mode).

Then, add the Boolean modifier to either of the objects, set it to Union, and select the other object. You can further edit the mesh by applying the modifier (⎈ CtrlA).


After applying the modifier, you will be left with two objects: the joined mesh and the target object of the modifier.