Should the neutral and earth bars be connected in this situation?
Better idea -- move that 50A breaker to the box, and put a non-fused safety switch at the pole

What you can do instead of having the redundant neutral/ground bonds at the pole and the house is:


Have the utility cut the power
Move the 50A breaker to the box on the house as it'll be the house main breaker.  Remove the box from the pole.
Put a 2 pole, non-fused, outdoor type (NEMA 3R enclosure) safety switch (UL98 listed, with a minimum fault current rating of 10,000A and a minimum working current rating of 50A) in place of the box on the pole that had the 50A breaker there.  Siemens, GE, Square-D, and Eaton all make quite suitable units, although they won't exactly be cheap.
Land the hots on one pole of the safety switch, and the neutrals on the neutral/grounding block in the safety switch.  Torque all connections to spec, and make sure that any grounding electrode wires at the pole that went into the old box go into the neutral/grounding block on the safety switch.  You will need to make sure this block is bonded to the enclosure as well.
Put a "METER DISCONNECT -- NOT SERVICE EQUIPMENT" label on the safety switch (inside cover is good, but don't put it over the existing labeling there).
Have the utility turn the power back on


This is a hot sequence meter disconnect as per NEC 230.82(3), and provides the best of both worlds:


You can cut off power to your house completely at the pole, allowing you to work on the box on the house safely
The neutral-ground bond, though, can still live at the house, allowing that to be the main panel, and allowing you to safely use the existing ungrounded feed from the pole.