What makes a magnetic field probe insensitive to E-fields?
It is sensitive to E-fields, like any antenna it depends on the frequency.

First, about the antenna performance:
This loop antenna will resonate and be quite sensitive when its circumference approaches \$\lambda\$, and will be somewhat sensitive from \$\lambda/4\$. Below this, the loop antenna becomes increasingly mismatched to the \$50 \Omega\$ coax and its effective gain or sensitivity drops quickly.

When the loop is small, its near fields are predominantly H-fields, because having an impedance close to zero, the voltages present on the antenna are small and the currents are large. (Just the opposite of a short dipole, whose impedance is almost open-circuit, so it has higher voltages and smaller currents, and hence its near fields of a short dipole are predominantly E-fields). The radiated far fields of both are of course in the ratio \$E/H=120\pi\$.

Second, about the "shielding" and balun:
The loop can be made of any solid wire or tube. The diameter affects its impedance slightly, but the general construction does not change its sensitivity to E and H fields.

The gap size has almost no effect on the antenna at all. When it's too small, the capacitance between the opposing faces will load the loop slightly.
(Interesting diversion - the capacitance can actually cause the loop to resonate, making a loop-gap resonator, a very high Q resonator).

However, you must take care not to create a loop-dipole hybrid, by connecting one part of the loop to a long piece of coax.

One solution is to mount the transmitting or receiving circuit inside the metal tube, or keep it small, so cannot influence the fields.

Another way is to use a transformer to feed it with balanced voltages on the two sides.

But the simplest is to do what you've done in the picture, which is:
a) lead the coax out from the loop along a the plane of zero electric field,
b) where the coax meets the loop, connect the coax braid to the loop so no stray fields can exist there,
c) pass the coax inside the loop, where there are no fields, until it reaches  the feedpoint, and
d) at this gap, connect the coaxial cable to the two sides of the loop, braid to the loop part that contained the coax, and inner to the other side.  

This answer at the Ham Stackexchange has a long writeup on the action of the balun