Floating power supply for Arduino - Is it still connected to the USB ground?
Does this mean that even with the external power supply the Arduino will be connected to the UGND, and in effect the protective earth?


Yes.


  Would it be ok to remove the jumper connection between UGND and GND?


No. I don't know quite why they provided that jumper. I can see no scenario in the datasheet where it suggests the two can be separate.


  Or should I try something like a linear USB isolator like this?


If you want it to float then you need galvanic isolation on all USB pins. The device you link to will do that for you. There are others as well.

Personally in these situations I prefer to use a wireless communication system so there is no worry about grounds and isolation - it's naturally isolated.