How to eliminate a small voltage in a home AC lighting circuit
The center ground is a red herring

It has nothing to do with your lighting issue.  The only other possible explanation is a combinatioj of two things: A) a seriously defective grounding electrode system/neutral-ground equipoetntial bond; and also a ground fault in some your equipment. 

If your lights are screw-in or snap-in types, most likely they only have two terminals - such as the two pins on a GU connector, or the center and threads of a screw-in.  If the bulb only has 2 terminals, it cannot interact with ground. So the odd ground is not causing the lighting issue. 

It's your switch

99% of the time, the LEDs that won't extinguish, that's because your switch is not a switch.  I bet incandescents work fine, or if it's a multi-lamp fixture, the problem goes away if one of the bulbs is incandescent. 

The problem is that you have an "active switch" - it is a dimmer, lighted switch, motion sensor or smart switch that is a powered device and needs power to run its own stuff.  A plain switch normally doesn't have access to ground, heck, doesn't even have access to neutral, which is the proper way for a device to power itself (between hot and neutral). So when these active switches are made for retrofit, they can't power themselves conventionally because neutral may not be present in the switch box. 

Even if the switch gets ground for safety, it should not interact with it electrically.  

So they use a trick of incandescent bulbs.  An incandescent which is off is basically a dead short.  That changes when the bulb warms up.  ** Anyway, the trick is that the powered device leaks a tiny amount of current through the incandescent bulb - too little to light it, but enough to power the device.  Effectively the device is "in series with" the bulb. 

This falls apart with CFLs and LEDs.  They have electronic switching power supplies, which Do Not Like weird mini-currents running through them, and they do unpredictable things.  Some LEDs are able to simply allow the current through, and due to their efficiency it is enough to light them.  

Change that obsolete switch out to either a plain switch or a modern one made for LED.  



** When you read a relay spec and it has a much lower number for "tungsten", it is referring to that "inrush current" you get from that dead-short load as the incandescent is warming up. This differs from the "ballast" or "HID" number, which refers to the very-high-voltage inductive kick you get from a tranformer based ballast when you shut off the light. Generally, that’s what you use a snubber for, across the relay contacts to discharge that kick.