Electricity passes through a bulb holder, even when the switch is off
Look closely at that light switch.  See the two switch blocks (one blank) and the almost-square translucent fringe around both of them?  That lights up.  That is a backlight designed to help you find it in the dark.  You may have known this and forgot to mention it, or maybe it doesn't work for a particular reason.  Here is how those switches are wired internally.  



As you can see, power flows through this circuit at all times.  If it's not going through the switch, it's going through the backlight.  Incandescent bulbs are quite low resistance when they are not burning (their resistance increases greatly as they heat up).  These backlights exploit that by leaking a tiny amount of current through the incandescent bulb, too little to make it glow, but enough to light the backlight. 

Well, LED bulbs monkeywrench this arrangement. They aren't low resistance when off.  If current will go through the LED bulb with the backlight impeding flow, it may make the LED bulb function.  It might even make the LED bulb work, and the backlight not.  And that kind of thing is what you're running into. 

Solution: 


Go back to incandescent bulbs.  If it's a multi-bulb lamp, only one needs to be incandescent. 
Go to a plain switch with no backlight. 
Get a more modern switch that works the new way (not depending on leakage current, but requiring a neutral to the switch).