Is it the socket or the bulb?
Bulbs don't really fail intermittently. 99.9% it's your socket that wants replacing.
And, yes, I would do it soon, because intermittent contacts like this can in rare circumstances cause fires due to arcing. This is extremely unlikely on a low-current device like an LED bulb, but it wants addressing sooner rather than later.
As pointed out in the comments, though, it's a simple matter to swap with another bulb and see if the fault follows the bulb or stays with the socket.