Remove lazy caching of backend config
Remove lazy caching of backend config

The current implementation is not thread-compatible which already
led to various intermittent threading bugs.

Let's just remove the lazy caching. There is no evidence that
it brings any improvements in the first place.
