Indirect Lighting is very slow proportional to vertex count even if they are not in the view
You are making wrong assumptions, something not being directly visible doesn't mean it is not contributing to the rendered pixels, in fact indirect light means exactly the opposite.

The walls out of view are still castings shadows in the visible ones, and the window behind the mist still lets light in even when you look away.

That is true for many direct light situations, even more so  with indirect light, all elements influence scene lighting everywhere, unless behind a watertight boundary.

Also Indirect light already calculates only relevant data as best as possible.

Some optimizations are done internally but it is not trivial to determine what contributes to a scene and what doesn't, more geometry will invariably make a render slower eventually.

If you want faster renders manually optimize your scene, reduce polygon counts, hide unnecessary stuff, simplify lightning, use instanced geometry wherever possible, reduce object count etc.

Also match your expectations to your hardware. If necessary bake lighting into textures to make animations faster.