Is there a way to analyze what's using frametime or view all simulated meshes at once?
Alright, turns out using simplify solved the issue. I didn't know that the subD modifieres are such a frame eater for nonanimated objects.