Viewport performance changes when in camera?
After more troubleshooting, I found it. My character's rig had an eye target bone constrained to copy the location of the camera. I had totally forgotten about it because the constraint was disabled. So apparently, disabled constraints still get calculated in some way and eat up a ton of resources? But that's a whole other question. Deleting it fixed the issue.