How to cache custom controller markup for logged-in users?
One thing that you can do is the same as the entity view builder is doing, use a #pre_render callback.

If you look at \Drupal\Core\Entity\EntityViewBuilder::viewMultiple(), you can see it doesn't really do anything except prepare the render array and add a #pre_render callback to the whole thing.

The reason for that is that the pre_render callback will only be called on a cache miss. So you return your array with the cache keys/tags/contexts/max-age, the renderer service will attempt to load from the render cache. On a cache hit, it will return the previously cached HTML, on a miss, it will call the pre_render callback and you can do the things you want to do there.