Using OAuth2 on non-REST endpoints
A) improve the response time on the REST endpoint to be as good as the
  custom controller


For the REST endpoint, you can use $response-&gt;addCacheableDependency() check this block for example:
https://blog.dcycle.com/blog/2018-01-24/caching-drupal-8-rest-resource/
https://spinningcode.org/2017/05/cached-json-responses-in-drupal-8/

Make sure to add cache context and cache tags, carefully.


  B) use OAuth2 with the custom controller. I've been looking for
  instructions on this but every OAuth2 resource I've come across so far
  assumes I'm using a REST endpoint.


Check the example from "simple_oauth_extras" sub module of "simple_oauth"
https://git.drupalcode.org/project/simple_oauth/blob/8.x-3.x/simple_oauth_extras/simple_oauth_extras.routing.yml#L22

Controller is:
https://git.drupalcode.org/project/simple_oauth/blob/8.x-3.x/simple_oauth_extras/src/Controller/DebugController.php#L39