How to remove stylesheets only for anonymous users?
Having tried to do this myself before, this is probably not a route you want to go down.  This approach becomes harder to maintain as you install more modules (different modules can inject CSS on different paths), and often can turn into a maintenance nightmare.
Drupal can aggregate CSS files out of the box; this is often all you need to improve performance dramatically.
To turn on the basic aggregation, you can check Aggregate CSS files at /admin/config/development/performance.
To do something more complicated, try using the Advagg module. Configuring this module can be complex but it allows you to do more specific aggregation, CSS minification, and so on.