How many libraries is too much before there is a performance issue?
In Drupal, the best way to do your CSS is to keep it componentalized, and attach the CSS using #attached directly to the item to which it belongs. This way, the CSS is only ever attached when needed, and if a user overrides the render array further down the pipeline, removing the element to which the assets are attached, the assets will not be attached to the page, including unnecessary CSS.

Then you turn on CSS aggregation (or even better, use the ADVAGG module), and Drupal will handle the heavy lifting, aggregating the files as needed for the page.

So the answer to your question is that there aren't too many libraries. What is more important is effective management of the CSS/JS you are attaching.