CSS or and JS aggregations all makes site slower!
You're confusing PSI and webpage loading times; they are not the same thing. 

PSI is series of benchmarks for optimizing performance. It's a checklist Google provides to give you suggestion for improving performance. It doesn't gauge what your site's actual speed is. Use your browser's network activity inspector if you'd like to analyze site load times against CSS/JS aggregation.