Could too many idle connections affect PostgreSQL 9.2 performance?
I seriously doubt idle connections are your issue.  PostgreSQL is very capable of handling idle sessions since they take almost no resources.  Active sessions on the other hand are much more likely the cause of slowness.

See https://wiki.postgresql.org/wiki/Number_Of_Database_Connections for reference on the number of database connections.