SQL Server Agent maintains thousands of sleeping connections
KB 3034297 - Cumulative Update 6 for SQL Server 2014 describes the issue that you are encountering and is fixed in SQL Server 2014 CU6


  When you turn on SQL Server Managed Backup in Microsoft SQL Server 2014. When you let SQL Server and SQL Server Agent run for several days, there are many connections with the program name SQLAgent - Temporary Worker. They are created by "sqlagent.exe," and cannot be closed.