Microsoft SQL Server, Error: 10054
You say that by using RDP directly for logging onto and managing that server you obviously have enough rights to manage the server.

What is different is that you are now no longer using a direct connection?  


If there are more than two computers involved, then probably Kerberos needs to be configured.
Are you sure that SQL Server Agent installed correctly.  Check for installation errors in the Error Log.
Have you manually tried starting or reconfiguring SQL Server Agent?  Or used the Services panel to start SQL Server?  Or run net start SQLSERVERAGENT.


If Kerberos is needed because of multiple hops between computers: Microsoft® Kerberos Configuration Manager for SQL Server® has some guidance on setting up and configuring Kerberos in your environment.