SQL Server Management studio slow connection or timeout when using Windows Authentication
Try running a trace with SQL Profiler while you, and then your coworker, connect to the server.
Select RPC, SQL Statement &amp; PreConnect - Starting/Completed.
Select Save Results To Table option, then compare the 2 tables to find the bottleneck.

Or, since you're connecting by IP, it could be doing a Reverse DNS lookup. If so, add a entry in your hosts file.