SSMS cannot connect remote server by IP through Windows authentication
So it turns out the timeout is actually a timeout...
After I expend the connection timeout to 90 sec it connected after about a minute.

Infrastructure guys say this might since it's trying to resolve the machine name or something? (I'm not really familiar with network and setting stuff).
I will update this answer if they figure out any way to speed up the process.