How is SQL Server host identity ensured? When I connect to some e-commerce website with my browser I use HTTPS that uses a certificate that (more or less) guarantees that example.com is indeed example.com.

Now I connect to a SQL Server somewhere on the Internet - it might be SQL Azure for example and then I just connect to a URL like abcdef.database.windows.net. There's a chance that there's a fake server there and then I expose my sensitive data. I haven't found any evidence that certificates or anything equivalent is used when connecting to SQL Server.

How is SQL Server host identity ensured?
I'm not familiar with Azure but I can speak to SQL Server in general. SQL has a property called Force Encryption (which is set through SQL's Configuration Manager), which as it sounds will force clients to connect over encrypted sessions. This is done with certificates just like SSL. This guarantees the authenticity of the server as well as providing for encrypted communications. 

It's a bit nuanced - you can use a certificate issued by a CA that the clients trust or SQL can use a self signed certificate (less secure). There's also client side settings you can set in the connection string that will influence the connection - depending on ODBC or OLEDB the pertinent properties are something like "Encrypt" and "TrustServerCertificate".