What is the purpose of a user's default database? As I understand it the default database is specified so that in case you don't specify one in the connection string, it will know where to start. Is this correct?

I've never seen a connection string without a database specified, is it even possible? Why would anyone want to do this? It just makes it less transparent to which database you are connecting.

In other words is there any reason to choose anything else than master as the default database?
Well Management Studio, for example, by default, only asks for (and only requires) server name and credentials. You can override these if you bother going into Options, but most people don't.

Do you really want all of your users connecting to master? In a lot of shops this can cause quite a bit of headache, especially with loose security. If users have the permission to create things in master, if they connect to master by default, they inevitably will create things there. And this can only mean cleanup will be needed later.