I run sp_who on the database, and there is a "TASK MANAGER" process. What is it? I have a sporadic problem, when upgrading SQL Server in a single-user mode, some other application somehow logs into the database and kicks my process out.

Last time when that happened, I ran sp_who, and I could see that a process that took control of the database is "TASK MANAGER". EDIT By "took control" I mean that my process crashed with exception Database '...' is already open and can only have one user at a time. while it was running database upgrade code. SqlConnection was NOT closed or disposed.

Anyone could tell me what is this process, what is it's purpose, and how in the world it could get into a database, which is in single-user mode, and there is an active connection?
Any SPIDs below 50 are system processes. "TASK MANAGER" is one of these. I'm not sure exactly what it does but it is initiated only by the server itself, dealing with system level jobs. Seeing this on the database "sp_who" is nothing to worry about.

I highly doubt this process is the reason why you are getting kicked out of the DB running in single user mode, but if you are curious what the process might be doing you could run dbcc inputbuffer(the_spid_of_process) to check if it is running any commands.