I run sp_who on the database, and there is a "TASK MANAGER" process. What is it?
Any SPIDs below 50 are system processes. "TASK MANAGER" is one of these. I'm not sure exactly what it does but it is initiated only by the server itself, dealing with system level jobs. Seeing this on the database "sp_who" is nothing to worry about.

I highly doubt this process is the reason why you are getting kicked out of the DB running in single user mode, but if you are curious what the process might be doing you could run dbcc inputbuffer(the_spid_of_process) to check if it is running any commands.