In what situation can snapshot restore be blocked after SET SINGLE_USER? In my tests, I have the following script:
USE master;
ALTER DATABASE [TestDB] SET SINGLE_USER WITH ROLLBACK IMMEDIATE
RESTORE DATABASE [TestDB] FROM DATABASE_SNAPSHOT = 'TestDBSnapshot'

In some case that I am not able to understand, I get the following error:
Database state cannot be changed while other users are using the database 'TestDB'
RESTORE DATABASE is terminating abnormally.
Changed database context to 'master'.
Nonqualified transactions are being rolled back. Estimated rollback completion: 100%.

It seems the messages are not in the correct order (.NET issue?), but aside from that, what is the situation when I can get RESTORE blocked after SET SINGLE_USER? I would expect SINGLE_USER to give me exclusive access (and it works most of the time).
After this error, the DB is indeed in SINGLE_USER state (but not restored).
When you set the database to single user, it allows a single user into the database.  If there's someone else trying to get into the database, and you don't have a connection to the database someone else could get into the database.
A single script like:
use master; 
alter database MyDb set single_user; 
use MyDb

...is still a single connection.  Something else would need to log into the SQL Server and into that database.  It could be a user, a job, an admin.
Setting the database into restricted user will block anyone who isn't a member of the db_owner fixed server role from connecting to the database.  I'd recommend that, or shutdown the app/web servers while you do the database restore.