Sql Server 2008 R2 DBCC SHRINKFILE Severe Error
Have you tried specifying both the file name and the target size in MB, e.g.

DBCC SHRINKFILE('filename', 20);


Or the TRUNCATEONLY option, which doesn't try any of the nasty stuff like reorganizing all of your physical data:

DBCC SHRINKFILE('filename', TRUNCATEONLY);


Also you could try making sure the database is in single user mode first:

ALTER DATABASE db SET SINGLE_USER WITH ROLLBACK IMMEDIATE;


...and turn off any backup / log backup or other maintenance jobs that may be conflicting with your shrink operation. If it still occurs, please run a CHECKDB against this database and let us know the results:

DBCC CHECKDB (db) WITH NO_INFOMSGS, ALL_ERRORMSGS;


But just as a side note, you do know that shrinking is bad and should usually be reserved for extraordinary events and circumstances, right?