Difficulty removing files from filegroup
Try running DBCC UPDATEUSAGE against the database as SQL may be reporting incorrect data on the space used in the database.

See: https://docs.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-updateusage-transact-sql?view=sql-server-2017

If this alters any space usage metadata then you could retry the EMPTYFILE again, then remove the files if it is successful.