Completely remove additional data file from SQL Server 2012 DB
Looks like changing files from one file group to another is not
  possible?


Correct, you would need to create a new file on the correct filegroup and recreate all your objects on the correct filegroup (if any existed) to move them, as you cannot move a file between filegroups.


  The data file is empty so I tried removing the file using alter DB
  remove file, when I run DBCC showfilestats, the file is gone, but when
  I check SYS.master_files, I can see still the entry of the file I want
  removed. Is there a way to remove it completely, even deleting it
  physically from the server as I need to recover the space?


I've seen this before in SQL Server 2012. If you query sys.master_files you will see the drop_lsn field has an LSN value in it for this file. This indicates the LSN at which this file was dropped. You need to perform a log backup to clear this value, then it will disappear from the DMVs.