How can I delete a file from the primary filegroup?
The primary database file can never be removed because it contains special page structures that exist only on that file and cannot be relocated.  If you want a single database file, move all data from the secondary file using DBCC SHRINKFILE with the EMPTYFILE option and then remove the secondary file.  Finally, rebuild/reorganize indexes to reduce fragmentation.

An example of unmovable page is the database boot page, which is page 9 of the primary data file of every database.  See Paul Randal's Anatomy of a Page for details.