Memory optimized DLLs not being removed
Is this a bug?


No it is not a bug. Its by design. They are kept for troubleshooting and supportability 
purposes. 

From the SQL_Server_2014_In-Memory_OLTP White_Paper 


  Database administrators do not need to maintain the files that are generated by native compilation. SQL 
  Server automatically removes generated files that are no longer needed, for example on table and 
  stored procedure deletion, and on drop database, but also on server or database restart.


I tried to repro your scenario on SQL Server 2014 + RTM + (Build12.0.2000.8) - Dev Edition server by creating a test memory optimized table and checking the dll loaded using 

     SELECT name, description FROM sys.dm_os_loaded_modules
WHERE description = 'XTP Native DLL'


After I dropped my table, the dll still appears in the output of above select statement and the files are still in the folder and after restart they are still there too.

From Books Online - 


  No user interaction is needed to manage these files (.c, .obj, .xml, .pdb., .dll). SQL Server will create and remove the files as necessary.


SO I guess, we just have to follow what Microsoft says - SQL server will manage them for us :-)

ONLY FOR EDUCATIONAL PURPOSE :

I managed to clean up the old files by 


Issuing a manual CHECKPOINT on the database.
Taking database offline and then bringing it online.


Ideally, you should not be restarting the server instance, just manual checkpoint and offline/online of the database will clear the files. 

e.g. Repro :

USE master
GO
create database db1
GO
ALTER DATABASE db1 ADD FILEGROUP db1_mod CONTAINS memory_optimized_data
GO
-- adapt filename as needed
ALTER DATABASE db1 ADD FILE (name='db1_mod', filename='D:\SQLServer2014\MSSQL12.SQL2014\MSSQL\DATA\db1_mod') -- change here as per your need !!
 TO FILEGROUP db1_mod
GO
USE db1
GO
CREATE TABLE dbo.t1
(c1 int not null primary key nonclustered,
c2 int)
WITH (MEMORY_OPTIMIZED=ON)
GO


--- now check if the dll is loaded or not 

SELECT name, description FROM sys.dm_os_loaded_modules
WHERE description = 'XTP Native DLL'



--- now drop table and do a manual checkpoint

use db1;
drop table dbo.t1;
checkpoint


Still the module is loaded in memory (even server restart will load the module sometimes)



The (.c, .obj, .xml, .pdb., .dll) are still present in the folder :



Now take the database offline and then bring it online - the (.c, .obj, .xml, .pdb., .dll) are all gone ...