mysql insert into indexed table taking long time after few million records
You should load such a huge file in chunk for faster loading of data,
Here it is mentioned problems loading huge Load local data file and solution how to solve it and make it faster.

http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/


MyISAM tables you should follow this steps to fasten loading data:


With some extra work, it is possible to make LOAD DATA INFILE run even faster for a MyISAM table when the table has many indexes. 


Execute a FLUSH TABLES statement or a mysqladmin flush-tables command.
Use myisamchk --keys-used=0 -rq /path/to/db/tbl_name to remove all use of indexes for the table.
Insert data into the table with LOAD DATA INFILE. This does not update any indexes and therefore is very fast.
Re-create the indexes with myisamchk -rq /path/to/db/tbl_name. This creates the index tree in memory before writing it to disk, which is much faster that updating the index during LOAD DATA INFILE because it avoids lots of disk seeks. The resulting index tree is also perfectly balanced.
Execute a FLUSH TABLES statement or a mysqladmin flush-tables command.


LOAD DATA INFILE performs the preceding optimization automatically if the MyISAM table into which you insert data is empty. The main difference between automatic optimization and using the procedure explicitly is that you can let myisamchk allocate much more temporary memory for the index creation than you might want the server to allocate for index re-creation when it executes the LOAD DATA INFILE statement.

You can also disable or enable the nonunique indexes for a MyISAM table by using the following statements rather than myisamchk. If you use these statements, you can skip the FLUSH TABLE operations:

ALTER TABLE tbl_name DISABLE KEYS;
ALTER TABLE tbl_name ENABLE KEYS;


For more information refer these:

http://dev.mysql.com/doc/refman/5.5/en/optimizing-myisam-bulk-data-loading.html

https://wiki.rice.edu/confluence/display/~as43/Make+LOAD+DATA+INFILE+run+even+faster+for+a+MyISAM+table