LOAD DATA INFILE decline in performance
Indexes are not disabled in InnoDB using DISABLE INDEX
(from http://dev.mysql.com/doc/refman/5.1/en/alter-table.html):

For MyISAM tables, key updating can be controlled explicitly. Use
ALTER TABLE ... DISABLE KEYS to tell MySQL to stop updating nonunique
indexes. Then use ALTER TABLE ... ENABLE KEYS to re-create missing
indexes


For InnoDB, you're stuck with these hints: http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html

I think you should also remove the indexes and re-create them after you populated the table (the same thing you tried to do with DISABLE/ENABLE KEYS)