Django with huge mysql database
Question: What would be the best way to import multi-million record csv files into
django.

Currently using python csv module, it takes 2-4 days for it process 1 million
record file. It does some checking if the record already exists, and few
others.

Can this process be achieved to execute in few hours.

Can memcache be used somehow.

Update: There are django ManyToManyField fields that get processed as well.
How will these used with direct load.


Answer: I'm not sure about your case, but we had similar scenario with Django where
~30 million records took more than one day to import.

Since our customer was totally unsatisfied (with the danger of losing the
project), after several failed optimization attempts with Python, we took a
radical strategy change and did the import(only) with Java and JDBC (+ some
mysql tuning), and got the import time down to ~45 minutes (with Java it was
very easy to optimize because of the very good IDE and profiler support).

