Sync +- 70 000 records from CSV into drupal I have a CSV file with approx. 70.000 records that I want to import into Drupal 7. Once this is done the client will provide a new CSV file on a daily base that has to sync the updated rows to Drupal. 

I wrote a import with Feeds and the Migrate module, but in both cases it takes like forever to import the 70.000 rows. Is there any other solution to import the data faster in Drupal 7?
If you need an update feature of your records i'm not sure how you'd get around scanning for Diffs to find changes -- this takes time.

I've used Migrate to upload simple CSV files and tables. Have you looked at Migrate 2.6 Hash() function to track data changes in stuff to perform only updates from your source data?
http://www.acquia.com/blog/migrate-26-framework-changes


  With Migrate 2.6, we have introduced the 'track_changes' option to
  MigrateSource. If you pass 'track_changes' => 1 in your options array
  on your source constructor, Migrate will hash the source data after
  prepareRow() is called and save the hash in the map table on initial
  import. On subsequent imports, it will again hash the incoming source
  data and compare it to the saved hash - if the hashes don't match, the
  object will be reimported and the new hash saved, otherwise it will be
  skipped. Of course, this is a slower method than highwater marks, but
  when you must pick up changed records and don't have a good highwater
  mark field in the source data, this is the way to go.


Is there any reason you cannot import your CSV file into MySQL tables and then import your data from there using Migrate or a Drush script? Once in tables you could partition the data/tables easily to run the inserts more concurrently for a speedup.

EDIT:

I hit a .NET webservice for 100,000s of records for campus data at an EDU. I found that parsing the data linearly took me on the order of an 1 hour from top of the XML file to the bottom. By requesting each department individually at-the-same-time (get ~60 XML files at once instead of 1 big xml file) and processing in parallel up to 8 threads (via Ultimate Cron) the same process now takes ~5 minutes.

At some point linear processing of data simply fails and you have to find other means.