I have a database which I regularly need to import large amounts of data into via some python scripts. Compacted, the data for a single months imports takes about 280mb, but during the import file size swells to over a gb.

Given the 2gb size limit on mdb files, this is a bit of a concern. Apart from breaking the inserts into chunks and compacting inbetween each, are there any techniques for avoiding the increase in file size?

Note that no temporary tables are being created/deleted during the process: just inserts into existing tables.

And to forstall the inevitable comments: yes, I am required to store this data in Access 2003. No, I can't upgrade to Access 2007.

If it could help, I could preprocess in sqlite.

Edit:

Just to add some further information (some already listed in my comments):

* The data is being generated in Python on a table by table basis, and then all of the records for that table batch inserted via odbc
* All processing is happening in Python: all the mdb file is doing is storing the data
* All of the fields being inserted are valid fields (none are being excluded due to unique key violations, etc.)

Given the above, I'll be looking into how to disable row level locking via odbc and considering presorting the data and/or removing then reinstating indexes. Thanks for the suggestions.

Any further suggestions still welcome.