Ram & Concurrency: MySQL InnoDB LOAD INLINE FILE on 50 30GB CSV files As the title suggests I'm moving around a lot of data. This is in an environment with a 24 core CPU and 2TB of M.2 storage.

My first question is: given I have 16GB of ram, does that mean it's impossible for me to load a 30GB file into SQL (as the entire thing must be stored in ram?)

The second question: This is a fresh table I'm loading into, so I have the luxury of dictating which engine I can use. I've read that InnoDB is better for writing (in which I'm doing alot of). However, MySQL manual says that the CONCURRENCY clause will work on MyISAM. So which engine should I use?

Any other helpful advice is appreciated.

My mysqld conf:

innodb_buffer_pool_size = 12G
innodb_log_file_size    = 4G
innodb_write_io_threads = 18
There are no restrictions on importing/loading a 30GB file to MySQL (Unless your proc/OS are 32 bit which I doubt). The only additional variable I'll suggest to increase is max_allowed_packet=1G... Later you can return it back to something about 128M (after import)

About the second question: As DBA you will select the storage engine depending on the use of the data after imported, not depending exclusively on the import step. So following standards and current support on transactional procedures, backups, reliability and so on, I suggest you to use Innodb, but again this will depend on the future use of your data.

Since 30 GB actually is not that big, concurrency won't make a huge difference here when importing data.