title: Insert 70 Million Reoced in MYSQL
body: <p>I have around <code>70 Million</code> record in files, and i have to read all the record and insert it into <code>MYSQL</code> database. I take about <code>2 days</code> to insert all the data, the issue is that now i have to add few conditions before adding them to the database. But when i add condition it take more days to complete. And this process is done <code>every week</code>. So if it take 4 days to read and sort the data in 7 days week, then there will be no use of it.</p>
  
  <p>Is there any quick why so that it take less time to read the data.</p>
  
  <p>Current File size: <code>500MB</code></p>
  
  <p>The main condition is that when i read the data and it not meat the requirement then that data store in an other table.</p>
  
  <p><code>I have an other option as well for this.</code> If this will not done. I have to show data in front-end after the insertion. Let's suppose there is no quick way to minimize the time. so after the insertion of all data, i have to show this data on <code>front-end</code> and <code>100</code> records per page, so is there any way i make condition in pagination that it will show only 100 record which meat my requirements and next <code>100</code> which meat requirements. </p>
  
  <p>Let suppose i Read <code>100</code> record and in 100 there is only 80 record which meat my requirements. so that 1st page must pull <code>20</code> more record which meat my requirements.</p>
  
  <p>Here is The example of the data and how it show work</p>
  
  <pre><code>master_account_number,sub_account_number,current_balance,first_name,last_name,city,state,zip,client_id
  
  01-962040041,01-000050003,11.25,IRENE,AULTMAN,MICHELLVILLE,MD,20721,000969
  </code></pre>
  
  <p>I need this format for every row. If any <code>field is empty</code> it will be consider as a <code>bad data</code> and store in other table. Or if format of the data not match then it also consider as a <code>bad data</code>. If balance is <code>zero</code> then also bad data, and many more conditions as well.</p>
  
  <p>i am using </p>
  
  <pre><code>Codeigniter 
  IIS 7 Server
  MYSQL
  </code></pre>
  
  <p>Storing all data as a string</p>
  
  <p>And i use this query to load the data into <code>DB</code>.</p>
  
  <pre><code>LOAD DATA INFILE 'C:\\inetpub\\wwwroot\\www.suburbancredit.com\\csv\\atest.csv'
  INTO TABLE accounts
  FIELDS TERMINATED BY ','
  OPTIONALLY ENCLOSED BY '"' 
  LINES TERMINATED BY '\n'
  IGNORE 1 LINES 
  (`accountNumber`, `caseNumber`, `current_balance`, `first_name`, `last_name` , `city`, `state`, `zip`, `client_id`)
  </code></pre>