title: How to properly set up DataImportHandler for MySQL database with large number or records?
body: <p>I have set up Solr's data import handler as instructed in manual. Solr reads the records from a MySQL database. The database has large number of records (expected is milliards/billions). 
  I have read that batch size does not work for MySQL because the JDBC driver does not support it. I have tried setting it up to -1. In this case, Solr performs one select and gets all records from the DB and indexes them.</p>
  
  <p>Now, I have problem, since a timeout occurred  while indexing and caused it to stop. I see that Solr hasn't written any id value in the properties file after the exception occurred. I am not sure how to proceed with indexing the rest of the records.</p>
  
  <p>Can anyone suggest to me how to set up Solr with MySQL for a proper data import?</p>
  
  <p>Below is data config I am currently using.</p>
  
  <pre class="lang-xml prettyprint-override"><code>&lt;dataConfig&gt;
      &lt;dataSource type="JdbcDataSource" name="ds-2" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/myowndb" batchSize="-1" /&gt;
      &lt;document name="statuses"&gt;
          &lt;entity name="status" query="select s.*, ti.id2, ti.value2 from tblTable1 s inner join tblTable2 ti on s.table2Id = ti.id;"&gt;
              &lt;field column="id" name="id" /&gt;
              &lt;field column="statusID" name="statusId" /&gt;
              &lt;field column="type" name="type" /&gt;
              &lt;field column="date" name="date" /&gt;
              &lt;field name="id2" column="id2" /&gt;
              &lt;field name="value2" column="value2" /&gt;
          &lt;/entity&gt;
      &lt;/document&gt;
  &lt;/dataConfig&gt;
  </code></pre>
  
  <p><strong>EDIT</strong>:</p>
  
  <p>Based on my tests today, it looks like batchSize is working. If batchSize is set to -1, it will make single request to MySQL retrieving all rows at once. If set to some value greater than 0, it will put every record in memory before processing.</p>
  
  <p>New question is next: how to set up data import handler so it can index in batches? Not only to perform batch select from database, but to index collected set before collecting next one.</p>
  
  <p><strong>EDIT: Specified question</strong>
  New question that came up from reading is next: is it possible to mark row in database as processed? There are only two events available in DIH, onImportStart and onImportEnd. </p>
  
  <p>Current flow in ideas lead me to implement EntityProcessor. If it would be possible to know when some row is indexed, it would also be easy to mark isIndexed flag in database for indexed row. This is in case I implement custom EntityProcessor.</p>