title: ElasticSearch, Logstash, MySQL: how do I speed up a large import?
body: <p>I'm trying to import a large (~30M row) MySQL database to ElasticSearch. Cool cool, there's a <code>logstash</code> tool that looks like it's built for this sort of thing; its JDBC plugin will let me connect right to the database and slurp up the rows real fast.</p>
  
  <p>However! When I try it, it bombs with <code>java.lang.OutOfMemoryError</code>. Okay. It's probably trying to batch up too many rows or something. So I add <code>jdbc_fetch_size =&gt; 1000</code> to my configuration. No dice, still out of memory. Okay, maybe that option doesn't work, or doesn't do what I think?</p>
  
  <p>So I try adding <code>jdbc_paging_enabled =&gt; true</code> and <code>jdbc_page_size =&gt; 10000</code> to my config. Success! It starts adding rows in batches of 10k to my index.</p>
  
  <p>But it slows down. At first I'm running 100k rows/minute; by the time I'm at 2M rows, however, I'm at maybe a tenth of that. And no surprise; I'm pretty sure this is using LIMIT and OFFSET, and using huge OFFSETs in queries is real slow, so I'm dealing with an O(n^2) kind of thing here.</p>
  
  <p>I'd really like to just run the whole big query and let the cursor iterate over the resultset, but it looks like that isn't working for some reason. If I had more control over the query, I could change the LIMIT/OFFSET thing out for an <code>WHERE id BETWEEN val1 AND val2</code> kind of thing, but I can't see where I could get in to do that.</p>
  
  <p>Any suggestions on how I can not crash, but still run at a reasonable speed?</p>