title: Generating big random JSON dataset for ElasticSearch
body: <p>I am currently required to generate a few hundred GB worth of test data for an ElasticSearch index for load testing. This means hundreds of millions of "docs". </p>
  
  <p>The format I need to generate looks like:</p>
  
  <pre><code>{
  field 1 : &lt;ip&gt;
  field 2 : &lt;uuid&gt;
  field 3 : &lt;date&gt;
  field 4: &lt;json-array of string&gt;
  ...
  ...
  }
  </code></pre>
  
  <p>for around 40-50 fields per doc. Generated data needs to match the index template for the specific index I am required to test.</p>
  
  <p>Ok, so sounds straight forward right? A normal JSON dataset generator that can handle generating a few hundred million Json docs is the way to go provided I can find one that supports format.</p>
  
  <p>The problem is that the ES Bulk upload API requires upload to be supplied following way. For EACH doc, first a "command json" containing meta-data for a doc, then the json doc itself:</p>
  
  <pre><code>POST _bulk
  { "index" : { "_index" : "test", "_id" : "1" } }
  { "field1" : "value1" }
  </code></pre>
  
  <p>The one free solution that look like it might support generating huge datasets only support generating Json with uniform format. Which means I can't make it generate the command followed by the doc.</p>
  
  <p>So I tried to generate using my own bash script based on some pre-existing data (I randomize doc-id and some other fields) to generate data. But the problem with that is I need to run my bash script in parallel up to 100s of times at once to generate the data in a timely manner. And the /dev/urandom in bash is "conflicting", as in it is generating the -same- random data across different scripts when ran in parallel when I need the doc-id to be unique.</p>
  
  <p>This is getting long, but any help for either </p>
  
  <p>1) A free solution which can generate the large datasets in JSON and in the format I need</p>
  
  <p>OR</p>
  
  <p>2) A solution for bash random generation process when ran in parallel</p>
  
  <p>Would be appreciated. Thanks.</p>