title: Generate Random Data for Cassandra DB
body: <p>I have a big data project for school that requires us to build and query a 8 node Cassandra system. The system must contain at least seven terabytes of data. I have to generate all this data myself. There is no requirement that the data be "relevant" to the assignment -- ie each column can just be a random int. That being said it is required that each value is random or based on a random sequence. </p>
  
  <p>So, I wrote a simple java program to just generate random <code>ints</code>. I can generate <code>~200 MB</code> of random test data in <code>~120s</code>. Now unless my math is off, then I think I'm in a pickle. </p>
  
  <p>There are <code>35000</code> <code>200MB</code> units in <code>7 terabytes</code>.</p>
  
  <p>35000 * 120 = <code>4 200 000 seconds</code></p>
  
  <p>4 200 000 / 3600 ~ <code>1167hours</code></p>
  
  <p>1167 / 24 = <code>49 days</code></p>
  
  <p>So, it appears that it will take 49 days to generate all the test data needed. Obviously, this is impractical. I'm looking for suggestions that will increase the rate at which I can generate data. </p>
  
  <p>I've considered/considering:</p>
  
  <blockquote>
    <p>setting replication factor to 8 to reduce the amount of data needed to be generated, and also running the data generation program on all 8 nodes.</p>
  </blockquote>
  
  <p>edit: how I'm generating the data</p>
  
  <pre><code>private void initializeCols(){
      cols = new ArrayList&lt;Generator&gt;();
      cols.add(new IntGenerator(400));
  }
  
  public ArrayList&lt;String&gt; generatePage(){
      ArrayList&lt;String&gt; page = new ArrayList&lt;String&gt;();
      String line = "";
      for(int i = 0; i &lt; PAGE_SIZE; i++){
          line = "";
          for(Generator column : cols){
              line += column.gen();
          }
          page.add(line);
      }
      return page;
  }
  </code></pre>
  
  <p>originally I was generating more <code>test specific</code> data like <code>phone numbers</code> etc. but then I decided to just generate random <code>ints</code> in order to shave some time off -- not much savings. Here is the <code>IntGenerator</code> class. </p>
  
  <pre><code>public IntGenerator(int series){
      this.series = series;
  }
  
  public String gen(){
      String output = "";
  
      for(int i = 0; i &lt; series; i++){
          output += Integer.toString(randomInt(1,1000));
          output += SEPERATOR; 
      }
      return output;
  }
  </code></pre>