title: Spark benchmark setup
body: <p>I would like to do a benchmark on Spark which consists in:</p>
  
  <p>Have 10TB of uncompressed data where each line is at this format: float;float;string.</p>
  
  <p>The job will basically do :</p>
  
  <ul>
  <li>map on lines to split</li>
  <li>filter on the string </li>
  <li>reduceByKey where key is based on the two floats.</li>
  <li>store result in HDFS</li>
  </ul>
  
  <p>Question are : 
   - how much disk space should I have
   - how much memory
   - what are the parameters for the job (#exec, #core, #mem) and for yarn-site.conf</p>
  
  <p>Currently I have 15To of storage on 10 nodes (16Cores, 16GB, 1.5To of storage) and it fails like : </p>
  
  <blockquote>
    <p>ERROR client.TransportClient: Failed to send RPC 6631382768729976966 to benchophadoopslaves1/A.B.C.D:43365: java.nio.channels.ClosedChannelException
    java.nio.channels.ClosedChannelException
    17/07/06 17:05:42</p>
    
    <p>WARN netty.NettyRpcEndpointRef: Error sending message [message = Heartbeat(41,[Lscala.Tuple2;@4f4d418,BlockManagerId(41, benchophadoopslaves1, 34521))] in 3 attempts
    java.io.IOException: Failed to send RPC 6631382768729976966 to benchophadoopslaves1/A.B.C.D:43365: java.nio.channels.ClosedChannelException
            at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
            at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
            at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)</p>
  </blockquote>