title: flink cluster params - how to set
body: <p>Im doing some proof-of-concept work with <strong>Flink</strong> and have gotten to the point where I want to try my various jobs (topologies?) on a 4 node cluster. </p>
  
  <p>Machines:  </p>
  
  <ul>
  <li>32 core  </li>
  <li>96g Gb RAM</li>
  </ul>
  
  <p>The topologies vary from 3 to 6 'tasks' (workers? bolt-analogs?). I'm hoping that someone here can offer some suggested settings. Specifically:</p>
  
  <ul>
  <li>taskmanager.numberOfTaskSlots: set this to # of cores?   </li>
  <li>taskmanager.heap.mb: "This value should be as large as possible." 96Gb? Really?   </li>
  <li>parallelism.default: tried setting this to 30. Got this error<sup>1</sup>.   </li>
  <li>parallelization.degree.default: I tried upping this value but it didn't seem to have any effect. Tasks always show '1' for parallelism.</li>
  <li>any other settings that people have found useful / interesting?   </li>
  </ul>
  
  <p>One task in particular: reading from <strong>Kafka</strong> where topic in question has 6 partitions. From each of these partitions I want to read, aggregate and write to <strong>Cassandra</strong>. When I implemented this job in <strong>Storm</strong> it had 6 bolts to read the data and several times as many to write. (IE 6 read, 18 write)</p>
  
  <p>If <strong>Flink</strong> gets adopted by my company each machine will run numerous, simultaneous jobs. How will the config params alter under such a circumstance? </p>
  
  <p>FWIW: cluster is v1.0-SNAPSHOT.</p>
  
  <hr>
  
  <p>EDIT: <a href="https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#configuring-taskmanager-processing-slots" rel="nofollow">This</a> seems useful.</p>
  
  <p><sup>1</sup> "Insufficient number of network buffers: required 30, but only 8 available. The total number of network buffers is currently set to 2048." Does this mean that 2000 buffers are being used when nothing is happening?</p>