title: Why can the maxparallelism of a flink job not be updated without losing state?
body: <p>I just read that the maximum parallelism (defined by setMaxParallelism) of a Flink job cannot be changed without losing state. This surprised me a bit, and it is not that hard to imagine a scenario where one starts running a job, only to find out the load is eventually 10x larger than expected (or perhaps the efficiency of the code is below expectations) resulting in a desire to increase parallelism.</p>
  <p>I could not find many reasons for this, other than some references to key groups. The most tangible statement I found <a href="https://github.com/apache/flink/pull/1988" rel="nofollow noreferrer">here</a>:</p>
  <blockquote>
  <p>The max parallelism mustn't change when scaling the job, because it would destroy the mapping of keys to key groups.</p>
  </blockquote>
  <p>However, this still leaves me with the questions:</p>
  <h1>Why is it hard/impossible to let a job change its max paralellism?</h1>
  <hr />
  <p>Based on the above, the following conceptual solution came to mind:</p>
  <ol>
  <li>In the state, keep track of the <em>last used max parallelism</em></li>
  <li>When starting a job, indicate the <em>desired max parallelism</em></li>
  <li>Given that both settings are known, it should be possible to infer how the mappings would need to change to remain valid initially.</li>
  <li>If needed a new state could be defined based on the old state with the new maxparallelism, to 'fit' the new job.</li>
  </ol>
  <p>I am not saying this conceptual solution is ideal, or that it would be trivial to implement. I just wonder if there is more to the very rigid nature of the maximum parallelism. And trying to understand whether it is just a matter of 'this flexibility is not implemented yet' or 'this goes so much against the nature of Flink that one should not want it'.</p>