title: Why do I need to shuffle my PCollection for it to autoscale on Cloud Dataflow?
body: <h2>Context</h2>
  <p>I am reading a file from Google Storage in Beam using a process that looks something like this:</p>
  <pre><code>data = pipeline | beam.Create(['gs://my/file.pkl']) | beam.ParDo(LoadFileDoFn)
  </code></pre>
  <p>Where <code>LoadFileDoFn</code> loads the file and creates a Python list of objects from it, which <code>ParDo</code> then returns as a <code>PCollection</code>.</p>
  <p>I know I could probably implement a custom source to achieve something similar, but <a href="https://stackoverflow.com/questions/41212272/google-cloud-dataflow-consume-external-source">this answer</a> and Beam's <a href="https://beam.apache.org/documentation/sdks/python-custom-io/" rel="noreferrer">own documentation</a> indicate that this approach with pseudo-dataset to read via <code>ParDo</code> is not uncommon and custom sources may be overkill.</p>
  <p>It also works - I get a <code>PCollection</code> with the correct number of elements, which I can process as I like! However..</p>
  <h2>Autoscaling problems</h2>
  <p>The resulting <code>PCollection</code> does not autoscale at all on Cloud Dataflow. I first have to transform it via:</p>
  <pre><code>shuffled_data = data | beam.Shuffle()
  </code></pre>
  <p>I know <a href="https://stackoverflow.com/questions/41212272/google-cloud-dataflow-consume-external-source">this answer</a> I also linked above explains pretty much this process - but it doesn't give any insight as to why this is necessary. As far as I can see at Beam's very high level of abstraction, I have a <code>PCollection</code> with N elements before the shuffle and a similar <code>PCollection</code> after the shuffle. Why does one scale, but the other not?</p>
  <p>The documentation is not very helpful in this case (or in general, but that's another matter). What hidden attribute does the first <code>PCollection</code> have that prevents it from being distributed to multiple workers that the other doesn't have?</p>