title: How to solve stability problems in Google Dataflow
body: <p>I have a Dataflow job that has been running stable for several months.
  The last 3 days or so, I've problems with the job, it's getting stuck after a certain amount of time and the only thing I can do is stop the job and start a new one. This happened after 2, 6 and 24 hours of processing. Here is the latest exception:</p>
  <pre><code>java.lang.ExceptionInInitializerError
  at org.apache.beam.runners.dataflow.worker.options.StreamingDataflowWorkerOptions$WindmillServerStubFactory.create (StreamingDataflowWorkerOptions.java:183)
  at org.apache.beam.runners.dataflow.worker.options.StreamingDataflowWorkerOptions$WindmillServerStubFactory.create (StreamingDataflowWorkerOptions.java:169)
  at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:592)
  at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:533)
  at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:158)
  at com.sun.proxy.$Proxy54.getWindmillServerStub (Unknown Source)
  at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.&lt;init&gt; (StreamingDataflowWorker.java:677)
  at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.fromDataflowWorkerHarnessOptions (StreamingDataflowWorker.java:562)
  at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.main (StreamingDataflowWorker.java:274)
  Caused by: java.lang.RuntimeException: Loading windmill_service failed:
  at org.apache.beam.runners.dataflow.worker.windmill.WindmillServer.&lt;clinit&gt; (WindmillServer.java:42)
  Caused by: java.io.IOException: No space left on device
  at sun.nio.ch.FileDispatcherImpl.write0 (Native Method)
  at sun.nio.ch.FileDispatcherImpl.write (FileDispatcherImpl.java:60)
  at sun.nio.ch.IOUtil.writeFromNativeBuffer (IOUtil.java:93)
  at sun.nio.ch.IOUtil.write (IOUtil.java:65)
  at sun.nio.ch.FileChannelImpl.write (FileChannelImpl.java:211)
  at java.nio.channels.Channels.writeFullyImpl (Channels.java:78)
  at java.nio.channels.Channels.writeFully (Channels.java:101)
  at java.nio.channels.Channels.access$000 (Channels.java:61)
  at java.nio.channels.Channels$1.write (Channels.java:174)
  at java.nio.file.Files.copy (Files.java:2909)
  at java.nio.file.Files.copy (Files.java:3027)
  at org.apache.beam.runners.dataflow.worker.windmill.WindmillServer.&lt;clinit&gt; (WindmillServer.java:39)
  </code></pre>
  <p>Seems like there is no space left on a device, but shouldn't this be managed by Google? Or is this an error in my job somehow?</p>
  <p>UPDATE:
  The workflow is as follows:</p>
  <ul>
  <li>Reading mass data from PubSub (up to 1500/s)</li>
  <li>Filter some messages</li>
  <li>Keeping session window on key and grouping by it</li>
  <li>Sort the data and do calculations</li>
  <li>Output the data to another PubSub</li>
  </ul>