title: Dataflow Pipeline - "Processing stuck in step <STEP_NAME> for at least <TIME> without outputting or completing in state finish..."
body: <p>The Dataflow pipelines developed by my team suddenly started getting stuck, stopping processing our events. Their worker logs became full of warning messages saying that one specific step got stuck. The peculiar thing is that the steps that are failing are different, one is a BigQuery output and another for Cloud Storage output.</p>
  
  <p>The following are the log messages that we are receiving:</p>
  
  <p>For BigQuery output:</p>
  
  <pre><code>Processing stuck in step &lt;STEP_NAME&gt;/StreamingInserts/StreamingWriteTables/StreamingWrite for at least &lt;TIME&gt; without outputting or completing in state finish
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:765)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:829)
    at org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:131)
    at org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:103)
    at org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source)
  </code></pre>
  
  <p>For Cloud Storage output:</p>
  
  <pre><code>Processing stuck in step &lt;STEP_NAME&gt;/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles for at least &lt;TIME&gt; without outputting or completing in state process
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:421)
    at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
    at org.apache.beam.sdk.io.FileBasedSink$Writer.close(FileBasedSink.java:1007)
    at org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn.processElement(WriteFiles.java:726)
    at org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn$DoFnInvoker.invokeProcessElement(Unknown Source)
  </code></pre>
  
  <p>All applications have been drained and redeployed  but the same thing happened after a while (period of 3 to 4 hours). Some of them were running for more than 40 days and they suddenly got into this without any changes in the code.</p>
  
  <p>I would like ask for some help to know the reason of this problem.
  These are the following ids of some of the Dataflow jobs with those problems:</p>
  
  <p>Stuck in BigQuery output: <strong>2019-03-04_04_46_31-3901977107649726570</strong></p>
  
  <p>Stuck in Cloud Storage output: <strong>2019-03-04_07_50_00-10623118563101608836</strong></p>