title: How to properly use the Dataflow / Apache beam wait_until_finish duration parameter?
body: <p>I have a batch processing job running in dataflow on gcp under version apache-beam[gcp]==2.19.0 on the dataflow runner. I created a custom template for the job. The job is running as expected, but I also want to add a max job duration. <a href="https://beam.apache.org/releases/pydoc/2.19.0/apache_beam.runners.runner.html#apache_beam.runners.runner.PipelineResult.wait_until_finish" rel="nofollow noreferrer">I found the duration (in milliseconds) parameter inside the wait_until_finish() method</a>, <a href="https://cloud.google.com/dataflow/docs/resources/release-notes-python#220_december_8_2017" rel="nofollow noreferrer">which should be available</a>. Question is: How to let the templated batch job stop automatically when it runs longer than duration? I do not need to keep any data, I just want the job to stop when it runs too long. I've implemented the run function as follows:</p>
  <pre><code>def run():
      opts = PipelineOptions()
      user_options = opts.view_as(UserOptions)
      p = beam.Pipeline(options=opts)
  
      (p |
       &quot;Read data&quot; &gt;&gt; beam.io.Read(beam.io.BigQuerySource(query=user_options.query,
                                                          use_standard_sql=StaticValueProvider(bool, True))) |
       &quot;Get data&quot; &gt;&gt; beam.ParDo(doStuff()) |
  
       &quot;Output data&quot; &gt;&gt; beam.ParDo(outputData(param1=user_options.input1)) |
  
       &quot;Write to BQ&quot; &gt;&gt; beam.io.WriteToBigQuery(
                  table=user_options.table_spec,
                  schema=user_options.table_schema,
                  write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
                  create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED
              )
       )
  
      result = p.run()
  
      result.wait_until_finish(duration=1800000)
  </code></pre>