title: BigQuery streaming insert from Dataflow - no results
body: <p>I have a Dataflow pipeline which is reading messages from PubSub Lite and streams data into a BigQuery table. The table is partitioned by day. When querying the table with:</p>
  <pre class="lang-sql prettyprint-override"><code>SELECT * FROM `my-project.my-dataset.my-table` WHERE DATE(timestamp) = &quot;2021-10-14&quot;
  </code></pre>
  <p>The BigQuery UI tells me <code>This query will process 1.9 GB when run</code>. But when actually running the query I don't get any results. My pipeline is running for a whole week now and I am getting the same results for the last two days. However, for <code>2021-10-11</code> and the days before that I am seeing actual results.</p>
  <p>I am currently using Apache Beam version 2.26 and my Dataflow writer looks like this:</p>
  <pre class="lang-java prettyprint-override"><code>return BigQueryIO.&lt;Event&gt;write()
      .withSchema(createTableSchema())
      .withFormatFunction(event -&gt; createTableRow(event))
      .withCreateDisposition(CreateDisposition.CREATE_NEVER)
      .withWriteDisposition(WriteDisposition.WRITE_APPEND)
      .withTimePartitioning(new TimePartitioning().setType(&quot;DAY&quot;).setField(&quot;timestamp&quot;))
      .to(TABLE);
  </code></pre>
  <p>Why is BigQuery taking so long for committing the values to the partitions but at the same time telling me there is actually data available?</p>
  <p>EDIT 1:</p>
  <p><a href="https://i.stack.imgur.com/4KK10.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4KK10.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.stack.imgur.com/xWcFe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xWcFe.png" alt="enter image description here" /></a></p>