Title: Apache Beam Java 2.26.0: BigQueryIO 'No rows present in the request'
Tags: google-bigquery;google-cloud-dataflow;apache-beam
Question: Since the Beam ```2.26.0``` update we ran into errors in our Java SDK streaming data pipelines. We have been investigating the issue for quite some time now but are unable to track down the root cause. When downgrading to ```2.25.0``` the pipeline works as expected.
Our pipelines are responsible for ingestion, i.e., consume from Pub/Sub and ingest into BigQuery. Specifically, we use the ```PubSubIO``` source and the ```BigQueryIO``` sink (streaming mode). When running the pipeline, we encounter the following error:
```{
  &quot;code&quot; : 400,
  &quot;errors&quot; : [ {
    &quot;domain&quot; : &quot;global&quot;,
    &quot;message&quot; : &quot;No rows present in the request.&quot;,
    &quot;reason&quot; : &quot;invalid&quot;
  } ],
  &quot;message&quot; : &quot;No rows present in the request.&quot;,
  &quot;status&quot; : &quot;INVALID_ARGUMENT&quot;
}
```
Our initial guess was that the pipeline's logic was somehow bugged, causing the ```BigQueryIO``` sink to fail. After investigation, we concluded that the ```PCollection``` feeding the sink is indeed containing correct data.
Earlier today I was looking in the changelog and noticed that the ```BigQueryIO``` sink received numerous updates. I was specifically worried about the following changes:

BigQuery’s DATETIME type now maps to Beam logical type org.apache.beam.sdk.schemas.logicaltypes.SqlTypes.DATETIME
Java BigQuery streaming inserts now have timeouts enabled by default. Pass ```--HTTPWriteTimeout=0``` to revert to the old behavior

With respect to the first update, I made sure to disable all ```DATETIME``` in the resulting ```TableRow``` objects. In this specific scenario, the error still stands.
For the second change, I'm unsure how to pass the ```--HTTPWriteTimeout=0``` flag to the pipeline. How is this best achieved?
Any other suggestions as to the root cause of this issue?
Thanks in advance!

Comment: The BigQueryOptions is defined in https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java and can be used like https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/options/PipelineOptions.html. Does setting this flag help?

Comment: This is for the Java SDK, right? I'm guessing so based on the changes you called out, but it might be good to specify at the beginning of the question.

Comment: I did! Initially ignored the issue and assumed it would be fixed in the upcoming version. Issue persists in `2.28.0`.

Comment: Indeed, Java SDK. I Will update the post accordingly.

Comment: Thanks for the responses, gents. I outlined my findings in the answer below.

Comment: Did you try using `2.28` - the latest version of beam?

Here is the accepted answer: We have finally been able to fix this issue and rest assured it has been a hell of a ride. We basically debugged the entire BigQueryIO connector and came to the following conclusions:

The ```TableRow``` objects that are being forwarded to BigQuery used to contain enum values. Due to these not being serializable, an empty payload is forwarded to BigQuery. In my opinion, this error should be made more explicit (why was this suddenly changed anyway?).

The issue was solved by adding the ```@value``` annotation to each enum entry (```com.google.api.client.util.Value```).


The same ```TableRow``` object also contained values of the type ```byte[]```. This value was injected in a BigQuery column with the ```bytes``` type. While this was working without explicitly computing a base64 before, it was now yielding errors.

The issue was solved by computing a base64 ourselves (this setup is also discussed in the following post).




Comment for this answer: Are there changes to Beam that could have made this easier to debug, or possibly the workaround unneeded entirely?
Comment for this answer: Hi Robert, thanks for the reply. The thing that made this particularly hard to debug, was the fact that we did not get an error regarding the `TableRow` not being serializable. I feel like this error should throw an exception that is visible to the user. I'm unsure whether this is part of Beam or an implementation detail of the executor though.

Here is another answer: ```--HTTPWriteTimeout``` is a pipeline option. You can set it the same way you set the runner, etc. (typically on the command line).


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 2, 'CommentCount': 6, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2021:00:03/09/21', 'FavoriteCount': 2.0, 'PostTypeId': 1, 'Tags': '<google-bigquery><google-cloud-dataflow><apache-beam>', 'Title': "Apache Beam Java 2.26.0: BigQueryIO 'No rows present in the request'", 'ViewCount': 454, 'id': 66550087, 'user_AboutMe': '', 'user_AccountId': 4849354.0, 'user_DisplayName': 'LaurensVijnck', 'user_Reputation': 462.0, 'user_Views': 76.0, 'user_id': 3912095.0}}