title: Writing Avro data to BigQuery
body: <p>Using <code>Confluent.Kafka</code> library, I've created a .NET Kafka consumer, which reads data from a topic and deserializes Avro data to .NET classes.</p>
  
  <p>Now, I'm trying to serialize it back to Avro format and send it to BigQuery. I expect BigQuery to automatically generate table structure based on Avro schema.</p>
  
  <p>I set my consumer deserializer like this(AvroGeneratedDto - .NET class that represents avro schema):</p>
  
  <pre><code>            var builder = new ConsumerBuilder&lt;string, AvroGeneratedDto&gt;(
                  consumerConfig
              ).SetValueDeserializer(
                      new SyncOverAsyncDeserializer&lt;AvroGeneratedDto&gt;(
                          new AvroDeserializer&lt;AvroGeneratedDto&gt;(_configProvider.SchemaRegistryClient)));
  </code></pre>
  
  <p>It deserializes data into AvroGeneratedDto OK.</p>
  
  <p>Now, I try to serialize it back using schema stored in schema registry:</p>
  
  <pre><code>var serializer = new AvroSerializer&lt;AvroGeneratedDto&gt;(_configProvider.SchemaRegistryClient);
  var avroSerialized = await serializer.SerializeAsync(item.Value, new SerializationContext(MessageComponentType.Value, topic: "MyTopic"));
  </code></pre>
  
  <p>And upload it to BigQuery:</p>
  
  <pre><code>                    MemoryStream avroStream = new MemoryStream(avroSerialized);
                      var job = await _bigQueryClient.UploadAvroAsync(projectId: projectId, datasetId: ArtsdatasetId, tableId: TableId, schema: null, input: avroStream, options: options);
  </code></pre>
  
  <p>However, I get the following from BigQuery:</p>
  
  <blockquote>
    <p>"<em>Error while reading data, error message: The Apache Avro library failed to parse the header with the following error: Invalid data file. Magic does not match</em>". </p>
  </blockquote>
  
  <p>What is the issue here?</p>