title: Kafka to Google Cloud Platform Dataflow ingestion
body: <p>What are the possible options that the Kafka data from the topics can be streamed, consumed and ingested into the BigQuery/Cloud storage. </p>
  
  <p>As per,
  <a href="https://stackoverflow.com/questions/29893342/is-it-possible-to-use-kafka-with-google-cloud-dataflow">is it possible to Use Kafka with Google cloud Dataflow</a></p>
  
  <p>GCP comes with Dataflow which is built on top of Apache Beam programming model. Is KafkaIO use with Beam Pipeline the recommended way to perform for real-time transformations on the incoming data?</p>
  
  <p><a href="https://beam.apache.org/releases/javadoc/2.5.0/org/apache/beam/sdk/io/kafka/KafkaIO.html" rel="nofollow noreferrer">https://beam.apache.org/releases/javadoc/2.5.0/org/apache/beam/sdk/io/kafka/KafkaIO.html</a></p>
  
  <p>Kafka data can be pushed to cloud pub-sub and then onto BigQuery table. Kafka streams/Spark job that would sit out of GCP can also be used. </p>
  
  <p>What are the factors to consider during the design decision given the Data is hosted entirely on Google Cloud Platform (GCP)?</p>