title: Is apache-beam a good choice when the event time ordering has to preserved when writing to sink?
body: <p>I'm considering using apache beam to write a streaming pipeline to apply a stream of mutations to replicate events from a source database into a destination database in the order of event time. The source could be either kafka or pubsub.</p>
  
  <p>An example would be something like <a href="https://github.com/GoogleCloudPlatform/dynamodb-spanner-migration/blob/master/dataflow/src/main/java/com/example/spanner_migration/SpannerStreamingWrite.java" rel="nofollow noreferrer">this</a> except that the order in which the mutations are applied to the sink must be in order in which they arrived. </p>
  
  <p>I did go over some of the previous questions asked on preserving order:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/45888719/processing-total-ordering-of-events-by-key-using-apache-beam">Processing Total Ordering of Events By Key using Apache Beam </a></li>
  <li><a href="https://stackoverflow.com/questions/55968634/sort-elements-within-a-fixed-window-cloud-dataflow">Sort elements within a fixed window - Cloud Dataflow</a> - This seems to be same use case i'm interested in.</li>
  </ul>
  
  <p>I understand that if I go down the apache beam road i would have to</p>
  
  <ol>
  <li>choose a windowing strategy with accommodation for late data (either a fixed windowing strategy with a allowed lateness or  with global window, have triggers to emit panes and buffer for late data)</li>
  <li>apply transformations </li>
  <li>GroupByKey over a single key(so that everthing goes to the same worker), sort and write to sink</li>
  </ol>
  
  <p>In addition to the above, I would have to make sure the windows(if i follow a fixed window strategy) are executed in order. Step 3 is bound to be the bottleneck.</p>
  
  <p>If [2] above in the list of steps is a lot of computation then apache beam would make sense to take advantage of parallelism which beam offers. But if [2] is just a simple one to one mapping, does apache beam make sense for this replication usecase. Please let me know if i'm missing something.</p>
  
  <p>Note: We do have a batch pipeline on dataflow using apache beam to load a datadump on gcs to database where the entire data is on disk and the order in which its written to sink does not matter.</p>