title: Recommendation needed for performing reconciliation of batch process using messages in kafka
body: <p>I am developing a batch processes using the micro-service architecture. The series of steps that are needed to be performed within a whole batch process will be performed by different micro-services. Kafka is being used as the message bus 
  and all the micro-services interact using kafka.</p>
  
  <p>During the batch process first micro service pulls data from database and then for each record that is pulled it will create a message , which is processed a second micro service , it then again publishes message to the bus.</p>
  
  <p>At the end of the day we need to perform reconciliation of the messages published by the first micro service vs messages published by the last micro service.</p>
  
  <p>Right now trying to perform the reconciliation using the another microservice which uses spring integration kafka to read the messages from these two topics, but this approach seems to be complex for the following reasons:
  1) The starting topic and ending topic will be used my multiple batch process.
     so I will end up in reading the same messages again and again while performing 
     the reconciliation for each process.
  2) Not sure how to aggregate messages read from these two topics and then perform reconciliation ?
  3) Challenges involved in case I need to re perform the reconciliation?</p>
  
  <p>Is Kafka meant for using as store to put messages/data which can be used for reconciliation or I need to look for some other store where I need to push my messages which I can use for performing reconciliation  ? If so what kind of store can I use in this scenario ?</p>
  
  <p>Any pointers would be helpful ?</p>