title: Kafka validate messages in stateful processing
body: <p>I have an application where multiple users can send REST operations to modify the state of <strong>shared</strong> objects.
  When an object is modified, then multiple actions will happen (DB, audit, logging...).</p>
  
  <p>Not all the operations are valid for example you can not Modify an object after it was Deleted.</p>
  
  <p>Using Kafka I was thinking about the following architecture:</p>
  
  <ol>
  <li>Rest operations are queuing in a Kafka topic. </li>
  <li>Operations to the same object are going to the same partition. So all the object's operations will be in sequence and processed by a consumer</li>
  <li>Consumers are listening to a partition and <strong>validate the operation using an in-memory database</strong></li>
  <li>If the operation was valid then is sent to a "Valid operation topic" otherways is sent to an "Invalid operation topic"</li>
  <li>Other consumers (db, log, audit) are listening to the "Valid operation topic" </li>
  </ol>
  
  <p>I am not very sure about point number 3. 
  I don't like the idea to keep the state of all my objects. (<em>I have billions of objects and even if an object can be of 10mb in size, what I need to store to validate its state is just few Kbytes...</em>)</p>
  
  <p>However, is this a common pattern? Otherwise how can you verify the validity of certain operations? </p>
  
  <p>Also what would do you use as a in-memory database? Surely it has to be highly available, fault-tolerant and support transaction (read and write).</p>