title: Integration Testing Multiple Celery Workers and a DB Backed Django API
body: <p>I'm working with a Software Oriented Architecture that has multiple celery workers (let's call them <code>worker1</code>, <code>worker2</code>, and <code>worker3</code>). All three workers are separate entities (i.e., separate code bases, separate repos, separate celery instances, separate machines) and none of them are connected to a Django app. </p>
  
  <p>Communicating with each of these three workers is a Django based, MySQL backed RESTful API. </p>
  
  <p>In development, these services are all on a vagrant box, each acting as a separate machine running off of a separate port. We have a single RabbitMQ broker for all of the Celery tasks.</p>
  
  <p>A typical path through these services might look something like this: <code>worker1</code> gets a message from a device, does some processing, queues up a task on <code>worker2</code>, which does further processing and makes a POST to the <code>API</code>, which writes to the MySQL DB and triggers a task on <code>worker3</code>, which does some other processing and makes another POST to the <code>API</code> which results in a MySQL write. </p>
  
  <p>The services are communicating nicely, but it's very annoying to test this flow every time we make a change to any service. I really want to get some full integration tests (i.e., starting at a message sent to <code>worker1</code> and going through the entire chain) in place but I'm not sure where to start. The main problems I'm facing are these:</p>
  
  <p>If I queue up something on <code>worker1</code>, how can I possibly tell when the whole flow is over? How can I make reasonable assertions about results when I don't know if the results have even arrived?</p>
  
  <p>How do I deal with DB set up/tear down? I want to delete all of the entries made during a test at the end of each test, but if I'm starting the test from outside of the Django app, I'm not sure how to efficiently clear it out. Manually deleting it and recreating it after every test seems like it might be too much overhead. </p>