title: How to implement Pipelining in Python?
body: <p>I have a program that processes a live video of some markers.</p>
  
  <p>It is divided into:</p>
  
  <ol>
  <li>Import next image of video</li>
  <li>Convert Image to readable form</li>
  <li>Detection of Markers</li>
  <li>Tracking of Markers</li>
  <li>Draw UI</li>
  </ol>
  
  <p>This is working pretty well on my pc but it needs to work on a Raspberry Pi as well, so using just one core the whole time wont cut it.</p>
  
  <p>That's why I want to introduce pipelining.
  In my computer architecture course in the university I learned about hardware pipelining so I was wondering if it would be possible to implement something like that in python:</p>
  
  <p>So instead of doing
  Import -> Conversion -> Processing -> Tracking -> Draw -> ...</p>
  
  <p>I want to do it like this:</p>
  
  <pre><code>-1----2----3----4-----5----...
  Imp--Imp--Imp--Imp---Imp---...
  -----Conv-Conv-Conv--Conv--...
  ----------Pro--Pro---Pro---...
  ---------------Track-Track-...
  ---------------------Draw--...
  </code></pre>
  
  <p>So that every "clock cycle" an image is ready and not only every 5th.</p>
  
  <p>So I was thinking about using the Multiprocessing library of python for this but I have no experience with it but some simple test programs so im not sure what would best suit this use case i.e Queue, Pool, Manager,...  </p>
  
  <h1>SOLVED:</h1>
  
  <p>This can be done with mpipe a cool pipelining tool kit for python.
  [<a href="http://vmlaker.github.io/mpipe/][1]" rel="nofollow noreferrer">http://vmlaker.github.io/mpipe/][1]</a></p>
  
  <pre><code>while True:
      stage1 = mpipe.OrderedStage(conversion, 3)
      stage2 = mpipe.OrderedStage(processing, 3)
      stage3 = mpipe.OrderedStage(tracking, 3)
      stage4 = mpipe.OrderedStage(draw_squares, 3)
      stage5 = mpipe.OrderedStage(ui, 3)
  
      pipe = mpipe.Pipeline(stage1.link(stage2.link(stage3.link(stage4.link(stage5)))))
  
      images = []
      while len(images) &lt; 3:
          ret = False
          while not ret:
              ret, image = cap.read()
          images.append(image)
  
      for i in images:
          t = (i, frame_counter, multi_tracker)
          pipe.put(t)
  
      pipe.put(None)
  
      for result in pipe.results():
          image, multi_tracker, frame_counter = result
          Show.show_win("video", image)
  </code></pre>
  
  <p>As @r_e suggested I read multiple images at the start and fill a pipeline with it. Now in every step of the calculations multiple worker-processes are started so that everyone can work on a separate image.</p>
  
  <p>As some additional information needs to be passed aside from just the image I just return image and the additional information and unpack it in the next stage again.</p>
  
  <p>At the moment I had to disable the tracking so im not able to compare it to the old version. Atm it is a bit slower (tracking would imporve speed as I would not need to detected objects in every frame but only every 30th). But ill give you an update if I get it to work.</p>