title: Running nested routines concurrently in Python 3.5
body: <p>I’m doing I/O operations using pySerial and some self-written post-processing modules. A request is sent to a COM port and the response is then processed and saved to a text file. Everything's works fine, but I have some difficulties running it concurrently.</p>
  
  <p>It simply goes this way:</p>
  
  <pre><code>while True:
      serial_port.action()
      # ^ will send the data to the post-processor
      #                             ^ will send the data to the file exporter
  </code></pre>
  
  <h2>First approach: Threads</h2>
  
  <p>The serial port class inherits from <code>threading.Thread</code> and is started by the main method. A message broker is used by the serial port thread to send the response of the COM device to the post-processor. The post-processor uses the message broker in the same way to forward the data afterwards to the file exporter and so on.</p>
  
  <p>The problem: it’s not concurrently, because the serial port thread could have been working again while the post-processor is still doing its processing stuff.</p>
  
  <h2>Second approach: async/await</h2>
  
  <p>The classes use an asyncronous method <code>action()</code> to do their work. After the data object is processed it is send via the message broker to the (asyncronous) <code>action()</code> method of the next class.</p>
  
  <pre><code># Within a while loop run:
  async def action(self, *args):
      # Do some stuff...
      await asyncio.sleep(1)
  
      # Send the data object to the next processing module.
      # The message broker will call the action() method of the receiver.
      await self._message_next(my_data)
  </code></pre>
  
  <p>Well, that works not how intended. On the first iteration of the loop the data is processed by all classes. On the next iterations the data doesn’t leave the first <code>action()</code> method (of the serial port). Log says:</p>
  
  <pre><code>serial port: action()
  post-processor: action()
  file exporter: action()
  serial port: action()
  serial port: action()
  serial port: action()
  serial port: action()
  serial port: action()
  ...
  </code></pre>
  
  <h2>Third approach: ThreadPoolExecutor</h2>
  
  <p>The next approach was to use a <code>ThreadPoolExecutor</code>. But the result is the same like in the second approach.</p>
  
  <pre><code>executor = ThreadPoolExecutor(1)
  loop = asyncio.get_event_loop()
  
  try:
      asyncio.ensure_future(loop.run_in_executor(executor, serial_port.action()))
      loop.run_forever()
  finally:
      loop.close()
  </code></pre>
  
  <h2>Question</h2>
  
  <p>What is the right approach/design to reach the described goal of doing I/O operations and post-processing stuff concurrently in Python 3.5?</p>
  
  <h1>Revision of Question</h1>
  
  <p>Let’s say we have some routines. One is doing I/O and therefore blocking, whereas the other ones are used for data processing and are non-blocking. How can I achieve that blocking and non-blocking routines run asynchronously? Example:</p>
  
  <pre><code>#!/usr/bin/env python3
  
  import asyncio
  import random
  
  async def foo():
      # blocking I/O
      await asyncio.sleep(random.uniform(0, 0.5))
      print('(1) foo')
  
  async def bar():
      # non-blocking data processing
      await sub_bar_1()
      await sub_bar_2()
  
  async def sub_bar_1():
      await asyncio.sleep(random.uniform(0.5, 2.0))
      print('(2) sub_bar_1')
  
  async def sub_bar_2():
      await asyncio.sleep(random.uniform(0.5, 2.0))
      print('(3) sub_bar_2')
  
  async def main():
      while True:
          await foo()
          await bar()
  
  loop = asyncio.get_event_loop()
  
  try:
      asyncio.ensure_future(main())
      loop.run_forever()
  finally:
      loop.close()
  </code></pre>
  
  <p><code>foo()</code> has to wait until <code>bar()</code> is finished:</p>
  
  <pre><code>(1) foo
  (2) sub_bar_1
  (3) sub_bar_2
  (1) foo
  (2) sub_bar_1
  (3) sub_bar_2
  </code></pre>
  
  <p>How can <code>foo()</code> run independently and <code>bar()</code> is called after every I/O operation?</p>