title: Synchronizing embedded Python in multi-threaded program
body: <p>Here is the example of using Python interpreter in multi-threaded program:</p>
  
  <pre><code>#include &lt;python.h&gt;
  #include &lt;boost/thread.hpp&gt;
  
  void f(const char* code)
  {
      static volatile auto counter = 0;
      for(; counter &lt; 20; ++counter)
      {
          auto state = PyGILState_Ensure();
          PyRun_SimpleString(code);
          PyGILState_Release(state);
  
          boost::this_thread::yield();
      }
  }
  
  int main()
  {
      PyEval_InitThreads();
      Py_Initialize();
      PyRun_SimpleString("x = 0\n");
      auto mainstate = PyEval_SaveThread();
  
      auto thread1 = boost::thread(f, "print('thread #1, x =', x)\nx += 1\n");
      auto thread2 = boost::thread(f, "print('thread #2, x =', x)\nx += 1\n");
      thread1.join();
      thread2.join();
  
      PyEval_RestoreThread(mainstate);
      Py_Finalize();
  }
  </code></pre>
  
  <p>It looks fine, but it isn't synchronized. Python interpreter releases and reacquires GIL multiple times during PyRun_SimpleString (see <a href="http://docs.python.org/py3k/c-api/init.html#thread-state-and-the-global-interpreter-lock" rel="nofollow">docs, p.#2</a>).</p>
  
  <p>We can serialize PyRun_SimpleString call by using our own synchronization object, but it's a wrong way.</p>
  
  <p>Python has its own synchronization modules - <code>_thread</code> and <code>threading</code>. But they don't work in this code:</p>
  
  <pre><code>Py_Initialize();
  PyRun_SimpleString(R"(
  import _thread
  sync = _thread.allocate_lock()
  
  x = 0
  )");
  
  auto mainstate = PyEval_SaveThread();
  
  auto thread1 = boost::thread(f, R"(
  with sync:
      print('thread #1, x =', x)
      x += 1
  )");
  </code></pre>
  
  <ul>
  <li>it yields an error <code>File "&lt;string&gt;", line 3, in &lt;module&gt; NameError: name '_[1]' is not defined</code> and deadlocks.</li>
  </ul>
  
  <p>How to synchronize embedded python code most efficient way?</p>