title: multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?
body: <p>NB. I have seen <a href="https://stackoverflow.com/questions/1501651/log-output-of-multiprocessing-process">Log output of multiprocessing.Process</a> - unfortunately, it doesn't answer this question.</p>
  
  <p>I am creating a child process (on windows) via multiprocessing. I want <em>all</em> of the child process's stdout and stderr output to be redirected to a log file, rather than appearing at the console. The only suggestion I have seen is for the child process to set sys.stdout to a file. However, this does not effectively redirect all stdout output, due to the behaviour of stdout redirection on Windows. </p>
  
  <p>To illustrate the problem, build a Windows DLL with the following code</p>
  
  <pre><code>#include &lt;iostream&gt;
  
  extern "C"
  {
      __declspec(dllexport) void writeToStdOut()
      {
          std::cout &lt;&lt; "Writing to STDOUT from test DLL" &lt;&lt; std::endl;
      }
  }
  </code></pre>
  
  <p>Then create and run a python script like the following, which imports this DLL and calls the function:</p>
  
  <pre><code>from ctypes import *
  import sys
  
  print
  print "Writing to STDOUT from python, before redirect"
  print
  sys.stdout = open("stdout_redirect_log.txt", "w")
  print "Writing to STDOUT from python, after redirect"
  
  testdll = CDLL("Release/stdout_test.dll")
  testdll.writeToStdOut()
  </code></pre>
  
  <p>In order to see the same behaviour as me, it is probably necessary for the DLL to be built against a different C runtime than than the one Python uses. In my case, python is built with Visual Studio 2010, but my DLL is built with VS 2005.</p>
  
  <p>The behaviour I see is that the console shows:</p>
  
  <pre><code>&gt; stdout_test.py
  
  Writing to STDOUT from python, before redirect
  
  Writing to STDOUT from test DLL
  </code></pre>
  
  <p>While the file stdout_redirect_log.txt ends up containing:</p>
  
  <pre><code>Writing to STDOUT from python, after redirect
  </code></pre>
  
  <p>In other words, setting sys.stdout failed to redirect the stdout output generated by the DLL. This is unsurprising given the nature of the underlying APIs for stdout redirection in Windows. I have encountered this problem at the native/C++ level before and never found a way to reliably redirect stdout from within a process. It has to be done externally.</p>
  
  <p>This is actually the very reason I am launching a child process - it's so that I can connect externally to its pipes and thus guarantee that I am intercepting all of its output. I can definitely do this by launching the process manually with pywin32, but I would very much like to be able to use the facilities of multiprocessing, in particular the ability to communicate with the child process via a multiprocessing Pipe object, in order to get progress updates. The question is whether there is any way to both use multiprocessing for its IPC facilities <strong>and</strong> to reliably redirect all of the child's stdout and stderr output to a file.</p>
  
  <p><strong>UPDATE:</strong> Looking at the source code for multiprocessing.Processs, it has a static member, _Popen, which looks like it can be used to override the class used to create the process. If it's set to None (default), it uses a multiprocessing.forking._Popen, but it looks like by saying</p>
  
  <pre><code>multiprocessing.Process._Popen = MyPopenClass
  </code></pre>
  
  <p>I could override the process creation. However, although I could derive this from multiprocessing.forking._Popen, it looks like I would have to copy a bunch of internal stuff into my implementation, which sounds flaky and not very future-proof. If that's the only choice I think I'd probably plump for doing the whole thing manually with pywin32 instead.</p>