<p>I'm trying to debug a deadlock in a multi-threaded Python application after it has locked up.  Is there a way to attach a debugger to inspect the state of the process?  </p>

<p>Edit: I'm attempting this on Linux, but it would be great if there were a cross-platform solution.  It's Python after all :)</p>

[{'answer_id': 47744, 'author': 'Henk', 'author_id': 4613, 'author_profile': 'https://Stackoverflow.com/users/4613', 'pm_score': 0, 'selected': False, 'text': '<p>What platform are you attempting this on? Most debuggers allow you to attach to a running process by using the process id.\nYou can either output the process id via logging or using something like Task Manager.\nOnce that is achieved it will be possible to inspect individual threads and their call stacks.</p>\n\n<p>EDIT: I don\'t have any experience with GNU Debugger (GDB), which is cross platform, however I found this <a href="http://wiki.python.org/moin/DebuggingWithGdb" rel="nofollow noreferrer">link</a> and it may start you on the right path. It explains how to add debug symbols (handy for reading stack traces) and how to instruct gdb to attach to a running python process.</p>\n'}, {'answer_id': 56510, 'author': 'asksol', 'author_id': 5577, 'author_profile': 'https://Stackoverflow.com/users/5577', 'pm_score': 5, 'selected': True, 'text': '<p>Yeah, gdb is good for lower level debugging.</p>\n\n<p>You can change threads with the <em>thread</em> command.</p>\n\n<p>e.g</p>\n\n<pre><code>(gdb) thr 2\n[Switching to thread 2 (process 6159 thread 0x3f1b)]\n(gdb) backtrace\n....\n</code></pre>\n\n<p>You could also check out Python specific debuggers like <a href="http://winpdb.org/about/" rel="noreferrer">Winpdb</a>, or <a href="http://bashdb.sourceforge.net/pydb/" rel="noreferrer">pydb</a>. Both platform independent. </p>\n'}, {'answer_id': 59179, 'author': 'Bartosz Radaczyński', 'author_id': 985, 'author_profile': 'https://Stackoverflow.com/users/985', 'pm_score': 2, 'selected': False, 'text': '<p>If you mean the pydb, there is no way to do it. There was some effort in that direction:\n<a href="http://svn.python.org/projects/sandbox/trunk/pdb/mpdb.py" rel="nofollow noreferrer">see the svn commit</a>, but it was abandoned. Supposedly <a href="http://winpdb.org/docs/launch-time/" rel="nofollow noreferrer">winpdb supports it</a>.</p>\n'}, {'answer_id': 277800, 'author': 'Community', 'author_id': -1, 'author_profile': 'https://Stackoverflow.com/users/-1', 'pm_score': 2, 'selected': False, 'text': '<p>My experience debugging multi-threaded programs in PyDev (Eclipse on Windows XP) is, threads created using thread.start_new_thread could not be hooked, but thread created using threading.Thread could be hooked. Hope the information is helpful.</p>\n'}, {'answer_id': 550795, 'author': 'Peter S Magnusson', 'author_id': 31024, 'author_profile': 'https://Stackoverflow.com/users/31024', 'pm_score': 3, 'selected': False, 'text': '<p>You can attach a debugger to a multi-threaded Python process, but you need to do it at the C level.  To make sense of what\'s going on, you need the Python interpreter to be compiled with symbols.  If you don\'t have one, you need to download source from python.org and build it yourself:</p>\n\n<pre><code>./configure --prefix=/usr/local/pydbg\nmake OPT=-g\nsudo make install\nsudo ln -s /usr/local/pydbg/bin/python /usr/local/bin/dbgpy\n</code></pre>\n\n<p>Make sure your workload is running on that version of the interpreter.  You can then attach to it with GDB at any time.  The Python folks have included a sample ".gdbinit" in their Misc directory, which has some useful macros.  However it\'s broken for multi-threaded debugging (!).  You need to replace lines like this</p>\n\n<pre><code>while $pc &lt; Py_Main || $pc &gt; Py_GetArgcArgv\n</code></pre>\n\n<p>with the following:</p>\n\n<pre><code>while ($pc &lt; Py_Main || $pc &gt; Py_GetArgcArgv) &amp;&amp; ($pc &lt; t_bootstrap || $pc &gt; thread_PyThread_start_new_thread)\n</code></pre>\n\n<p>Otherwise commands like <code>pystack</code> won\'t terminate on threads other than the main thread.  With this stuff in place, you can do stuff like</p>\n\n<pre><code>gdb&gt; attach &lt;PID&gt;\ngdb&gt; info threads\ngdb&gt; thread &lt;N&gt;\ngdb&gt; bt\ngdb&gt; pystack\ngdb&gt; detach\n</code></pre>\n\n<p>and see what\'s going on.  Kind of.</p>\n\n<p>You can parse what the objects are with the "pyo" macro.  <a href="http://chrismiles.livejournal.com/20226.html" rel="noreferrer">Chris has some examples</a> on his blog.</p>\n\n<p>Good luck.</p>\n\n<p>(Shoutout for <a href="http://mcfunley.com/cs/blogs/dan/archive/2007/10/19/1530.aspx" rel="noreferrer">Dan\'s blog</a> for some key information for me, notably the threading fix!)</p>\n'}, {'answer_id': 553633, 'author': 'nosklo', 'author_id': 17160, 'author_profile': 'https://Stackoverflow.com/users/17160', 'pm_score': 4, 'selected': False, 'text': '<p>Use <a href="http://winpdb.org/" rel="nofollow noreferrer">Winpdb</a>. It is a <strong>platform independent</strong> graphical GPL Python debugger with support for remote debugging over a network, multiple threads, namespace modification, embedded debugging, encrypted communication and is up to 20 times faster than pdb.</p>\n\n<p>Features:</p>\n\n<ul>\n<li>GPL license. Winpdb is Free Software.</li>\n<li>Compatible with CPython 2.3 through 2.6 and Python 3000</li>\n<li>Compatible with wxPython 2.6 through 2.8</li>\n<li>Platform independent, and tested on Ubuntu Gutsy and Windows XP.</li>\n<li>User Interfaces: rpdb2 is console based, while winpdb requires wxPython 2.6 or later.</li>\n</ul>\n\n<p><a href="https://i.stack.imgur.com/HKYjO.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HKYjO.jpg" alt="Screenshot"></a><br>\n<sub>(source: <a href="http://winpdb.org/images/screenshot_winpdb_small.jpg" rel="nofollow noreferrer">winpdb.org</a>)</sub>  </p>\n'}, {'answer_id': 29093342, 'author': 'Tarnay Kálmán', 'author_id': 55267, 'author_profile': 'https://Stackoverflow.com/users/55267', 'pm_score': 0, 'selected': False, 'text': '<p><a href="https://github.com/dcramer/pdbinject" rel="nofollow">pdbinject</a> allows you to inject pdb into an already running python process.</p>\n\n<p>The pdbinject executable only works under python2, but can inject into python3 just fine too. </p>\n'}, {'answer_id': 29354135, 'author': 'Dmitry Trofimov', 'author_id': 429873, 'author_profile': 'https://Stackoverflow.com/users/429873', 'pm_score': 1, 'selected': False, 'text': '<p><a href="https://www.jetbrains.com/pycharm/" rel="nofollow">PyCharm IDE</a> allows attaching to a running Python process since version 4.0. </p>\n\n<p><a href="https://www.jetbrains.com/pycharm/help/attaching-to-process.html" rel="nofollow">Here</a> is described how to do that.</p>\n'}, {'answer_id': 59023050, 'author': 'Błażej Michalik', 'author_id': 2146491, 'author_profile': 'https://Stackoverflow.com/users/2146491', 'pm_score': 0, 'selected': False, 'text': '<p>This can be used as a dead simple "remote" debugger:</p>\n\n<pre class="lang-py prettyprint-override"><code>import sys\nimport socket\nimport pdb\n\ndef remote_trace():\n    server = socket.socket()\n    server.bind((\'0.0.0.0\', 12345))\n    server.listen()\n    client, _= server.accept()\n    stream = client.makefile(\'rw\')\n    sys.stdin = sys.stdout = sys.stderr = stream\n    pdb.set_trace()\n\nremote_trace()\n\n# Execute in the shell: `telnet 127.0.0.1 12345`\n</code></pre>\n\n<p><em>On Windows it\'s easier to use Netcat instead of Telnet (which will also work on linux).</em></p>\n'}, {'answer_id': 71794451, 'author': 'gobenji', 'author_id': 1327601, 'author_profile': 'https://Stackoverflow.com/users/1327601', 'pm_score': 1, 'selected': False, 'text': '<p>python3 provides <a href="https://devguide.python.org/gdb/" rel="nofollow noreferrer">gdb extensions</a>. Using\nthem, gdb can attach to a running program, select a thread and print its\npython backtrace.</p>\n<p>On Debian (since at least Buster) the extensions are part of the\npython3.<em>x</em>-dbg package (ex. <code>python3.10-dbg</code> installs\n<code>/usr/share/gdb/auto-load/usr/bin/python3.10-gdb.py</code>) and gdb auto-loads them.</p>\n<p>Example with a simple threaded python script:</p>\n<pre><code>#!/usr/bin/env python3\n\nimport signal\nimport threading\n\ndef a():\n    while True:\n        pass\n\ndef b():\n    while True:\n        signal.pause()\n\nthreading.Thread(target=a).start()\nthreading.Thread(target=b).start()\n</code></pre>\n<p>Running gdb:</p>\n<pre><code>user@vsid:~$ ps -C python3 -L\n    PID     LWP TTY          TIME CMD\n   1215    1215 pts/0    00:00:00 python3\n   1215    1216 pts/0    00:00:19 python3\n   1215    1217 pts/0    00:00:00 python3\nuser@vsid:~$ gdb -p 1215\nGNU gdb (Debian 10.1-2+b1) 10.1.90.20210103-git\nCopyright (C) 2021 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;\n[...]\n(gdb) info auto-load python-scripts\nLoaded  Script\nYes     /usr/share/gdb/auto-load/usr/bin/python3.10-gdb.py\n(gdb) info threads\n  Id   Target Id                                  Frame\n* 1    Thread 0x7f2f034b4740 (LWP 1215) &quot;python3&quot; 0x00007f2f036a60fa in __futex_abstimed_wait_common64 (futex_word=futex_word@entry=0x7f2ef4000b60, expected=expected@entry=0,\n    clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=&lt;optimized out&gt;,\n    cancel=cancel@entry=true) at ../sysdeps/nptl/futex-internal.c:74\n  2    Thread 0x7f2f02ea7640 (LWP 1216) &quot;python3&quot; 0x000000000051b858 in _PyEval_EvalFrameDefault\n    (tstate=&lt;optimized out&gt;, f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;)\n    at ../Python/ceval.c:3850\n  3    Thread 0x7f2f026a6640 (LWP 1217) &quot;python3&quot; 0x00007f2f036a3932 in __libc_pause ()\n    at ../sysdeps/unix/sysv/linux/pause.c:29\n(gdb) thread 2\n(gdb) py-bt\nTraceback (most recent call first):\n  File &quot;/root/./threaded.py&quot;, line 7, in a\n    while True:\n  File &quot;/usr/lib/python3.10/threading.py&quot;, line 946, in run\n    self._target(*self._args, **self._kwargs)\n  File &quot;/usr/lib/python3.10/threading.py&quot;, line 1009, in _bootstrap_inner\n    self.run()\n  File &quot;/usr/lib/python3.10/threading.py&quot;, line 966, in _bootstrap\n    self._bootstrap_inner()\n(gdb)\n</code></pre>\n<p>We can confirm that thread <code>1216</code> which used the most cpu time according to\n<code>ps</code> is indeed the thread running function <code>a()</code> that is busy-looping.</p>\n'}]
['https://Stackoverflow.com/questions/47701', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/4883/']