title: Python best practice for interacting with blocking loop via REPL
body: <p>My program involves a blocking loop. I'd like to use the a python REPL to modify program state while the program is running.</p>
  <h2>Context:</h2>
  <p>Often I have a blocking loop in my python code:</p>
  <pre><code># main.py
  import itertools
  import time
  global_state = 123
  
  if __name__ == &quot;__main__&quot;:
      print(&quot;Starting main loop&quot;)
      m = 0
      for n in itertools.count():
          time.sleep(1)  # &quot;computation&quot;
          global_state += 1
          m += 10
          print(f&quot;{n=}, {m=}, {global_state=}&quot;)
  </code></pre>
  <p>When I run this program at the command line, I get something like this:</p>
  <pre><code>$ python -i main.py
  Starting main loop
  n=0, m=10, global_state=124
  n=1, m=20, global_state=125
  n=2, m=30, global_state=126
  ...
  </code></pre>
  <p>The loop will run for hours or days. Sometimes, while the loop is running, I desire to interactively modify some program state. For example, I'd like to set <code>global_state = -1000</code> or <code>m = 75</code>. But the <code>loop</code> function is blocking...</p>
  <h2>Approaches to interactive modification of program state</h2>
  <h3>Approach 1: Stopping and restarting the loop</h3>
  <p>I can use a <code>KeyboardInterrupt</code> to stop the loop. Since I have invoked <code>python</code> with the <code>-i</code> interactive flag, I can then modify the global state at the interactive REPL. I then restart the loop. This is a simple approach, but it has drawbacks:</p>
  <ul>
  <li>If the loop modifies the global state (or has any side effects), the global state might be left in an inconsistent state when KeyboardInterrupt is raised; the exception could be raised at any point in the loop's control flow, and handling this exception properly can be tricky, especially if the business logic is complex.</li>
  <li>When restarting the loop, the counter <code>n</code> will be reset to zero. Other state variables used by the loop (e.g. <code>m</code>) might also be reset or be in an inconsistent state.</li>
  </ul>
  <h3>Approach 2: Using threading</h3>
  <p>I can wrap the loop in a function and use <code>threading</code> to run the function in another thread:</p>
  <pre><code>def loop():
      print(&quot;Starting main loop&quot;)
      m = 0
      for n in itertools.count():
          time.sleep(1)  # &quot;computation&quot;
          global_state += 1
          m += 10
          print(f&quot;{n=}, {m=}, {global_state=}&quot;)
  import threading
  thread = threading.Thread(target=loop)
  thread.start()
  </code></pre>
  <p>When I invoke <code>python -i main.py</code>, I can use the REPL to interact with global state as the loop runs in another thread. The <code>loop</code> function still <code>print</code>s to <code>stdout</code> as before. The drawbacks of this approach are:</p>
  <ul>
  <li>I can no longer interact with the loop's local state (e.g. I can't modify <code>m</code>) because the local state is wrapped in a function. This is a major drawback.</li>
  <li>Using threading makes the program more fragile; threads can crash, and exceptions raised in threads need to be handled carefully. If the thread blocks or crashes or hangs, it can be hard to recover.</li>
  <li>Stopping the loop becomes more difficult, because KeyboardInterrupt is no longer an option (instead I need to &quot;cooperate&quot; with the thread, sending it some signal that it should self-terminate).</li>
  </ul>
  <h3>Approach 3: Drop into REPL from within the loop</h3>
  <p>I can cooperate with the loop to drop into the repl:</p>
  <pre><code>import os
  import code
  if __name__ == &quot;__main__&quot;:
      ...
      for n in itertools.count():
          ...  # business logic
          if os.stat(&quot;use_repl.txt&quot;).st_size &gt; 0:  # file nonempty             
              code.interact(local=locals())
  </code></pre>
  <p>Here we are using a file to signal to the loop when we want to modify state. If the &quot;use_repl.txt&quot; file is nonempty then loop then calls <code>code.interact</code> to create an interactive Console. This has the added benefit of working even if the loop is wrapped in a function; local variables are loaded into the interactive console.
  Drawbacks:</p>
  <ul>
  <li>Reading a &quot;use_repl.txt&quot; file at every pass through the loop feels a bit clunky. But how else would one signal (cooperatively) to the loop that it should create a REPL?</li>
  <li>The loop gets paused when <code>code.interact</code> is called. This differs from the <code>threading</code>-based solution, where the loop runs continuously. This may be an advantage or a disadvantage, depending on your use-case.</li>
  <li>The REPL created by <code>code.interact</code> is not as polished as python's native interactive mode. For example, tab-completion no longer works.</li>
  </ul>
  <p>An alternative to calling <code>code.interact</code> would be to call the <code>breakpoint</code> built-in function. Another alternative would be to invoke the <code>break</code> keyword, exiting the loop. One could then modify state and restart the loop (as in approach 1), without having to worry that a KeyboardInterrupt has created an inconsistent state.</p>
  <h2>The Question(s):</h2>
  <p>Are there any approaches that I've missed? Each of the approaches considered has significant drawbacks... Are there any best-practices regarding achievement of the desired result?</p>