title: How to run long background processes - Heroku App - Dash Python
body: <p>I'm working on a <strong>Baseball Simulator</strong> app with Dash. It uses a SGD model to simulate gameplay between a lineup and a pitcher. The app (<em><strong>under construction</strong></em>) can be found here: <a href="https://capstone-baseball-simulator.herokuapp.com/" rel="nofollow noreferrer">https://capstone-baseball-simulator.herokuapp.com/</a> and the repo: <a href="https://github.com/c-fried/capstone_heroku" rel="nofollow noreferrer">https://github.com/c-fried/capstone_heroku</a></p>
  <p><strong>To summarize the question:</strong> I want to be able to run the <em>lineup optimizer</em> on the heroku server.</p>
  <p><em>There are potentially two parts to this: 1. Running the actual function while avoiding timeout. &amp; 2. Displaying the progress of the function as it's running.</em></p>
  <hr />
  <p>There are several issues I'm having with solving this:</p>
  <ol>
  <li>The function is expensive and cannot be completed before the 30-second timeout. (It takes several minutes to complete.)</li>
  </ol>
  <p>For this, I attempted to follow these instructions (<a href="https://devcenter.heroku.com/articles/python-rq" rel="nofollow noreferrer">https://devcenter.heroku.com/articles/python-rq</a>) by creating a <code>worker.py</code> (still in the repo), moving the function to the external <code>.py</code>...etc. The problem I believe was that the process still was taking too long and therefore terminating.</p>
  <ol start="2">
  <li>I'm (knowingly) using a <code>global</code> variable in the function which works when I run locally but does not work when deployed (for reasons I somewhat understand - <em>workers don't share memory</em> <a href="https://dash.plotly.com/sharing-data-between-callbacks" rel="nofollow noreferrer">https://dash.plotly.com/sharing-data-between-callbacks</a>)</li>
  </ol>
  <p>I was using a <code>global</code> to be able to see live updates of what the function was doing as it was running. Again, worked as a hack locally, but doesn't work on the server. I don't know how else I can watch the progress of the function without some kind of global operation going on. I'd love a clever solution to this, but I can't think of it.</p>
  <hr />
  <p>I'm not experienced with web apps, so thanks for the advice in advance.</p>