<p dir="auto">Recently, we deployed a ML model with FastAPI, and encountered an issue.</p>
<p dir="auto">The code  looks like this.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from ocr_pipeline.model.ocr_wrapper import OcrWrapper
ocr_wrapper = OcrWrapper(**config.model_load_params) # loads 1.5 GB PyTorch model

...

@api.post('/')
async def predict(file: UploadFile = File(...)):
       preds = ocr_wrapper.predict(file.file, **config.model_predict_params)
       return json.dumps({&quot;data&quot;: preds})"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">ocr_pipeline</span>.<span class="pl-s1">model</span>.<span class="pl-s1">ocr_wrapper</span> <span class="pl-k">import</span> <span class="pl-v">OcrWrapper</span>
<span class="pl-s1">ocr_wrapper</span> <span class="pl-c1">=</span> <span class="pl-v">OcrWrapper</span>(<span class="pl-c1">**</span><span class="pl-s1">config</span>.<span class="pl-s1">model_load_params</span>) <span class="pl-c"># loads 1.5 GB PyTorch model</span>

...

<span class="pl-en">@<span class="pl-s1">api</span>.<span class="pl-en">post</span>(<span class="pl-s">'/'</span>)</span>
<span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">predict</span>(<span class="pl-s1">file</span>: <span class="pl-v">UploadFile</span> <span class="pl-c1">=</span> <span class="pl-v">File</span>(...)):
       <span class="pl-s1">preds</span> <span class="pl-c1">=</span> <span class="pl-s1">ocr_wrapper</span>.<span class="pl-en">predict</span>(<span class="pl-s1">file</span>.<span class="pl-s1">file</span>, <span class="pl-c1">**</span><span class="pl-s1">config</span>.<span class="pl-s1">model_predict_params</span>)
       <span class="pl-k">return</span> <span class="pl-s1">json</span>.<span class="pl-en">dumps</span>({<span class="pl-s">"data"</span>: <span class="pl-s1">preds</span>})</pre></div>
<p dir="auto">The above written command consumes min. 3GB of RAM.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="gunicorn --workers 2 --worker-class=uvicorn.workers.UvicornWorker app.main:api"><pre class="notranslate">gunicorn --workers 2 --worker-class=uvicorn.workers.UvicornWorker app.main:api</pre></div>
<p dir="auto">Is there any way to scale the number of workers without consuming too much RAM?</p>
<p dir="auto">ENVIRONMENT:<br>
Ubuntu 18.04<br>
Python 3.6.9</p>
<p dir="auto">fastapi==0.61.2<br>
uvicorn==0.12.2<br>
gunicorn==20.0.4<br>
uvloop==0.14.0</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/tiangolo/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/tiangolo">@tiangolo</a></p>
<p dir="auto"><strong>Describe the bug</strong><br>
I have deployed FastAPI which queries the database and returns the results. I made sure closing the DB connection and all. I'm running gunicorn with this line ;<br>
<code class="notranslate">gunicorn -w 8 -k uvicorn.workers.UvicornH11Worker -b 0.0.0.0 app:app --timeout 10</code><br>
So after exposing it to the web, I run a load test which makes 30-40 requests in parallel to the fastapi. And the problem starts here. I'm watching the 'HTOP' in the mean time and I see that RAM usage is always growing, seems like no task is killed after completing it's job. Then I checked the Task numbers, same goes for it too, seems like gunicorn workers do not get killed. After some time RAM usage gets at it's maximum, and starts to throw errors. So I killed the gunicorn app but the thing is processes spawned by main gunicorn proces did not get killed and still using all the memory.</p>
<p dir="auto"><strong>Environment:</strong></p>
<ul dir="auto">
<li>
<p dir="auto">OS:  Ubuntu 18.04</p>
</li>
<li>
<p dir="auto">FastAPI Version : 0.38.1</p>
</li>
<li>
<p dir="auto">Python version : 3.7.4</p>
</li>
</ul>