PyTorch model + HTTP API = very slow execution
I have:

ML model (PyTorch) that vectorizes data and makes a prediction in ~3.5ms (median ≈ mean)
HTTP API (FastAPI + uvicorn) that serves simple requests in ~2ms

But when I combine them, the median response time becomes almost 200ms.
What can be the reason for such degradation?

Note that:

I also tried aiohttp alone, aiohttp + gunicorn and Flask development server for serving - same result
I tried to send 2, 20 and 100 requests per second - same result
I do realize that parallel requests can lead to decreased latency, but not 30 times!
CPU load is only ~7%


Here's how I measured model performance (I measured the median time separately, it's nearly the same as the mean time):
def predict_all(predictor, data):
    for i in range(len(data)):
        predictor(data[i])

data = load_random_data()
predictor = load_predictor()
%timeit predict_all(predictor, data)
# manually divide total time by number of records in data

Here's FastAPI version:
from fastapi import FastAPI
from starlette.requests import Request
from my_code import load_predictor

app = FastAPI()

app.predictor = load_predictor()


@app.post(&quot;/&quot;)
async def root(request: Request):
    predictor = request.app.predictor
    data = await request.json()
    return predictor(data)

HTTP performance test:
wrk2 -t2 -c50 -d30s -R100 --latency -s post.lua http://localhost:8000/

EDIT.
Here's a slightly modified version which I tried with and without async:
@app.post(&quot;/&quot;)
# async def root(request: Request, user_dict: dict):
def root(request: Request, user_dict: dict):
    predictor = request.app.predictor
    start_time = time.time()
    y = predictor(user_dict)
    finish_time = time.time()
    logging.info(f&quot;user {user_dict['user_id']}: &quot;
                 &quot;prediction made in {:.2f}ms&quot;.format((finish_time - start_time) * 1000))
    return y

So I just added logging of prediction time.
Log for async version:
2021-02-03 11:14:31,822: user 12345678-1234-1234-1234-123456789123: prediction made in 2.87ms
INFO:     127.0.0.1:49284 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,329: user 12345678-1234-1234-1234-123456789123: prediction made in 3.93ms
INFO:     127.0.0.1:49286 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,345: user 12345678-1234-1234-1234-123456789123: prediction made in 15.06ms
INFO:     127.0.0.1:49287 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,351: user 12345678-1234-1234-1234-123456789123: prediction made in 4.78ms
INFO:     127.0.0.1:49288 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,358: user 12345678-1234-1234-1234-123456789123: prediction made in 6.85ms
INFO:     127.0.0.1:49289 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,363: user 12345678-1234-1234-1234-123456789123: prediction made in 3.71ms
INFO:     127.0.0.1:49290 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,369: user 12345678-1234-1234-1234-123456789123: prediction made in 5.49ms
INFO:     127.0.0.1:49291 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:14:56,374: user 12345678-1234-1234-1234-123456789123: prediction made in 5.00ms

So prediction is fast, less than 10ms on average, but whole request takes 200ms.
Log for sync version:
2021-02-03 11:17:58,332: user 12345678-1234-1234-1234-123456789123: prediction made in 65.49ms
2021-02-03 11:17:58,334: user 12345678-1234-1234-1234-123456789123: prediction made in 23.05ms
INFO:     127.0.0.1:49481 - &quot;POST / HTTP/1.1&quot; 200 OK
INFO:     127.0.0.1:49482 - &quot;POST / HTTP/1.1&quot; 200 OK
2021-02-03 11:17:58,338: user 12345678-1234-1234-1234-123456789123: prediction made in 72.39ms
2021-02-03 11:17:58,341: user 12345678-1234-1234-1234-123456789123: prediction made in 78.66ms
2021-02-03 11:17:58,341: user 12345678-1234-1234-1234-123456789123: prediction made in 85.74ms

Now prediction takes long! For whatever reason, exactly the same call, but made in synchronous context, started to take ~30 times longer. But the whole request takes approximately the same time - 160-200ms.

In endpoints that does highly intensive calculations and which presumably takes longer when compared to the other endpoints, use a non-coroutine handler.
When you use def instead of async def, by default FastAPI will use run_in_threadpool from Starlette and which also uses loop.run_in_executor underneath.
run_in_executor will execute the function in the default loops executor, it executes the function in a seperate thread, also you might want to check options like ProcessPoolExecutor and ThreadPoolExecutor if you are doing highly CPU intensive work.
This math simple math helps a lot when working with coroutines.
function
   if function_takes ≥ 500ms
       use `def`
   else
       use `async def`

Making your function non-coroutine should do good.
@app.post(&quot;/&quot;)
def root(request: Request):
    predictor = request.app.predictor
    data = await request.json()
    return predictor(data)

