Through searching the space, over different batch sizes (1-32), number of workers (1-16) and devices (CPU,GPU), we have run a set of experiments that summarized the best ones in the table below.


Device 
   
Concurrency 
   
# Requests
   
#workers
   
Batch size
   
Payload/image
   
Optimization 
   
Throughput 
   
Latency P99
   


CPU
   
10
   
1000
   
1
   
1
   
small
   
N/A
   
3.45
   
305.3 ms
   


CPU
   
1
   
1000
   
1
   
1
   
small
   
N/A
   
3.45
   
291.8 ms
   


GPU
   
10
   
1000
   
1
   
1
   
small
   
N/A
   
41.05