
Batch size can directly affect the latency and the throughput. To better utilize the compute resources batch size needs to be increased. However, there is a tradeoff between latency and throughput. Larger batch sizes can increase the throughput but results in a higher latency as well.  Batch size can be set in Torchserve in two ways, either through model config in config.properties or while registering the model using Management API. 

In the next section, we are going to use Torchserve benchmark suite to decide the best combination of model optimization,  hardware, workers, and batch size. 
Animated Drawings Performance Tuning