
layout: blog_detail
title: 'Model Serving in PyTorch'
author: Jeff Smith
redirect_from: /2019/05/08/model-serving-in-pyorch.html

PyTorch has seen a lot of adoption in research, but people can get confused about how well PyTorch models can be taken into production. This blog post is meant to clear up any confusion people might have about the road to production in PyTorch.
Usually when people talk about taking a model “to production,” they usually mean performing inference, sometimes called model evaluation or prediction or serving. At the level of a function call, in PyTorch, inference looks something like this:

In Python
module(input)


In traced modules
module(input)


In C++
at::Tensor output = module->forward(inputs).toTensor();



Since we at Facebook perform inference operations using PyTorch hundreds of trillions of times per day, we've done a lot to make sure that inference runs as efficiently as possible.
Serving Strategies