Cannot Obtain Similar DL Prediction Result in Pytorch C++ API Compared to Python
I have trained a deep learning model using unet architecture in order to segment the nuclei in python and pytorch. I would like to load this pretrained model and make prediction in C++. For this reason, I obtained trace file(with pt extension). Then, I have run this code:

#include &lt;iostream&gt;

#include &lt;torch/script.h&gt; // One-stop header.

#include &lt;iostream&gt;
#include &lt;memory&gt;

#include &lt;opencv2/core/core.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;


using namespace cv;


int main(int argc, const char* argv[]) {


    Mat image;
    image = imread("C:/Users/Sercan/PycharmProjects/samplepyTorch/test_2.png", CV_LOAD_IMAGE_COLOR);

    std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load("C:/Users/Sercan/PycharmProjects/samplepyTorch/epistroma_unet_best_model_trace.pt");
    module-&gt;to(torch::kCUDA);


    std::vector&lt;int64_t&gt; sizes = { 1, 3, image.rows, image.cols };
    torch::TensorOptions options(torch::ScalarType::Byte);
    torch::Tensor tensor_image = torch::from_blob(image.data, torch::IntList(sizes), options);
    tensor_image = tensor_image.toType(torch::kFloat);
    auto result = module-&gt;forward({ tensor_image.to(at::kCUDA) }).toTensor();


    result = result.squeeze().cpu();
    result = at::sigmoid(result);

    cv::Mat img_out(image.rows, image.cols, CV_32F, result.data&lt;float&gt;());

    cv::imwrite("img_out.png", img_out);

}


Image outputs ( First image: test image, Second image: Python prediction result, Third image: C++ prediction result):



As you see, C++ prediction output is not similar to python prediction output. Could you offer a solution to fix this problem?

Even though the question is old it might be useful to some. This answer is based on pytorch 1.5.0 release (and first stable version of C++ frontend), the case might be a little different in previous versions (though 1.4.0+ would work the same IIRC).

PyTorch C++ frontend code


no need to explicitly create torch::TensorOptions object if you only want to specify the type in torch::from_blob. Check Configuring Properties of Tensor in PyTorch notes, this will clear it up further. Basically, you can just use torch::ScalarType::Byte.
This type is equivalent to torch::kUInt8 which is easier to find in the docs IMO
No need to create std::vector object to keep shape as torch::from_blob has it's second argument of type IntArrayRef, which is a typedef for ArrayRef&lt;int64_t&gt; (see ArrayRef documentation). This class, in turn, has multiple overloaded constructors, one of which takes std::initializer_list (which is exactly yours { 1, 3, image.rows, image.cols })


With all that in mind you can create tensor_image in a single line like so (added auto as returned type is IMO obvious and const as it won't be modified further as the type is changed in the same line):

const auto tensor_image =
    torch::from_blob(image.data, {1, 3, image.rows, image.cols},
                     torch::kUInt8)
        .toType(torch::kFloat);


Actual error

OpenCV loads images in BGR (blue-green-red) format, while PyTorch usually uses RGB (say in torchvision in Python). Solution is to permute your image so the colors match. 

Including above change, whole code becomes:

const auto tensor_image =
    torch::from_blob(image.data, {1, 3, image.rows, image.cols},
                     torch::kUInt8)
        .toType(torch::kFloat)
        .permute(0, 3, 2, 1);


And you should be fine now with your predictions. Maybe it would be beneficial to get tensor &gt; 0 instead of sigmoid as it's probably binary classification and there is no need for this operation per se.

Other PyTorch related stuff

There is no need to use at (ATen - as described in docs, foundational tensor and mathematical operation library on which all else is built) namespace anymore as torch:: namespace redirects to it.

Clearer and less confusing options would be:


torch::kCUDA instead of at::kCUDA
torch::sigmoid instead of at::sigmoid


Also .data&lt;T&gt; is deprecated in favor of .data_ptr&lt;T&gt;

All in all you rarely need to use different namespace than torch:: and it's sub-namespaces.
