Loading or building cuda engine crashes occassionaly after upgrading to TensorRT 7
I'm trying to run TensorRT inference in C++. Sometimes the code crashes when trying to build a new engine or load the engine from the file. It happens occasionally (sometimes it runs without any problem). I follow the below steps to prepare network:
initLibNvInferPlugins(&amp;gLogger.getTRTLogger(), &quot;&quot;);

if (mParams.loadEngine.size() &gt; 0)
{
    std::vector&lt;char&gt; trtModelStream;
    size_t size{0};
    std::ifstream file(mParams.loadEngine, std::ios::binary);

    if (file.good())
    {
        file.seekg(0, file.end);
        size = file.tellg();
        file.seekg(0, file.beg);
        trtModelStream.resize(size);
        file.read(trtModelStream.data(), size);
        file.close();
    }
    IRuntime* infer_Runtime = nvinfer1::createInferRuntime(gLogger);
    if (mParams.dlaCore &gt;= 0)
    {
        infer_Runtime-&gt;setDLACore(mParams.dlaCore);
    }


    mEngine = std::shared_ptr&lt;nvinfer1::ICudaEngine&gt;(
                infer_Runtime-&gt;deserializeCudaEngine(trtModelStream.data(), size, nullptr), samplesCommon::InferDeleter());

    gLogInfo &lt;&lt; &quot;TRT Engine loaded from: &quot; &lt;&lt; mParams.loadEngine &lt;&lt; endl;

    infer_Runtime-&gt;destroy();
    if (!mEngine)
    {
        return false;
    }
    else
    {
        return true;
    }
}

auto builder = SampleUniquePtr&lt;nvinfer1::IBuilder&gt;(nvinfer1::createInferBuilder(gLogger.getTRTLogger()));


const auto explicitBatch = 1U &lt;&lt; static_cast&lt;uint32_t&gt;(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
auto network = SampleUniquePtr&lt;nvinfer1::INetworkDefinition&gt;(builder-&gt;createNetworkV2(explicitBatch));
auto config = SampleUniquePtr&lt;nvinfer1::IBuilderConfig&gt;(builder-&gt;createBuilderConfig());
auto parser = SampleUniquePtr&lt;nvonnxparser::IParser&gt;(nvonnxparser::createParser(*network, gLogger.getTRTLogger()));

mEngine = nullptr;

parser-&gt;parseFromFile(
        locateFile(mParams.onnxFileName, mParams.dataDirs).c_str(), static_cast&lt;int&gt;(gLogger.getReportableSeverity()));

// Calibrator life time needs to last until after the engine is built.
std::unique_ptr&lt;IInt8Calibrator&gt; calibrator;

config-&gt;setAvgTimingIterations(1);
config-&gt;setMinTimingIterations(1);
config-&gt;setMaxWorkspaceSize(4_GiB);
builder-&gt;setMaxBatchSize(mParams.batchSize);


mEngine = std::shared_ptr&lt;nvinfer1::ICudaEngine&gt;(
            builder-&gt;buildEngineWithConfig(*network, *config), samplesCommon::InferDeleter());


The error occurs here:
[05/12/2021-16:46:42] [I] [TRT] Detected 1 inputs and 1 output network tensors.

16:46:42: The program has unexpectedly finished.

This line crashes when loading existing engine:
mEngine = std::shared_ptr&lt;nvinfer1::ICudaEngine(
    infer_Runtime-&gt;deserializeCudaEngine(trtModelStream.data(), size, nullptr), samplesCommon::InferDeleter());

Or when building the engine:
mEngine = std::shared_ptr&lt;nvinfer1::ICudaEngine&gt;(
            builder-&gt;buildEngineWithConfig(*network, *config), samplesCommon::InferDeleter()); 

More info:

TensorRT 7.2.3 
Ubuntu 18.04 
cuDNN 8.1.1 
CUDA 11.1 update1 
ONNX 1.6.0 
Pytorch 1.5.0 


Finally got it!
I rewrote the CMake.txt and add all required libs and paths and removed duplicate ones. That might be a lib conflict in cuBLAS.
