<p dir="auto">I installed TensorFlow version 1.0.0-rc2 on Windows 7 SP1 x64 Ultimate (Python 3.5.2 |Anaconda custom (64-bit)) using:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.0.0rc2-cp35-cp35m-win_amd64.whl"><pre class="notranslate"><code class="notranslate">pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.0.0rc2-cp35-cp35m-win_amd64.whl
</code></pre></div>
<p dir="auto">When I try running the test script from <a href="https://web.archive.org/web/20170214034751/https://www.tensorflow.org/get_started/os_setup#test_the_tensorflow_installation" rel="nofollow">https://web.archive.org/web/20170214034751/https://www.tensorflow.org/get_started/os_setup#test_the_tensorflow_installation</a> in Eclipse 4.5 or in the console:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
print('TensorFlow version: {0}'.format(tf.__version__))
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
print('TensorFlow version: {0}'.format(tf.__version__))
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
</code></pre></div>
<p dir="auto">I obtain some error message:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="TensorFlow version: 1.0.0-rc2
'Hello, TensorFlow!'
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflob
w\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;BestSplits&quot; device_type: &quot;CPU&quot;') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;CountExtremelyRandomStats&quot; device_type: &quot;CPU&quot;') for unknown op: CountExtremelyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;FinishedNodes&quot; device_type: &quot;CPU&quot;') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;GrowTree&quot; device_type: &quot;CPU&quot;') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;ReinterpretStringToFloat&quot; device_type: &quot;CPU&quot;') for unknown op: ReinterpretStringToFloat
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;SampleInputs&quot; device_type: &quot;CPU&quot;') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;ScatterAddNdim&quot; device_type: &quot;CPU&quot;') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;TopNInsert&quot; device_type: &quot;CPU&quot;') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;TopNRemove&quot; device_type: &quot;CPU&quot;') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;TreePredictions&quot; device_type: &quot;CPU&quot;') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: &quot;UpdateFertileSlots&quot; device_type: &quot;CPU&quot;') for unknown op: UpdateFertileSlots"><pre class="notranslate"><code class="notranslate">TensorFlow version: 1.0.0-rc2
'Hello, TensorFlow!'
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflob
w\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "CountExtremelyRandomStats" device_type: "CPU"') for unknown op: CountExtremelyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "FinishedNodes" device_type: "CPU"') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "GrowTree" device_type: "CPU"') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ReinterpretStringToFloat" device_type: "CPU"') for unknown op: ReinterpretStringToFloat
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "SampleInputs" device_type: "CPU"') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ScatterAddNdim" device_type: "CPU"') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNInsert" device_type: "CPU"') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNRemove" device_type: "CPU"') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TreePredictions" device_type: "CPU"') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "UpdateFertileSlots" device_type: "CPU"') for unknown op: UpdateFertileSlots
</code></pre></div>
<p dir="auto">Why?</p>
<p dir="auto">I didn't have such issues with TensorFlow 0.12.1 (installed with <code class="notranslate">pip install tensorflow==0.12.1</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="TensorFlow version: 0.12.1
b'Hello, TensorFlow!'"><pre class="notranslate"><code class="notranslate">TensorFlow version: 0.12.1
b'Hello, TensorFlow!'
</code></pre></div>
<p dir="auto">Stack Exchange thread: <a href="http://stackoverflow.com/q/42217532/395857" rel="nofollow">TensorFlow version 1.0.0-rc2 on Windows: "OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits" with test code</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/drpngx/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/drpngx">@drpngx</a></p>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>:</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: source</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.10.1</li>
<li><strong>Python version</strong>: 3.5</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.15.1</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: 5</li>
<li><strong>CUDA/cuDNN version</strong>: 9.0 / 7.0</li>
<li><strong>GPU model and memory</strong>:</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">I successfully build from source TF 1.10.1 with TensorRT 3.04 support. Using the Python <code class="notranslate">tensorflow.contrib.tensorrt</code> API, I can create a protobuf that contains some TRTEngineOp operations, giving me a substantial inference speed-up.<br>
However, now I need to run this optimized protobuf via a C++ API. I build the required .so libraries for the C++ inference API (such as libtensorflow.so, libtensorflow_cc.so and libtensorflow_framework.so). I can successfully load and run inference on the native TF protobuf, but not on the TRT optimized protobuf.<br>
It gives me an error saying that <code class="notranslate">TRTEngineOp not available.</code> Looking through Stackoverflow, I found some suggestions to use <code class="notranslate">TF_LoadLibrary</code> in the C API to load the <code class="notranslate">python/ops/_trt_engine_op.so</code> available in <code class="notranslate">tensorflow/contrib/tensorrt</code>. When I do this, the <code class="notranslate">TRTEngineOp</code> not available error disappears, but now it complains that it has no registered kernels. I noticed that the <code class="notranslate">python/ops/_trt_engine_op.so</code> Bazel target does not include any kernel definition, which explains why the TF_LoadLibrary doesn't find any kernels I guess. I am wondering how the Python API loads this. Can anyone help? Is it possible to run TensorRT optimized protobufs in C++ ?</p>
<h3 dir="auto">Steps to reproduce</h3>
<ol dir="auto">
<li>Create an optimized protobuf in python, following for example this blog from <a href="https://developers.googleblog.com/2018/03/tensorrt-integration-with-tensorflow.html" rel="nofollow">Google</a></li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Reserve memory for TensorRT inference engine
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction = number_between_0_and_1)
...  
trt_graph = trt.create_inference_graph(
                 input_graph_def = frozen_graph_def, 
                 outputs = output_node_name,
                 max_batch_size=batch_size,
                 max_workspace_size_bytes=workspace_size,
                 precision_mode=precision)  # Get optimized graph"><pre class="notranslate"><code class="notranslate"># Reserve memory for TensorRT inference engine
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction = number_between_0_and_1)
...  
trt_graph = trt.create_inference_graph(
                 input_graph_def = frozen_graph_def, 
                 outputs = output_node_name,
                 max_batch_size=batch_size,
                 max_workspace_size_bytes=workspace_size,
                 precision_mode=precision)  # Get optimized graph
</code></pre></div>
<p dir="auto">then save as protobuf file.</p>
<ol start="2" dir="auto">
<li>Load optimized protobuf in C++, where we have access to the libtensorflow.so, libtensorflow_cc.so and libtensorflow_framework.so precompiled.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#include &lt;tensorflow/core/graph/graph.h&gt;
#include &lt;tensorflow/core/graph/default_device.h&gt;
#include &lt;cuda_runtime.h&gt;
#include tensorflow/c/c_api.h

auto status = TF_NewStatus();
TF_LoadLibrary(getPathToTrtEngineOpSo(), status);

tensorflow::SessionOptions sessionOptions;
sessionOptions.config.set_allow_soft_placement(getAllowSoftPlacement());
sessionOptions.config.set_log_device_placement(getLogDevicePlacement());
auto gpu_options = sessionOptions.config.mutable_gpu_options();
gpu_options-&gt;set_allow_growth(getAllowMemoryGrowth());
gpu_options-&gt;set_per_process_gpu_memory_fraction(getMemoryFraction());
_session.reset(tensorflow::NewSession(sessionOptions));

tensorflow::GraphDef graphDef;
checkStatusTF(tensorflow::ReadBinaryProto(tensorflow::Env::Default(),
                                                  getModel(),
                                                  &amp;graphDef),
                      _logger);
tensorflow::graph::SetDefaultDevice(getDevice(), &amp;graphDef);
checkStatusTF(_session-&gt;Create(graphDef), _logger);"><pre class="notranslate"><code class="notranslate">#include &lt;tensorflow/core/graph/graph.h&gt;
#include &lt;tensorflow/core/graph/default_device.h&gt;
#include &lt;cuda_runtime.h&gt;
#include tensorflow/c/c_api.h

auto status = TF_NewStatus();
TF_LoadLibrary(getPathToTrtEngineOpSo(), status);

tensorflow::SessionOptions sessionOptions;
sessionOptions.config.set_allow_soft_placement(getAllowSoftPlacement());
sessionOptions.config.set_log_device_placement(getLogDevicePlacement());
auto gpu_options = sessionOptions.config.mutable_gpu_options();
gpu_options-&gt;set_allow_growth(getAllowMemoryGrowth());
gpu_options-&gt;set_per_process_gpu_memory_fraction(getMemoryFraction());
_session.reset(tensorflow::NewSession(sessionOptions));

tensorflow::GraphDef graphDef;
checkStatusTF(tensorflow::ReadBinaryProto(tensorflow::Env::Default(),
                                                  getModel(),
                                                  &amp;graphDef),
                      _logger);
tensorflow::graph::SetDefaultDevice(getDevice(), &amp;graphDef);
checkStatusTF(_session-&gt;Create(graphDef), _logger);
</code></pre></div>
<p dir="auto">followed by the appropriate run call.</p>
<p dir="auto">The libtensorflow_cc.so, libtensorflow_framework.so and libtensorflow.so are unmodified targets from Tensorflow repostory r1.10.</p>