title: Fastest way to evaluate Tensorflow model in production environment (under C# if possible)
body: <p>Consider I have already trained model which is saved using <code>tf.train.Saver()</code>. Now I would like to use the model in a production environment which is in C#. I was already able to use <a href="https://github.com/migueldeicaza/TensorFlowSharp" rel="nofollow noreferrer">TensorFlowSharp</a> and load (which is a little hack described <a href="https://stackoverflow.com/a/37671613/1815451">here</a>) the model using Saver tensors into TensorFlowSharp and execute it well. </p>
  
  <p>In a batch evaluation, I get fast and quite expected performance results per case which is like 10-20 microseconds. Problem is that in the production environment the model must predict sample by sample (online usage). When calling prediction for a single vector, I get average execution time around 1 millisecond, which is much slower. </p>
  
  <p>When comparing to <a href="https://github.com/Microsoft/CNTK" rel="nofollow noreferrer">CNTK</a> models, which gives evaluation execution time in microseconds even for single calls, this seems quite unfortunate. </p>
  
  <p>I blame calling session.GetRunner().Run() with passing new input each time. I would like to ask if there is some way how to execute/evaluate Tensorflow graph in such environment in the fastest way possible, considering I don't know inputs ahead (so fast, online evaluation). Is this issue only related to C# wrapper? Would it help if I would use C/C++ api? Or is it a problem in the session-runner calls? </p>
  
  <p>I have compiled binaries, Also compiling TFSharp for myself with allowed unsafe code. </p>
  
  <p>Thank you</p>