title: TensorFlow Inference
body: <p>I've been digging around on this for a while. I have found a ton of articles; but none really show just tensorflow inference as a plain inference.  Its always "use the serving engine" or using a graph that is pre-coded/defined.</p>
  
  <p>Here is the problem: I have a device which occasionally checks for updated models. It then needs to load that model and run input predictions through the model.</p>
  
  <p>In keras this was simple: build a model; train the model and the call model.predict(). In scikit-learn same thing.</p>
  
  <p>I am able to grab a new model and load it; I can print out all of the weights; but how in the world do I run inference against it?</p>
  
  <p>Code to load model and print weights:</p>
  
  <pre><code>    with tf.Session() as sess:
          new_saver = tf.train.import_meta_graph(MODEL_PATH + '.meta', clear_devices=True)
          new_saver.restore(sess, MODEL_PATH)
          for var in tf.trainable_variables():
              print(sess.run(var))
  </code></pre>
  
  <p>I printed out all of my collections and I have:
  ['queue_runners', 'variables', 'losses', 'summaries', 'train_op', 'cond_context', 'trainable_variables']</p>
  
  <p>I tried using <code>sess.run(train_op)</code>; however that just started kicking up a full training session; which is not what I want to do. I just want to run inference against a different set of inputs that I provide which are not TF Records.</p>
  
  <p>Just a little more detail:</p>
  
  <p>The device can use C++ or Python; as long as I can produce a .exe.  I can set up a feed dict if I want to feed the system.  I trained with <code>TFRecords</code>; but in production I'm not going to use <code>TFRecords</code>; its a real/near real time system.</p>
  
  <p>Thanks for any input.  I am posting sample code to this repo: <a href="https://github.com/drcrook1/CIFAR10/TensorFlow" rel="noreferrer">https://github.com/drcrook1/CIFAR10/TensorFlow</a> which does all the training and sample inference.</p>
  
  <p>Any hints are greatly appreciated! </p>
  
  <p>------------EDITS-----------------
  I rebuilt the model to be as below:</p>
  
  <pre><code>def inference(images):
      '''
      Portion of the compute graph that takes an input and converts it into a Y output
      '''
      with tf.variable_scope('Conv1') as scope:
          C_1_1 = ld.cnn_layer(images, (5, 5, 3, 32), (1, 1, 1, 1), scope, name_postfix='1')
          C_1_2 = ld.cnn_layer(C_1_1, (5, 5, 32, 32), (1, 1, 1, 1), scope, name_postfix='2')
          P_1 = ld.pool_layer(C_1_2, (1, 2, 2, 1), (1, 2, 2, 1), scope)
      with tf.variable_scope('Dense1') as scope:
          P_1 = tf.reshape(C_1_2, (CONSTANTS.BATCH_SIZE, -1))
          dim = P_1.get_shape()[1].value
          D_1 = ld.mlp_layer(P_1, dim, NUM_DENSE_NEURONS, scope, act_func=tf.nn.relu)
      with tf.variable_scope('Dense2') as scope:
          D_2 = ld.mlp_layer(D_1, NUM_DENSE_NEURONS, CONSTANTS.NUM_CLASSES, scope)
      H = tf.nn.softmax(D_2, name='prediction')
      return H
  </code></pre>
  
  <p>notice I add the name <code>'prediction'</code> to the TF operation so I can retrieve it later.</p>
  
  <p>When training I used the input pipeline for <code>tfrecords</code> and input queues.</p>
  
  <pre><code>GRAPH = tf.Graph()
  with GRAPH.as_default():
      examples, labels = Inputs.read_inputs(CONSTANTS.RecordPaths,
                                            batch_size=CONSTANTS.BATCH_SIZE,
                                            img_shape=CONSTANTS.IMAGE_SHAPE,
                                            num_threads=CONSTANTS.INPUT_PIPELINE_THREADS)
      examples = tf.reshape(examples, [CONSTANTS.BATCH_SIZE, CONSTANTS.IMAGE_SHAPE[0],
                                       CONSTANTS.IMAGE_SHAPE[1], CONSTANTS.IMAGE_SHAPE[2]])
      logits = Vgg3CIFAR10.inference(examples)
      loss = Vgg3CIFAR10.loss(logits, labels)
      OPTIMIZER = tf.train.AdamOptimizer(CONSTANTS.LEARNING_RATE)
  </code></pre>
  
  <p>I am attempting to use <code>feed_dict</code> on the loaded operation in the graph; however now it is just simply hanging....</p>
  
  <pre><code>MODEL_PATH = 'models/' + CONSTANTS.MODEL_NAME + '.model'
  
  images = tf.placeholder(tf.float32, shape=(1, 32, 32, 3))
  
  def run_inference():
      '''Runs inference against a loaded model'''
      with tf.Session() as sess:
          #sess.run(tf.global_variables_initializer())
          new_saver = tf.train.import_meta_graph(MODEL_PATH + '.meta', clear_devices=True)
          new_saver.restore(sess, MODEL_PATH)
          pred = tf.get_default_graph().get_operation_by_name('prediction')
          rand = np.random.rand(1, 32, 32, 3)
          print(rand)
          print(pred)
          print(sess.run(pred, feed_dict={images: rand}))
          print('done')
  
  run_inference()
  </code></pre>
  
  <p>I believe this is not working because the original network was trained using TFRecords.  In the sample CIFAR data set the data is small; our real data set is huge and it is my understanding TFRecords the the default best practice for training a network.  The <code>feed_dict</code> makes great perfect sense from a productionizing perspective; we can spin up some threads and populate that thing from our input systems.</p>
  
  <p>So I guess I have a network that is trained, I can get the predict operation; but how do I tell it to stop using the input queues and start using the <code>feed_dict</code>?  Remember that from the production perspective I do not have access to whatever the scientists did to make it.  They do their thing; and we stick it in production using whatever agreed upon standard.</p>
  
  <p>-------INPUT OPS--------</p>
  
  <p>tf.Operation 'input/input_producer/Const' type=Const, tf.Operation 'input/input_producer/Size' type=Const, tf.Operation 'input/input_producer/Greater/y' type=Const, tf.Operation 'input/input_producer/Greater' type=Greater, tf.Operation 'input/input_producer/Assert/Const' type=Const, tf.Operation 'input/input_producer/Assert/Assert/data_0' type=Const, tf.Operation 'input/input_producer/Assert/Assert' type=Assert, tf.Operation 'input/input_producer/Identity' type=Identity, tf.Operation 'input/input_producer/RandomShuffle' type=RandomShuffle, tf.Operation 'input/input_producer' type=FIFOQueueV2, tf.Operation 'input/input_producer/input_producer_EnqueueMany' type=QueueEnqueueManyV2, tf.Operation 'input/input_producer/input_producer_Close' type=QueueCloseV2, tf.Operation 'input/input_producer/input_producer_Close_1' type=QueueCloseV2, tf.Operation 'input/input_producer/input_producer_Size' type=QueueSizeV2, tf.Operation 'input/input_producer/Cast' type=Cast, tf.Operation 'input/input_producer/mul/y' type=Const, tf.Operation 'input/input_producer/mul' type=Mul, tf.Operation 'input/input_producer/fraction_of_32_full/tags' type=Const, tf.Operation 'input/input_producer/fraction_of_32_full' type=ScalarSummary, tf.Operation 'input/TFRecordReaderV2' type=TFRecordReaderV2, tf.Operation 'input/ReaderReadV2' type=ReaderReadV2,</p>
  
  <p>------END INPUT OPS-----</p>
  
  <p>----UPDATE 3----</p>
  
  <p>I believe what I need to do is to kill the input section of the graph trained with TF Records and rewire the input to the first layer to a new input.  Its kinda like performing surgery; but this is the only way I can find to do inference if I trained using TFRecords as crazy as it sounds...</p>
  
  <p>Full Graph:</p>
  
  <p><a href="https://i.stack.imgur.com/xFrYN.png" rel="noreferrer"><img src="https://i.stack.imgur.com/xFrYN.png" alt="enter image description here"></a></p>
  
  <p>Section to kill:</p>
  
  <p><a href="https://i.stack.imgur.com/mRpEf.png" rel="noreferrer"><img src="https://i.stack.imgur.com/mRpEf.png" alt="enter image description here"></a></p>
  
  <p>So I think the question becomes: How does one kill the input section of the graph and replace it with a <code>feed_dict</code>?</p>
  
  <p>A follow up to this would be: is this really the right way to do it?  This seems bonkers.</p>
  
  <p>----END UPDATE 3----</p>
  
  <p>---link to checkpoint files---</p>
  
  <p><a href="https://drcdata.blob.core.windows.net/checkpoints/CIFAR_10_VGG3_50neuron_1pool_1e-3lr_adam.model.zip?st=2017-05-01T21%3A56%3A00Z&amp;se=2020-05-02T21%3A56%3A00Z&amp;sp=rl&amp;sv=2015-12-11&amp;sr=b&amp;sig=oBCGxlOusB4NOEKnSnD%2FTlRYa5NKNIwAX1IyuZXAr9o%3D" rel="noreferrer">https://drcdata.blob.core.windows.net/checkpoints/CIFAR_10_VGG3_50neuron_1pool_1e-3lr_adam.model.zip?st=2017-05-01T21%3A56%3A00Z&amp;se=2020-05-02T21%3A56%3A00Z&amp;sp=rl&amp;sv=2015-12-11&amp;sr=b&amp;sig=oBCGxlOusB4NOEKnSnD%2FTlRYa5NKNIwAX1IyuZXAr9o%3D</a></p>
  
  <p>--end link to checkpoint files---</p>
  
  <p>-----UPDATE 4 -----</p>
  
  <p>I gave in and just gave a shot at the 'normal' way of performing inference assuming I could have the scientists simply just pickle their models and we could grab the model pickle; unpack it and then run inference on it.  So to test I tried the normal way assuming we already unpacked it...It doesn't work worth a beans either...</p>
  
  <pre><code>import tensorflow as tf
  import CONSTANTS
  import Vgg3CIFAR10
  import numpy as np
  from scipy import misc
  import time
  
  MODEL_PATH = 'models/' + CONSTANTS.MODEL_NAME + '.model'
  imgs_bsdir = 'C:/data/cifar_10/train/'
  
  images = tf.placeholder(tf.float32, shape=(1, 32, 32, 3))
  
  logits = Vgg3CIFAR10.inference(images)
  
  def run_inference():
  '''Runs inference against a loaded model'''
      with tf.Session() as sess:
          sess.run(tf.global_variables_initializer())
          new_saver = tf.train.import_meta_graph(MODEL_PATH + '.meta')#, import_scope='1', input_map={'input:0': images})
          new_saver.restore(sess, MODEL_PATH)
          pred = tf.get_default_graph().get_operation_by_name('prediction')
          enq = sess.graph.get_operation_by_name(enqueue_op)
          #tf.train.start_queue_runners(sess)
          print(rand)
          print(pred)
          print(enq)
          for i in range(1, 25):
              img = misc.imread(imgs_bsdir + str(i) + '.png').astype(np.float32) / 255.0
              img = img.reshape(1, 32, 32, 3)
              print(sess.run(logits, feed_dict={images : img}))
              time.sleep(3)
          print('done')
  
  run_inference()
  </code></pre>
  
  <p>Tensorflow ends up building a new graph with the inference function from the loaded model; then it appends all the other stuff from the other graph to the end of it.  So then when I populate a <code>feed_dict</code> expecting to get inferences back; I just get a bunch of random garbage as if it were the first pass through the network...</p>
  
  <p>Again; this seems nuts; do I really need to write my own framework for serializing and deserializing random networks?  This has had to have been done before...</p>
  
  <p>-----UPDATE 4 -----</p>
  
  <p>Again; thanks!</p>