title: Inference speed in tensorflow 2.0 (CPU) for simple models? Vanilla tensorflow/numpy is still 50x faster?
body: <p>Am finding that tensorflow keras (2.0) is still quite slow compared to numpy. Is tfdeploy still in use? What are the options for getting close to numpy speed in inference? </p>
  
  <p>UPDATE:</p>
  
  <p>Here is a cleaned up example showing three methods on a different machine. The tf.function wrapped tensorflow call is now 8x vanilla numpy which is possibly reasonable but still feels like there is some overhead to chase.</p>
  
  <pre><code>import tensorflow as tf
  import numpy as np
  
  x = tf.convert_to_tensor(np.random.randn(1, 12).astype(np.float32)) # YES! one row only
  xnumpy = x.numpy()
  model = tf.keras.models.Sequential([
      tf.keras.layers.Dense(16, use_bias=False) for i in range(10)
      ])
  
  @tf.function
  def f(x):
      return model(x)
  
  
  def chain_layers_tf(x, kernels):
      y = x
      for kernel in kernels:
          y = tf.matmul(y, kernel)
      return y
  
  # call once
  f(x)
  chain_layers_tf_wrapped = tf.function(chain_layers_tf)
  
  kernels_tf = [x.kernel for x in model.layers]
  kernels_np = [x.kernel.numpy() for x in model.layers]
  
  chain_layers_tf_wrapped(x, kernels_np)
  chain_layers_tf_wrapped(x, kernels_tf)
  
  def chain_layers_np(x, kernels):
      y = x
      for kernel in kernels:
          y = y.dot(kernel)
      return y
  
  model(x)
  
  """
  %timeit t.model(t.x)
  %timeit t.f(t.x)
  %timeit t.chain_layers_tf(t.x, t.kernels_tf)
  %timeit t.chain_layers_tf_wrapped(t.x, t.kernels_tf)
  %timeit t.chain_layers_np(t.xnumpy, t.kernels_np)
  """
  
  # In [107]: %timeit t.model(t.x)
  #      ...: %timeit t.f(t.x)
  #      ...: %timeit t.chain_layers_tf(t.x, t.kernels_tf)
  #      ...: %timeit t.chain_layers_tf_wrapped(t.x, t.kernels_tf)
  #      ...: %timeit t.chain_layers_np(t.xnumpy, t.kernels_np)
  # 1.51 ms ± 1.73 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
  # 131 µs ± 856 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  # 491 µs ± 1.17 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
  # 417 µs ± 1.53 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
  # 5.21 µs ± 7.56 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
  </code></pre>
  
  <p>Previous notes:</p>
  
  <p><a href="https://i.stack.imgur.com/nzNiy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nzNiy.png" alt="enter image description here"></a></p>
  
  <p>UPDATE: </p>
  
  <pre><code>Not just numpy as above, but tensorflow is faster than tensorflow.keras
  
  In [22]: x = np.random.randn(100, 12).astype(np.float32)
  
  In [23]: a = np.random.randn(12, 16).astype(np.float32)
  
  In [24]: %timeit tf.matmul(x, a)
  39.3 µs ± 98.5 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  
  
  UPDATE: 
  
  more layers not much different
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/c7vB1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/c7vB1.png" alt="enter image description here"></a></p>