title: Keras not using multiple cores
body: <p>Based on the famous <code>check_blas.py</code> script, I wrote this one to check that theano can in fact use multiple cores:</p>
  
  
  
  <pre class="lang-py prettyprint-override"><code>import os
  os.environ['MKL_NUM_THREADS'] = '8'
  os.environ['GOTO_NUM_THREADS'] = '8'
  os.environ['OMP_NUM_THREADS'] = '8'
  os.environ['THEANO_FLAGS'] = 'device=cpu,blas.ldflags=-lblas -lgfortran'
  
  import numpy
  import theano
  import theano.tensor as T
  
  M=2000
  N=2000
  K=2000
  iters=100
  order='C'
  
  a = theano.shared(numpy.ones((M, N), dtype=theano.config.floatX, order=order))
  b = theano.shared(numpy.ones((N, K), dtype=theano.config.floatX, order=order))
  c = theano.shared(numpy.ones((M, K), dtype=theano.config.floatX, order=order))
  f = theano.function([], updates=[(c, 0.4 * c + .8 * T.dot(a, b))])
  
  for i in range(iters):
      f(y)
  </code></pre>
  
  <p>Running this as <code>python3 check_theano.py</code> shows that 8 threads are being used. And more importantly, the code runs approximately 9 times faster than without the <code>os.environ</code> settings, which apply just 1 core: 7.863s vs 71.292s on a single run.</p>
  
  <p>So, I would expect that Keras now also uses multiple cores when calling <code>fit</code> (or <code>predict</code> for that matter). However this is not the case for the following code:</p>
  
  <pre class="lang-py prettyprint-override"><code>import os
  os.environ['MKL_NUM_THREADS'] = '8'
  os.environ['GOTO_NUM_THREADS'] = '8'
  os.environ['OMP_NUM_THREADS'] = '8'
  os.environ['THEANO_FLAGS'] = 'device=cpu,blas.ldflags=-lblas -lgfortran'
  
  import numpy
  from keras.models import Sequential
  from keras.layers import Dense
  
  coeffs = numpy.random.randn(100)
  
  x = numpy.random.randn(100000, 100);
  y = numpy.dot(x, coeffs) + numpy.random.randn(100000) * 0.01
  
  model = Sequential()
  model.add(Dense(20, input_shape=(100,)))
  model.add(Dense(1, input_shape=(20,)))
  model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
  
  model.fit(x, y, verbose=0, nb_epoch=10)
  </code></pre>
  
  <p>This script uses only 1 core with this output:</p>
  
  <pre class="lang-py prettyprint-override"><code>Using Theano backend.
  /home/herbert/venv3/lib/python3.4/site-packages/theano/tensor/signal/downsample.py:5: UserWarning: downsample module has been moved to the pool module.
  warnings.warn("downsample module has been moved to the pool module.")
  </code></pre>
  
  <p>Why does the <code>fit</code> of Keras only use 1 core for the same setup? Is the <code>check_blas.py</code> script actually representative for neural network training calculations?</p>
  
  <p>FYI:</p>
  
  <pre class="lang-py prettyprint-override"><code>(venv3)herbert@machine:~/ $ python3 -c 'import numpy, theano, keras; print(numpy.__version__); print(theano.__version__); print(keras.__version__);'
  ERROR (theano.sandbox.cuda): nvcc compiler not found on $PATH. Check your nvcc installation and try again.
  1.11.0
  0.8.0rc1.dev-e6e88ce21df4fbb21c76e68da342e276548d4afd
  0.3.2
  (venv3)herbert@machine:~/ $
  </code></pre>
  
  <p><strong>EDIT</strong></p>
  
  <p>I created a Theano implementaiton of a simple MLP as well, which also does not run multi-core:</p>
  
  <pre class="lang-py prettyprint-override"><code>import os
  os.environ['MKL_NUM_THREADS'] = '8'
  os.environ['GOTO_NUM_THREADS'] = '8'
  os.environ['OMP_NUM_THREADS'] = '8'
  os.environ['THEANO_FLAGS'] = 'device=cpu,blas.ldflags=-lblas -lgfortran'
  
  import numpy
  import theano
  import theano.tensor as T
  
  M=2000
  N=2000
  K=2000
  iters=100
  order='C'
  
  coeffs = numpy.random.randn(100)
  x = numpy.random.randn(100000, 100).astype(theano.config.floatX)
  y = (numpy.dot(x, coeffs) + numpy.random.randn(100000) * 0.01).astype(theano.config.floatX).reshape(100000, 1)
  
  x_shared = theano.shared(x)
  y_shared = theano.shared(y)
  
  x_tensor = T.matrix('x')
  y_tensor = T.matrix('y')
  
  W0_values = numpy.asarray(
      numpy.random.uniform(
          low=-numpy.sqrt(6. / 120),
          high=numpy.sqrt(6. / 120),
          size=(100, 20)
      ),
      dtype=theano.config.floatX
  )
  W0 = theano.shared(value=W0_values, name='W0', borrow=True)
  
  b0_values = numpy.zeros((20,), dtype=theano.config.floatX)
  b0 = theano.shared(value=b0_values, name='b0', borrow=True)
  
  output0 = T.dot(x_tensor, W0) + b0
  
  W1_values = numpy.asarray(
      numpy.random.uniform(
          low=-numpy.sqrt(6. / 120),
          high=numpy.sqrt(6. / 120),
          size=(20, 1)
      ),
      dtype=theano.config.floatX
  )
  W1 = theano.shared(value=W1_values, name='W1', borrow=True)
  
  b1_values = numpy.zeros((1,), dtype=theano.config.floatX)
  b1 = theano.shared(value=b1_values, name='b1', borrow=True)
  
  output1 = T.dot(output0, W1) + b1
  
  params = [W0, b0, W1, b1]
  cost = ((output1 - y_tensor) ** 2).sum()
  
  gradients = [T.grad(cost, param) for param in params]
  
  learning_rate = 0.0000001
  
  updates = [
      (param, param - learning_rate * gradient)
      for param, gradient in zip(params, gradients)
  ]
  
  train_model = theano.function(
      inputs=[],#x_tensor, y_tensor],
      outputs=cost,
      updates=updates,
      givens={
          x_tensor: x_shared,
          y_tensor: y_shared
      }
  )
  
  errors = []
  for i in range(1000):
      errors.append(train_model())
  
  print(errors[0:50:])
  </code></pre>