title: How does batch size impact time execution in neural networks?
body: <p>I am using CNN (convolution neural network) model to train cifar10.
  I tried to change the batch size at each execution to see its impact on time</p>
  
  <p>My conclusion was : the bigger the batch size the more time the model took to be executed.</p>
  
  <p>Does this seem logical because at the end of every batch we apply back propagation algorithm, meaning with larger batch size we apply less gradient descent so logically we should have less execution time.</p>
  
  <p>I found the opposite. What do you think guys ! 
  Thanks</p>
  
  <p>Here is my session code : </p>
  
  <pre><code>       with tf.Session() as sess:
           sess.run(init)
           summary_writer =tf.summary.FileWriter(logs_path,graph=tf.get_default_graph())
          start_time = time.time()
          for i in range(iteration_number):
            j = (i - epoch) * batch_size % number_of_examples
            k= (i - epoch + 1) * batch_size % number_of_examples
            if (k &lt; j): # THE END OF DATA SET ------------------------
              k = number_of_examples
              batch_x = train_images[j:number_of_examples, :]
              batch_y = train_labels[j:number_of_examples, :]
              print("Iter " + str(i) + ", epoch Loss= " + \
                      "{:.6f}".format(loss) + ", Training Accuracy= " + \
                      "{:.5f}".format(acc))
  
              data = numpy.concatenate((train_images, train_labels), axis=1)
              numpy.random.shuffle(data)
              train_images = data[:, :3072]
              train_labels = data[:, 3072:3082]
              epoch = i + 1
  
            else:
              batch_x = train_images[j:k, :]
              batch_y = train_labels[j:k, :]
              loss, acc, summary = sess.run([cost, accuracy, merged_summary_op], feed_dict={x: batch_x,
                                                          y: batch_y,
                                                          keep_prob: 0.3})
              summary_writer.add_summary(summary)
  
              sess.run(optimizer, feed_dict={x: batch_x, y: batch_y,
                                          keep_prob: dropout})
  </code></pre>