title: How can I feed .csv training data to a convolutional neural network in mxnet?
body: <p>I recently installed mxnet (python package) with GPU support on Windows 10 and Python 3.5. I run through a couple of examples and they seem to work fine.</p>
  
  <p>I am used to scikit-learn style machine learning packages and very new to Python deep learning packages such as Mxnet although I have already used Mxnet in R. I'm having a hard time understanding how to feed .csv training data to the model.</p>
  
  <p>I would like to feed to a simple CNN some images. The images are 28x28 pixel and stored as flattened arrays in a .csv. I have two .csv files, one for training and the other for testing. Each .csv file has the following structure:</p>
  
  <pre><code>label, pixel1, pixel2, ..., pixel784
  0,...
  1,...
  </code></pre>
  
  <p>There are 10 labels in total and around 1000/300 images in the training set/test set.</p>
  
  <p>I am using the following code to load the data and train the model:</p>
  
  <pre><code>import mxnet as mx
  import pandas as pd
  import numpy as np
  import os
  
  path = "C://users//me//data"
  os.chdir(path)
  
  df_train = pd.read_csv("train_28.csv")
  df_test = pd.read_csv("test_28.csv")
  
  keys = ['pixel.'+str(i) for i in range(1,785)]
  
  X_train = df_train[keys].get_values().T
  X_train = X_train.reshape((1200,28,28,1))
  y_train = df_train['label'].get_values().reshape((1200,1))
  #y_train = y_train.reshape((28,28,1,1200))
  
  
  
  data = mx.symbol.Variable('data')
  
  # First conv layer
  conv1 = mx.symbol.Convolution(data=data, kernel=(5,5), num_filter=20)
  tanh1 = mx.symbol.Activation(data=conv1, act_type="tanh")
  pool1 = mx.symbol.Pooling(data=tanh1, pool_type="max",
                                kernel=(2,2), stride=(2,2))
  
  # Second conv layer
  conv2 = mx.symbol.Convolution(data=pool1, kernel=(5,5), num_filter=50)
  tanh2 = mx.symbol.Activation(data=conv2, act_type="tanh")
  pool2 = mx.symbol.Pooling(data=tanh2, pool_type="max",
                                kernel=(2,2), stride=(2,2))
  
  # First fully connected
  flatten = mx.symbol.Flatten(data=pool2)
  fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=500)
  tanh3 = mx.symbol.Activation(data=fc1, act_type="tanh")
  # second fullc
  fc2 = mx.symbol.FullyConnected(data=tanh3, num_hidden=10)
  # loss
  lenet = mx.symbol.SoftmaxOutput(data=fc2, name='softmax')                              
  
  device = mx.gpu()                              
  
  model = mx.model.FeedForward.create(lenet,
                                      X = X_train,
                                      y = y_train,
                                      ctx = device,
                                      num_epoch = 30)
  </code></pre>
  
  <p>I am using this approach which is similar to the one I was using with mxnet in R, (btw on R it works perfectly, however I cannot use the GPU on R so I need to use Python for better performances...) however I am getting the following error:</p>
  
  <pre><code>[16:54:11] D:\chhong\mxnet\dmlc-core\include\dmlc/logging.h:235: [16:54:11] d:\chhong\mxnet\src\operator\./convolution-inl.h:347: Check failed: ksize_x &lt;= dshape[3] &amp;&amp; ksize_y &lt;= dshape[2] kernel size exceed input
  Traceback (most recent call last):
    File "C:\Users\Me\Desktop\esempio_lenet.py", line 57, in &lt;module&gt;
      num_epoch = 30)
    File "C:\Users\Me\Anaconda3\lib\site-packages\mxnet-0.7.0-py3.5.egg\mxnet\model.py", line 901, in create
      eval_batch_end_callback=eval_batch_end_callback)
    File "C:\Users\Me\Anaconda3\lib\site-packages\mxnet-0.7.0-py3.5.egg\mxnet\model.py", line 745, in fit
      self._init_params(dict(data.provide_data+data.provide_label))
    File "C:\Users\Me\Anaconda3\lib\site-packages\mxnet-0.7.0-py3.5.egg\mxnet\model.py", line 485, in _init_params
      arg_shapes, _, aux_shapes = self.symbol.infer_shape(**input_shapes)
    File "C:\Users\Me\Anaconda3\lib\site-packages\mxnet-0.7.0-py3.5.egg\mxnet\symbol.py", line 453, in infer_shape
      return self._infer_shape_impl(False, *args, **kwargs)
    File "C:\Users\Me\Anaconda3\lib\site-packages\mxnet-0.7.0-py3.5.egg\mxnet\symbol.py", line 513, in _infer_shape_impl
      ctypes.byref(complete)))
    File "C:\Users\Me\Anaconda3\lib\site-packages\mxnet-0.7.0-py3.5.egg\mxnet\base.py", line 77, in check_call
      raise MXNetError(py_str(_LIB.MXGetLastError()))
  mxnet.base.MXNetError: InferShape Error in convolution0: [16:54:11] d:\chhong\mxnet\src\operator\./convolution-inl.h:347: Check failed: ksize_x &lt;= dshape[3] &amp;&amp; ksize_y &lt;= dshape[2] kernel size exceed input
  </code></pre>
  
  <p>And I cannot figure out what I am doing wrong. Could please someone tell me what is this error about and provide me with a clear set of instructions on how to load .csv files with the same structure as above and train a mxnet model? I took a look at the documentation but could not figure out on my own how to load .csv files properly...</p>
  
  <p>The reason why I am asking for a procedure for loading such .csv files is that I mostly deal with data in that format and it would be very valuable to me to be able to run a script against a folder with these .csv files and have them ready to be used for training a deep convolutional neural network.</p>
  
  <p>A set of train and test .csv files are available <a href="https://www.dropbox.com/s/pkrymi1x0v8776g/example_csv.zip?dl=0" rel="nofollow">here</a> should you need them to reproduce the example code I wrote.</p>
  
  <p>Thank you</p>