text: An Introduction to Deep Learning using nolearn want a computer to be able to correctly identify each number. This would come in handy at the post office, along with many other applications such as identifying address numbers in images.
  Different machine learning experts have worked on this problem for several years trying a variety of approaches. Ultimately, however, the best algorithm for the task of image classification would be an algorithm that can fit features that aren’t easily described. This is where neural networks truly shine.
  Deep learning has become quite the trendy subject recently. Here is an example. Essentially, researchers are trying to use Deep Learning to categorize images better than a human could, something humans are typically better at doing naturally than a computer.
  If you want an idea as to how Deep Learning is being used (and how it works) listening to a talk from Dr. Andrew Ng is always a good idea!
  Basically, the key part of deep learning is finding features that can be used for classification from UNLABELED training examples. More data = a more accurate model, most of the time. Labeling the myriad of images available on the web one at a time into categories would be very difficult and time consuming (although Amazon paid people money to do this through their Mechnical Turk program). Instead, just find features from your unlabeled images (such as certain curves, color gradients, or other features that humans would have a difficult time describing). Then, use these features to help categorize your images. It greatly improves the accuracy because now you have more data to work with!
  As a way of practicing this, we are going to use some Deep Learning on the MNIST dataset of numbers to see if we can get a low error rate. I couldn’t decide which neural network library to use at first, since scikit-learn doesn’t really have any. I became inspired by this post here from Dr. Adrian Rosebrock. He used the library nolearn, which we will also be using.
  I just found out while writing this that the dbn class of nolearn is being removed entirely for the new version 0.6 and that the author suggests switching to lasagne. If you are going to follow along with the IPython Notebook, make sure you have a version prior to nolearn 0.6.
  Processing the Images
  Just to get a starting idea of where we are beginning, let’s try a very basic run of a deep belief network using nolearn. Because image processing can be fairly memory intensive, I am including the key steps of processing the data and wrapping them inside a function.
  Our function will load the MNIST data, transform it to a 0 to 1 scale for all of the pixel intensity values, and finally split the data into train and test sets. We need to scale the data between 0 to 1 so that the neural network will perform more efficiently..
  Notice that our version is not using cudamat. It would run a lot faster if we had a nice GPU to use, but since I am on a laptop that doesn’t have a nice one, we are going to run this using the slower, CPU
  So, our starting accuracy is pretty close to 98% at 97.95%. It looks like the model is doing a good job, but it is having more trouble with the number 5 and 9 based on the lower f1-scores for those two numbers. Is there any way we can boost our score?
  Well, it would be nice if we had more images we could work with. That way, we could fit the missed examples better. There is a way we can solve this problem: create our own training examples!
  This is a common solution in the area of image classification. Some of the handwritten images may not be aligned correctly, or may be slightly rotated. To account for this in the model, let’s add some “noise” to our training set and see if we can improve some. In addition, let’s try using some more advanced features of the Deep Belief Network, such as more epochs and some regularization to avoid overfitting.
  Artificial Training Examples
  To augment the size of our training set, we need to create a function that will change our original images just a little bit. We want to include a bit of translation (moving the pixels around) and rotation, both at random. To see how this will work, let’s create a function that will randomly alter an existing training example.
  First, let’s take a look at one of the training examples, reverting it back to its original form and plotting it.
  import matplotlib.pyplot as plt sample = np.reshape(x_train[0], ((28,28))) # Get the training data back to its original form. sample = sample*255. # Get the original pixel values. plt.imshow(sample, cmap = plt.cm.gray)
  We can see this is clearly a 7. Now, let’s make our function that will rotate and translate the image and view the sample again.
  from scipy.ndimage import convolve, rotate def random_image_generator(image): ''' This function will randomly translate and rotate an image, producing a new, altered version as output. ''' # Create our movement vectors for translation first. move_up = [[0, 1, 0], [0, 0, 0], [0, 0, 0]] move_left = [[0, 0, 0], [1, 0, 0], [0, 0, 0]] move_right = [[0, 0, 0], [0, 0, 1], [0, 0, 0]] move_down = [[0, 0, 0], [0, 0, 0], [0, 1, 0]] # Create a dict to store these directions in. dir_dict = {1:move_up, 2:move_left, 3:move_right, 4:move_down} # Pick a random direction to move. direction = dir_dict[np.random.randint(1,5)] # Pick a random angle to rotate (10 degrees clockwise to 10 degrees counter-clockwise). angle = np.random.randint(-10,11) # Move the random direction and change the pixel data back to a 2D shape. moved = convolve(image.reshape(28,28), direction, mode = 'constant') # Rotate the image rotated = rotate(moved, angle, reshape = False) return rotated
  Now what happens to the image if we change it randomly?
  distorted_example = random_image_generator(x_train[0]*255.) plt.imshow(distorted_example, cmap = plt.cm.gray)
  What you see will vary if you are using this notebook interactively (since the distortions are random), but in this case the 7 is more upright now and slightly lower.
  In theory we could make an infinite number of these artifical training examples to help improve our Deep Belief Network (if we had a lot of memory of course!)
  To make sure we don’t run out of memory, let’s create a function that will augment our training set internally with an additional amount of artifical examples. It will call the “random image generator” function inside itself.
  def extra_training_examples(features, targets, num_new): ''' This function will take the training set and increase it by artifically adding new training examples. We can also specify how many training examples we wish to add with the num_new parameter. ''' # First, create empty arrays that will hold our new training examples. x_holder = np.zeros((num_new, features.shape[1])) y_holder = np.zeros(num_new) # Now, loop through our training examples, selecting them at random for distortion. for i in xrange(num_new): # Pick a random index to decide which image to alter. random_ind = np.random.randint(0, features.shape[0]) # Select our training example and target. x_samp = features[random_ind] y_samp = targets[random_ind] # Change our image and convert back to 1D. new_image = random_image_generator(x_samp).ravel() # Store these in our arrays. x_holder[i,:] = new_image y_holder[i] = y_samp # Now that our loop is over, combine our original training examples with the new ones. combined_x = np.vstack((features, x_holder)) combined_y = np.hstack((targets, y_holder)) # Return our new training examples and targets. return combined_x, combined_y
  Let’s now make our new x_train and y_train objects and replace the old ones with these new, larger ones.], 300, 10], learn_rates = 0.3, learn_rate_decays = 0.9, epochs = 10) # I am turning off the verbose statements this time..98 0.99 0.99 1604 2.0 0.97 0.98 0.97 1348 3.0 0.98 0.97 0.98 1427 4.0 0.99 0.96 0.97 1362 5.0 0.98 0.97 0.97 1280 6.0 0.99 0.99 0.99 1397 7.0 0.99 0.97 0.98 1461 8.0 0.95 0.98 0.97 1390 9.0 0.95 0.97 0.96 1419 avg / total 0.98 0.98 0.98 14000 The accuracy is: 0.976857142857.
  Tuning the Network
  Because of our larger training set, we need to make some changes to our original Deep Belief Network. Next, we are going to try this again with three changes:
  - Decrease our learning rate from 0.3 to 0.2 for smaller steps during gradient descent
  - Increase the number of hidden nodes from 300 to 500
  - Increase the number of training epochs from 10 to 50.
  dbn_model = DBN([x_train.shape[1], 500, 10], # Increased the hidden nodes to add more complex features. learn_rates = 0.2, # Smaller steps for gradient descent. learn_rate_decays = 0.9, epochs = 50) # Give the model more time to converge..99 0.99 0.99 1604 2.0 0.98 0.98 0.98 1348 3.0 0.98 0.98 0.98 1427 4.0 0.98 0.98 0.98 1362 5.0 0.99 0.97 0.98 1280 6.0 0.99 0.99 0.99 1397 7.0 0.98 0.98 0.98 1461 8.0 0.98 0.98 0.98 1390 9.0 0.98 0.97 0.98 1419 avg / total 0.98 0.98 0.98 14000 The accuracy is: 0.983714285714.
  Regularization of The Network: Dropout).
  dbn_model = DBN([x_train.shape[1], 500, 10], learn_rates = 0.2, learn_rate_decays = 0.9, dropouts = 0.25, # Express the percentage of nodes that will be randomly dropped as a decimal. epochs = 50) 1.00 0.99 1312 1.0 0.99 0.99 0.99 1604 2.0 0.98 0.99 0.98 1348 3.0 0.99 0.98 0.98 1427 4.0 0.99 0.99 0.99 1362 5.0 0.99 0.97 0.98 1280 6.0 0.99 0.99 0.99 1397 7.0 0.98 0.98 0.98 1461 8.0 0.98 0.99 0.98 1390 9.0 0.98 0.98 0.98 1419 avg / total 0.99 0.99 0.99 14000 The accuracy is: 0.985357142857?
  Ideas for Improvement and Summary:
  - Try utilizing a GPU (graphics processing unit) to train our network instead. Neural networks tend to be much faster when using one (or several!) You could either configure nolearn with cudamat on a remote instance such as AWS (Amazon Web Services) or use a graphics card on a more powerful desktop computer. Adrian Rosebrook’s post experiments with that here.
  - Experiment with different parameters for further fine tuning. We could try increasing the number of layers for further complexity or increasing the number of epochs.
  - Try a larger number of artificial examples (if we have more memory available) to see how that improves the accuracy further. Doing so with enough additional examples should make pushing 99% accuracy a reality.
  This post was originally posted at jessesw.com. If you would like the IPython Notebook for this blog post, you can find it here.