!pip freeze

"""
Explanation: Normalisation should now be implemented in dense_dataset.py. This notebook is going to investigate the difference between using per-pixel versus global normalisation. It's also going to act as a walkthrough for general experimentation using our code.
Setting up
To set up this notebook I've sourced my Python 2.7.8 virtualenv, set up as described in the README on the tools repo. Along with the recommended packages I've installed there I've installed some others. This is the exact list:
End of explanation
"""


cd ..

!cat start_script.sh

"""
Explanation: Also, I've sourced the start_script.sh to set the required environment variables for Theano on this machine:
End of explanation
"""


!nvidia-smi

"""
Explanation: Before running this, check if any of the GPUs are in use:
End of explanation
"""


%%bash
echo $LD_LIBRARY_PATH
echo $CUDA_ROOT
echo $THEANO_FLAGS

"""
Explanation: Looking at that only one of the GPUs is in use, but the program isn't doing anything (because it's another program I'm debugging). Nothing is happening on GPU 0 so I've chosen that:
End of explanation
"""


ls run_settings/

"""
Explanation: Settings
To provide settings for an experiment on these GPUs we use a json file loaded as a dictionary with a few specific keys. These are in the directory run_settings in the work repository:
End of explanation
"""


!cat run_settings/alexnet_based.json

"""
Explanation: At the moment, our best performing model was the alexnet_based.json, a fairly unprocessed port of Matt's work on a vegetable detector to this problem.
End of explanation
"""


!cat yaml_templates/alexnet_based.yaml

"""
Explanation: The most important part of the above file is the fact it points to the alexnet_based.yaml file, which is where the entire model is defined. Looking at that:
End of explanation
"""


%run ~/repos/pylearn2/pylearn2/scripts/print_monitor.py /disk/scratch/neuroglycerin/models/alexnet_based.pkl.backup

"""
Explanation: Here's some of the information we can read out of this:

It's a six layer convolutional network, with an MLP layer at the end.
It has dropout implemented in layer h5. It has weight decay in the first 5 layers.
It won't finish until it's run 500 epochs (that's going to take a while).
It has extensions to adjust the momentum depending on the misclassification value.
It's monitoring the channel valid_y_misclass and saving the model with the best score (we would probably prefer log loss score for this).
It is modulating the momentum with MomentumAdjustor and LinearDecayOverEpoch.

Without normalisation, and with only resizing the images to 48 by 48 as described in the above json the score achieved in our saved model was:
End of explanation
"""


%matplotlib inline
import holoviews as hl
%load_ext holoviews.ipython

%run ~/repos/pylearn2/pylearn2/scripts/plot_monitor.py /disk/scratch/neuroglycerin/models/alexnet_based.pkl.backup

"""
Explanation: Submitting this model to the Kaggle leaderboard achieved the best score we've been able to manage so far: 2.319966. Suspect that the valid_objective is our estimate of that score on the validation set, but I'm not actually sure yet.
End of explanation
"""


%env PYLEARN2_VIEWER_COMMAND=/afs/inf.ed.ac.uk/user/s08/s0805516/repos/neukrill-net-work/image_hack.sh

!echo $PYLEARN2_VIEWER_COMMAND

%run ~/repos/pylearn2/pylearn2/scripts/show_weights.py /disk/scratch/neuroglycerin/models/alexnet_based.pkl.backup

from IPython.display import Image

def plot_recent_pylearn2():
    pl2plt = Image(filename="/afs/inf.ed.ac.uk/user/s08/s0805516/tmp/pylearnplot.png", width=500)
    return pl2plt
plot_recent_pylearn2()

"""
Explanation: Unfortunately, it's difficult to view the weights in an IPython notebook (and we get the above problems with user input above) and the devs don't want to fix it because they don't like IPython notebooks. So we need to hack around this problem.
The hack I've made is to define their environment variable PYLEARN2_VIEWER_COMMAND to be a shell script that just uses ImageMagick to rewrite the file (whatever it is) to a file in ~/tmp called pylearnplot.png. Then, we can just use IPython's rich display system to show it in the notebook.
End of explanation
"""


import neukrill_net.utils as utils

!cp run_settings/alexnet_based.json run_settings/alexnet_based_norm_pixel.json

import os
import copy

settings = utils.Settings("settings.json")
bypixel = utils.load_run_settings("run_settings/alexnet_based_norm_pixel.json",settings)
byglobal = copy.deepcopy(bypixel)
byglobal['run_settings_path'] = os.path.abspath("run_settings/alexnet_based_norm_global.json")
bypixel['preprocessing']['normalise'] = {"global_or_pixel":"pixel"}
byglobal['preprocessing']['normalise'] = {"global_or_pixel":"global"}
utils.save_run_settings(byglobal)
utils.save_run_settings(bypixel)

import neukrill_net.dense_dataset as dense
reload(dense)

# force recompile
import neukrill_net.image_processing
reload(neukrill_net.image_processing)

global_data = dense.DensePNGDataset(run_settings=byglobal['run_settings_path'],verbose=True)

pixel_data = dense.DensePNGDataset(run_settings=bypixel['run_settings_path'],verbose=True)

"""
Explanation: Looking at these, quite a few of the weight matrices seems to be quite noisy. Maybe they've learned something about the texture of the classes? Anyway, this was just to show that we can investigate the weights like this in a notebook. Looking closely at the weights is a job for another time.
Normalisation
Taking a break from how to use Pylearn2, this notebook is also about looking at what Normalisation can do for us. We might as well inspect by eye the difference between images normalised by pixel and images normalised globally.
Loading some images into our Pylearn2 dataset class for two different cases. First, we have to prepare two run settings files:
End of explanation
"""


!cat run_settings/alexnet_based_norm_pixel.json

"""
Explanation: The normalisation statistics have now been saved to run settings json, and we can look at them if we want to:
End of explanation
"""


import numpy as np

np.random.seed(42)
random_indexes = np.random.randint(low=0,high=global_data.X.shape[0],size=20)
global_images = global_data.get_topological_view(
                    global_data.X[random_indexes,:]).reshape(20,48,48)

%opts Image style(cmap='gray')
channels = hl.Image(global_images[0])
for i in global_images[1:,:,:]:
    channels = channels + hl.Image(i)
# this is so easy it should be illegal
channels

"""
Explanation: Now that the data is loaded we can have a look at these images with Holoviews (it's from Edinburgh!).
End of explanation
"""


np.random.seed(42)
random_indexes = np.random.randint(low=0,high=pixel_data.X.shape[0],size=20)
pixel_images = pixel_data.get_topological_view(
                    pixel_data.X[random_indexes,:]).reshape(20,48,48)

channels = hl.Image(pixel_images[0])
for i in pixel_images[1:,:,:]:
    channels = channels + hl.Image(i)
channels

"""
Explanation: Global images are just scaled by a single value, so we'd expect them to look the same. There's no reason to expect that for images scaled over the pixel values though:
End of explanation
"""


import matplotlib.pyplot as plt

data = global_data.X
print("Max",np.max(data))
print("Min",np.min(data))
print("Mean",np.mean(data))
print("Stdev",np.sqrt(np.var(data)))
_ = plt.hist(np.ravel(data),bins=50)
plt.title("Global normalisation results")

data = pixel_data.X
print("Max",np.max(data))
print("Min",np.min(data))
print("Mean",np.mean(data))
print("Stdev",np.sqrt(np.var(data)))
_ = plt.hist(np.ravel(data),bins=50)
plt.title("Pixel normalisation results")

"""
Explanation: It's easy to see the difference. There's a washout in the middle of images compared with the outside because so little happens at the edges that when something does happen it doesn't get squashed.
End of explanation
"""


%run train.py -h

"""
Explanation: Similar distributions, but getting some massive negative values on some of these pixels. At this point all we can do is try to train a model with them and see if we get better results than without scaling.
Running a model
We're going to be running two models, using a utility called longjob. If you leave a job running on an active ssh connection then it will be deemed inactive after a period of time (12 hours?) and you'll suddenly stop being able to write anything. Using longjob this won't happen. This notebook server is being run through longjob for this reason.

longjob is a wrapper script which simplifies the task of running jobs which require
access to valid kerberos credentials for a longer period than the default ticket
length (18 hours). The most common reason for this is so that a job can continue to
access the AFS file system after the user’s initial ticket has expired.

Another useful utility if you're not in a notebook, running jobs through the shell is screen. Using this it's possible to keep a shell open without being connected to it. You can run something in a shell then disconnect from it and leave it running in the background, then come back to it later. Simple usage in our case:
bash
screen -D -R &lt;session_name&gt;
Where &lt;session_name&gt; is whatever name you want to give your shell session. Then you can use ctrl-a + d to detach and leave it running.
We want to run two nets on different GPUs. One for global and one for pixel-wise normalisation. The two run settings files will be an input to the general-purpose training script train.py. To get the help message for train.py:
End of explanation
"""


!screen -ls

"""
Explanation: Can't figure out how to send off longjobs from IPython notebooks yet. So the two commands I'm running are:
bash
longjob -28day -c "python train.py run_settings/alexnet_based_norm_global.json" &gt; /disk/scratch/neuroglycerin/logs/norm_global.log
and
bash
longjob -28day -c "python train.py run_settings/alexnet_based_norm_pixel.json" &gt; /disk/scratch/neuroglycerin/logs/norm_pixel.log
These were started in a screen session we can see with the following command:
End of explanation
"""


!tail -n 20 /disk/scratch/neuroglycerin/logs/norm_global.log

!tail -n 20 /disk/scratch/neuroglycerin/logs/norm_pixel.log

"""
Explanation: Because we're writing to these logs we can check the most recent progress by just tailing the logs:
End of explanation
"""


!nvidia-smi

"""
Explanation: We can also check how the GPUs are doing with the same command as at the start:
End of explanation
"""


!tail -n 20 /disk/scratch/neuroglycerin/logs/norm_global.log

!tail -n 200 /disk/scratch/neuroglycerin/logs/norm_pixel.log

"""
Explanation: One is running on GPU 0 and one is running on GPU 2. The above were run shortly after starting the training. After leaving it overnight we get the following logs:
End of explanation
"""


!df -h

"""
Explanation: So that file just stops halfway through writing? Has someone filled up scratch again?
End of explanation
"""


%run test.py run_settings/alexnet_based_norm_global.json
%run test.py run_settings/alexnet_based_norm_pixel.json

"""
Explanation: Looks like it was probably filled during the night and has now been cleaned up a bit.
Results logging
Pending section on how these results are logged back into the run settings. (Pending code to do that).
Analysing these networks
Looking into the pickle files, seeing which is better and why.
Prediction
So we've trained these models and now we want to make a submission to the Kaggle leaderboard. For this, we use our general purpose test.py script (I would've preferred to call it predict.py at this point). It takes a stored pickle file, loads the test data and runs forward passes of the network to generate predictions for every test point. It then writes these results to gzipped csv file. All we have to do is specify which run settings file to use.
Doing this for both of these:
End of explanation
"""


