%pwd

root_path = %pwd
root_path

%matplotlib inline
import numpy as np

"""
Explanation: This notebook records a personal learning project in which a convnet is taught to ID right whales. 
Planning
Tasks
After agreeing to the rules of the competition, the following tasks must be completed by the notebook, in order:
* set up the data;
    - set up the 'train', 'test' folders, and the 'sample' folders 'train' and 'valid';
    - create a validation set;
    - create a sample set;
    - check that each class is in a separate folder;
* explore the data:
    - get some idea of how it is organised, what is available to use, dimensions, classes, etc.;
* set up the model:
    - establish which pre-trained model to use
    - import the model
    - load the pre-trained weights
* set up for fine-tuning:
    - remove output layer
    - append new output layer (n classes)
    - set the batch size
* train the network (sample data)
Principles to follow

record every step, marking which can be skipped on future runs (dependencies), e.g. using 'raw' format
use relative folder referencing
while remaining 'agile', look for easy opportunities to make the code reusable
understand every line of code used
experiment, to understand
try for 30 mins, before looking for help

Import packages, setup, instantiate vgg
End of explanation
"""


import utils
from utils import *

"""
Explanation: <strong> N.B. </strong> You'll need to ensure that the files...<strong>
* 'utils.py' (by fast.ai), 
* 'vgg16.py' (adapted by them?), and
* 'vgg16bn' </strong>
...are findable by python - e.g. present in the root of the current directory.
End of explanation
"""


%ls

"""
Explanation: Download, unzip, and organise the data
First, we'll create the 'data' directory into which to load the Kaggle sets.
End of explanation
"""


%pwd

!tree -d

"""
Explanation: Download the data
<strong>N.B. Don't rerun this - it takes ~45 min!</strong>
Create the folder structure
At this point, we're expecting that the current directory is '/home/mark/Study/dl.fast.ai/deeplearning1/_MDR/right_whales'. Let's check this, and look at what's already there.
End of explanation
"""


import os
all_data_path = "data/"
sampled_path = all_data_path + "sample/"

# The following allows for easy switching...
data_path = all_data_path   
#data_path = sampled_path      

%ls $data_path

"""
Explanation: I'm not currently sure whether this is the 'best' way to do this, but we'll set the path variables this way, for now...
<strong>DO re-run this cell...</strong>
End of explanation
"""


%pwd

!tree -d

"""
Explanation: N.B. the folder structure that we're aiming for (under the root folder for the project) looks like this:
Let's check that.
End of explanation
"""


%ls -l $all_data_path

"""
Explanation: Unzip the data
Quoted from the Kaggle data page: 
<blockquote>In this competition you are given aerial images, each containing a single Right whale. These images were taken over the course of 10 years and hundreds of helicopter trips, and have been selected and labeled by NOAA scientists with their whale IDs.

To ensure that this is a computer vision problem, we have removed metadata such as creation dates and geo tags from the photos. To discourage hand labeling, we have supplemented the test dataset with some images that are resized, cropped, or flipped. These processed images are ignored and don't count towards your score.  

Your goal is to build a "face recognition system" for whales.</blockquote>
File descriptions
imgs.zip - zipped folder of all (train/test) images
train.csv - the training set with image file name and whale id. 
sample_submission.csv - a sample submission file in the correct format
imgs_subset.zip - zipped folder of a subset (first 500 images) of the full dataset
End of explanation
"""


!head -n5 $all_data_path/train.csv

"""
Explanation: Let's unzip the train.csv file first, and take a look at it.
So, it looks like the this file gives the correct whaleID for each image.
End of explanation
"""


import csv
reader = csv.reader(open('train.csv', 'r'))
class_data = {}
for k, v in reader:
   class_data[k] = v    

"""
Explanation: We need to identify each of the class names (whaleID) used, and - for each - create a folder for it, and save all examples into that folder. First we load in the full data linking image files to classes as an array, from the 'train.csv' file.
Unzip the image data
Import the file describing the image data
Iterate over the rows, creating a dictionary as you go.
End of explanation
"""


print(class_data['w_7812.jpg'])
# print(class_data['w_87.jpg'])  # not classified!

"""
Explanation: Here, the key is the filename, and the value is the ID. Some quick checks...
End of explanation
"""


print(len(class_data.keys()))

"""
Explanation: So when we find out how many classified images there are in this file...
End of explanation
"""


DIR = all_data_path + 'train/imgs'
print (len([name for name in os.listdir(DIR) if os.path.isfile(os.path.join(DIR, name))]))

"""
Explanation: Let's count the number of files under 'imgs' - it's over double the number of classified images!
End of explanation
"""


import os
import errno

def make_sure_path_exists(path):
    try:
        os.makedirs(path)
    except OSError as exception:
        if exception.errno != errno.EEXIST:
            raise

! tree -d data

from shutil import copyfile    # N.B. destination must be the complete target file name

"""
Explanation: A function for checking for a folder name and creating one if needed.
End of explanation
"""


# for each ID, create a folder with that name, and copy all the files keyed to that ID into this folder 
for filename in class_data:
    source = all_data_path + 'train/imgs/' + filename
    tgt_dir = all_data_path + 'train/' + class_data[filename]
    print('Checking for (possibly creating) folder: ' + tgt_dir + '... ')
    make_sure_path_exists(tgt_dir)
    # print('Copying file: ' + source + ' into folder: ' + tgt_dir)
    copyfile(source, tgt_dir + '/' + filename)     

"""
Explanation: Copy the training data into separate folders
For each ID, create a folder with that name, and copy all the files keyed to that ID into this folder.
NOTE that we use the information on whaleID to copy only those images that have been identified. Can't yet work out what I'd do with the others. Unsupervised learning?
End of explanation
"""


%ls $data_path

!unzip $all_data_path/imgs_subset.zip -d $all_data_path/sample

"""
Explanation: Unzip the provided 'subset' data sample file
End of explanation
"""


# for each ID, create a folder with that name, and copy all the files keyed to that ID into this folder 
src = sampled_path + 'imgs_subset/' 
g = glob(src +'w_*.jpg')
print(len(g))
print(g[:5])

sampled_path

"""
Explanation: Copy the 'subset' data sample into separate folders (fails - see below)
Get the names of the files in the subset
End of explanation
"""


print (g[1])
foo = g[1].replace("data/sample/imgs_subset/", "")
print (foo)

for src in g:
    filename = src.replace("data/sample/imgs_subset/", "")
    print(filename)
    tgt_dir = sampled_path + 'train/' + class_data[filename]
    print('Checking for (possibly creating) folder: ' + tgt_dir + '... ')
    # make_sure_path_exists(tgt_dir)
    print('Copying file: ' + src + ' into folder: ' + tgt_dir)
    # copyfile(src, tgt_dir + '/' + filename)     

"""
Explanation: It's clear that calling glob from a higher-level directory has preserved the local path to each file. We'll strip these to make the saving to folders process work. First, let's check we've got it right.
End of explanation
"""


%ls $all_data_path

!tree -d $all_data_path

src = os.path.join(all_data_path + 'train/imgs', '*.jpg')
print(src)
g = glob(src)
print(len(g))
shuf = np.random.permutation(g) 
print(g[:3])

tgt = os.path.join(sampled_path)
print (tgt)

"""
Explanation: So, it turns out that both image sets - the full set, and the sample - contain images (such as 'w_87.jpg') that don't have classification information provided. For the purposes of this experiment, it's going to be worth sampling from the already-copied set instead!
(Better:) Create a sample, from the full set, on which to test the code
Now we need to copy a random sample of the image files into the sample directories, so that we can test the code quickly.
First, a quick reminder of the file structure.
End of explanation
"""


src = os.path.join(all_data_path + 'train/imgs')
tgt = os.path.join(all_data_path + 'imgs')
%mv $src $tgt

"""
Explanation: On reflection, it seems less clear that there's a real benefit to running the training process on a sample, when there are so many classes to learn. For now, I'll skip this bit.
Moving the 'imgs' folder out of 'train'
I think this might be needed, to ensure that Keras doesn't think 'imgs' is a class.
End of explanation
"""


import vgg16               # this is the package
from vgg16 import Vgg16    # this is the class defining the model
vgg = Vgg16()                # instantiates a 'Vgg16' object (remember the parentheses!)

"""
Explanation: Setting up the model
A reminder of the steps to come:

set up the model:
establish which pre-trained model to use
import the model
load the pre-trained weights


set up for fine-tuning:
remove output layer
append new output layer (n classes)
set the batch size


train the network (sample data)

First, we create a Vgg16 object:
End of explanation
"""


src = os.path.join(all_data_path + 'train')
print(src)
batches = vgg.get_batches(path=src, batch_size=1)

"""
Explanation: And we grab some batches from the training folder.
End of explanation
"""


imgs,labels = next(batches)

plots(imgs, titles=labels)

"""
Explanation: Er. Woooo!
End of explanation
"""


