"""
Used methods:
* Convolutional Neural Network
* Data Augmentation

"""
"""
**Scientists want an automatic system to recognize whale species when monitoring their activities with a surveillance system. Thus, in this competition, numerous pictures of whales’ tales are given to identify whales species.

In the train set, there are 9850 images with 4251 classes, so the data set is highly class imbalance. For example, up to 2220 classes only have one samples in each class. Thus, data augmentation and different class weights in loss function are applied to mitigate the effect of class imbalance.**
In addition, all of images in train set are larger than common CNN input size, 224 x 224. For instance, the size of 11% of data are 1050 x 600. Therefore, Auto Encoder is applied to encode image from larger size to common CNN input size. Nevertheless, even though the performance of Auto Encoder is not bad, the improvement in all process is not obvious because of series class imbalance.

In the test set, there are 15610 images. In this competition, the method can predict 5 labels. The submission is evaluated with the Mean Average Precision.
"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

import os
print(os.listdir("../input"))

# Any results you write to the current directory are saved as output.
"""
> create a data frame named by df and store the image names and it's ID which we will be using it as our label later
"""
df = pd.read_csv('../input/train.csv')
df.head()
"""
Adding a column to our data frame named by imagepath and specify the exact image path in order to read the images through it as seen
if you are not familiar with lambda fn here http://book.pythontips.com/en/latest/lambdas.html
"""
df['imagepath'] = df["Image"].map(lambda x:'../input/train/'+x)
labels=df.Id #using the ids of whale as labels to our model
"""
> What is Categorical Data?
> Categorical data are variables that contain label values rather than numeric values.

The number of possible values is often limited to a fixed set.

Categorical variables are often called nominal.

Some examples include:

A “pet” variable with the values: “dog” and “cat“.
A “color” variable with the values: “red“, “green” and “blue“.
A “place” variable with the values: “first”, “second” and “third“.
Each value represents a different category.

Some categories may have a natural relationship to each other, such as a natural ordering.

The “place” variable above does have a natural ordering of values. This type of categorical variable is called an ordinal variable.

What is the Problem with Categorical Data?
Some algorithms can work with categorical data directly.

For example, a decision tree can be learned directly from categorical data with no data transform required (this depends on the specific implementation).

Many machine learning algorithms cannot operate on label data directly. They require all input variables and output variables to be numeric.

In general, this is mostly a constraint of the efficient implementation of machine learning algorithms rather than hard limitations on the algorithms themselves.

This means that categorical data must be converted to a numerical form. If the categorical variable is an output variable, you may also want to convert predictions by the model back into a categorical form in order to present them or use them in some application.

How to Convert Categorical Data to Numerical Data?
This involves two steps:

Integer Encoding
One-Hot Encoding
**1. Integer Encoding
**As a first step, each unique category value is assigned an integer value.

For example, “red” is 1, “green” is 2, and “blue” is 3.

This is called a label encoding or an integer encoding and is easily reversible.

For some variables, this may be enough.

The integer values have a natural ordered relationship between each other and machine learning algorithms may be able to understand and harness this relationship.

For example, ordinal variables like the “place” example above would be a good example where a label encoding would be sufficient.

**2. One-Hot Encoding
**For categorical variables where no such ordinal relationship exists, the integer encoding is not enough.

In fact, using this encoding and allowing the model to assume a natural ordering between categories may result in poor performance or unexpected results (predictions halfway between categories).

In this case, a one-hot encoding can be applied to the integer representation. This is where the integer encoded variable is removed and a new binary variable is added for each unique integer value.

In the “color” variable example, there are 3 categories and therefore 3 binary variables are needed. A “1” value is placed in the binary variable for the color and “0” values for the other colors.

For example:
------------------------------

red,	green,	blue
1,		0,		0
0,		1,		0
0,		0,		1
1
2
3
4
red,	green,	blue
1,		0,		0
0,		1,		0
0,		0,		1
"""
from sklearn import preprocessing
from keras.utils import np_utils

labels1= preprocessing.LabelEncoder()
labels1.fit(labels)
encodedlabels = labels1.transform(labels) #integer encoding
clearalllabels = np_utils.to_categorical(encodedlabels) #one hot encoding


df.head()
"""
**Here fun begins ,
**
we are creating a function with 2 arguments (imagepath and the image name) and this fn will read the image , check if the channels is more than 1 i.e the pictured is colored for the value 3 and convert it back to gray scale for the sake of memory , put it as float in order to normalize it in step 3,4 
**Normalization
**
the most commonly used normalization — making the data zero mean and unit variance along each feature. That is, given the data matrix X, where rows represent training instances and columns represent features, you compute the normalized matrix Xnorm with element (i,j) given by

Xnorm,(i,j)=X(i,j)−mean(Xj)std(Xj)

where Xj is the jth column of matrix X.

There are several advantages of doing that, many of which are interrelated:

Makes training less sensitive to the scale of features: Consider a regression problem where you’re given features of an apartment and are required to predict the price of the apartment. Let’s say there are 2 features — no. of bedrooms and the area of the apartment. Now, the no. of bedrooms will be in the range 1–4 typically, while the area will be in the range 100–200m2. If you’re modelling the task as linear regression, you want to solve for coefficients w1 and w2 corresponding to no. of bedrooms and area. Now, because of the scale of the features, a small change in w2 will change the prediction by a lot compared to the same change in w1, to the point that setting w2 correctly might dominate the optimization process.
Regularization behaves differently for different scaling: Suppose you have an ℓ2 regularization on the problem above. It is easy to see that ℓ2 regularization pushes larger weights towards zero more strongly than smaller weights. So consider that you obtain some optimal values of w1 and w2 using your given unnormalized data matrix X. Now instead of using m2 as the unit of area, if I change the data to represent area in ft2, the corresponding column of X will get multiplied by a factor of ~10. Therefore, you would expect the corresponding optimal coefficient w2 to go down by a factor of 10 to maintain the value of y. But, as stated before, the ℓ2 regularization now has a smaller effect because of the smaller value of the coefficient. So you will end up getting a larger value of w2 than you would have expected. This does not make sense — you did not change the information content of the data, and therefore, your optimal coefficients should not have changed.
Consistency for comparing results across models: As covered in point 2, scaling of features affects performance. So, if there are scientists developing new methods, and compare previous state-of-the-art methods with their new methods, which uses more carefully chosen scaling, then the results will not be reliable.
Makes optimization well-conditioned: Most machine learning optimizations are solved using gradient descent, or a variant thereof. And the speed of convergence depends on the scaling of features (or more precisely, the eigenvalues of XTX). Normalization makes the problem better conditioned, improving the convergence rate of gradient descent. I give an intuition of this using a simple example below.
Consider the simplest case where A is a 2 x 2 diagonal matrix, say A=diag([a1,a2]). Then, the contours of the objective function ∥Ax−b∥2 will be axis-aligned ellipses as shown in the figure below:


Suppose you start at the point marked in red. Observe that to reach the optimal point, you need to take a very large step in the horizontal direction but a small step in the vertical direction. The descent direction is given by the green arrow. If you go along this direction, then you will move larger distance in the vertical direction and smaller distance in the horizontal direction, which is the opposite of what you want to do!

If you take a small step along the gradient, covering the large horizontal distance to the optimal is going to take a large number of steps. If you take a large step along the gradient, you will overshoot the optimal in the vertical direction.

This behavior is due to the shape of the contours. The more circular the contours are, the faster you will converge to the optimal. The elongation of the ellipses is given by the ratio of the largest and the smallest eigenvalues of the matrix A. In general, the convergence of an optimization problem is measured by its condition number, which in this case is the ratio of the two extreme eigenvalues.

(Prasoon Goyal's answer to Why is the Speed Of Convergence of gradient descent depends on the maximal and minimal eigenvalues of A in solving AX=b through least squares.)

Finally, I should mention that normalization does not always help, as far as performance is concerned. Here's a simple example : consider a problem with only one feature with variance 1. Now suppose I add a dummy feature with variance 0.01. If you regularize your model correctly, the solution will not change much because of this dummy dimension. But if you now normalize it to have unit variance, it might hurt the performance.
----------------------------------------------------------------------------------------------
then creating a list and loop through a  loop with the length of training images saving these images to the list we have created
"""
import cv2
def imageProcessing(imagepath,name): 
    img=cv2.imread(imagepath)
    height,width,channels=img.shape
    if channels !=1:
        img = cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)
    img=img.astype(np.float)
    img = cv2.resize(img,(70,70))
    img = img-img.min()//img.max()-img.min()#step 3
    img=img*255#step 4
    return img
t=[]
    
for i in range(0,9850):
    t.append(imageProcessing(df.imagepath[i],df.Image[i]))
"""
Put t in the form of an array to be able to reshape it so we can feed it to the ImageGenerator that we will be discussing later
"""
t = np.asarray(t)
t=t.reshape(-1,70,70,1)
t.shape
#Same for test images
from glob import glob
path_to_images = '../input/test/*.jpg'
images=glob(path_to_images)
test=[]
for i in images:
    img = cv2.imread(i)
    height,width,channels=img.shape
    if channels !=1:
        img = cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)
    img=img.astype(np.float)
    img = cv2.resize(img,(70,70))
    img = img-img.min()//img.max()-img.min()
    img=img*255

    test.append(cv2.resize(img,(70,70)))
    # Get image label (folder name)

t.shape
test=np.asarray(test)
test.shape

"""
* **Data Augmentation**
Overfitting is caused by having too few samples to learn from, rendering you unable to train a model that can generalize to new data. Given infinite data, your model would be exposed to every possible aspect of the data distribution at hand: you would
never overfit. Data augmentation takes the approach of generating more training data
from existing training samples, by augmenting the samples via a number of random transformations that yield believable-looking images. The goal is that at training time,
your model will never see the exact same picture twice. This helps expose the model
to more aspects of the data and generalize better.
In Keras, this can be done by configuring a number of random transformations to
be performed on the images read by the ImageDataGenerator instance. Let’s get
started with an this example
--------------------------------------------------------

*These are just a few of the options available (for more, see the Keras documentation).
*Let’s quickly go over this code:
 rotation_range is a value in degrees (0–180), a range within which to randomly
rotate pictures.
 width_shift and height_shift are ranges (as a fraction of total width or
height) within which to randomly translate pictures vertically or horizontally.
 shear_range is for randomly applying shearing transformations.
 zoom_range is for randomly zooming inside pictures.
 horizontal_flip is for randomly flipping half the images horizontally—relevant
when there are no assumptions of horizontal asymmetry (for example,
real-world pictures).
 fill_mode is the strategy used for filling in newly created pixels, which can
appear after a rotation or a width/height shift.

"""
from keras.preprocessing.image import ImageDataGenerator
image_gen = ImageDataGenerator(
    #featurewise_center=True,
    #featurewise_std_normalization=True,
    rescale=1./255,
    rotation_range=15,
    width_shift_range=.15,
    height_shift_range=.15,
    horizontal_flip=True)
#training the image preprocessing
image_gen.fit(t) # fit t to the imageGenerator to let the magic happen

t.shape
clearalllabels.shape
test.shape
"""
This is the model we used for the classification process and it's inspired from VGG16 architecture , for more info i recommend reading deep learning with keras-packt by Antonio Gulli and for any info about the model i recommend visiting https://keras.io/
"""
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten,ZeroPadding2D
from keras.layers import Conv2D, MaxPooling2D,BatchNormalization,Activation
from keras import optimizers

model = Sequential()
model.add(ZeroPadding2D((1,1),input_shape=(70,70,1)))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(64,(3,3),activation='relu'))
model.add(MaxPooling2D((2,2),strides=(2,2)))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(128,(3,3),activation='relu'))
model.add(MaxPooling2D((2,2),strides=(2,2)))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(256,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(256,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(256,(3,3),activation='relu'))
model.add(MaxPooling2D((2,2),strides=(2,2)))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512,(3,3),activation='relu'))
model.add(MaxPooling2D((2,2),strides=(2,2)))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512,(3,3),activation='relu'))
model.add(ZeroPadding2D((1,1)))
model.add(Conv2D(512,(3,3),activation='relu'))
model.add(MaxPooling2D((2,2),strides=(2,2)))
model.add(Flatten())
model.add(Dense(4096,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(4096,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(4251,activation='softmax'))

optimizer=optimizers.SGD()


model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])
model.summary()

"""
Fitting the training data with it's labels , batch size is the number of instance computed at once before the weights are updated , steps per epochs=9850 because it's the data length so we have to pass over all the data for each epoch to see every possible combination , by 10th epoch it reaches like 94% accuracy on training data 
"""
#model.fit_generator(image_gen.flow(t,clearalllabels, batch_size=128),steps_per_epoch=  9850,epochs=10,verbose=1)

"""
Downsides : time of course it took hours to run 
**thoughts**
for data augmentation i could have used flow_from_directory but it would have taken time to mkdirs with all the labels inside the training directory ,see https://keras.io/preprocessing/image/ 
-------------------------------------------------------------------------
for model selection i could have used a pretrained model but it failed because of memory restrictions on kernels, see https://keras.io/applications/#resnet50
--------------------------------------------------------------------------
i couldn't use train_test_split and aside a validation data to check the performance of the model on it but because of our imbalanced data i couldn't as for some classes there is only 1 sample for 
**next step to predict on test data and submit to kaggle**

I hope i benefited you by walking you through my journey which i found amazing and full of knowledge , if you found this kernel useful please upvote , i will be doing some kernels on every idea i learn and find amazing
*************thanks******************
"""