from skimage.io import imread

image = imread('laptop.jpeg')
type(image)

"""
Explanation: Loading a JPEG file as a numpy array
Let's use scikit-image to load the content of a JPEG file into a numpy array:
End of explanation
"""


image.shape

"""
Explanation: The dimensions of the array are:
- height
- width
- color channels (RGB)
End of explanation
"""


image.dtype

image.min(), image.max()

plt.imshow(image);

"""
Explanation: For efficiency reasons, the pixel intensities of each channel are stored as 8-bit unsigned integer taking values in the [0-255] range:
End of explanation
"""


np.product(image.shape)

450 * 800 * 3 * (8 / 8)

"""
Explanation: Size of a numpy array
The size in bytes can be computed by multiplying the number of element by the size in byte of each element in the array.
The size of one element depend of the data type.
1 byte == 8 bits
A byte in English is an octet in French.
End of explanation
"""


print("image size: {:0.3} MB".format(image.nbytes / 1e6))

"""
Explanation: Let's check by asking numpy:
End of explanation
"""


red_channel = image[:, :, 0]
red_channel

plt.imshow(image[:, :, 0], cmap=plt.cm.Reds_r);

"""
Explanation: Indexing on the last dimension makes it possible to extract the 2D content of a specific color channel, for instance the red channel:
End of explanation
"""


# %load solutions/grey_levels.py

"""
Explanation: Exercise


Compute a grey-level version of the image with shape (height, width) by averaging the values across color channels using image.mean.


Plot the result with plt.imshow using a grey levels colormap.


Can the uint8 integer data type represent those average values? Check the data type used by numpy.


What is the size in (mega) bytes of this image? 


What are the expected range of values for the new pixels?
End of explanation
"""


from skimage.transform import resize

image = imread('laptop.jpeg')
lowres_image = resize(image, (50, 50), mode='reflect', anti_aliasing=True)
lowres_image.shape

plt.imshow(lowres_image, interpolation='nearest');

"""
Explanation: Resizing images, handling data types and dynamic ranges
When dealing with an heterogeneous collection of image of various sizes, it is often necessary to resize the image to the same size. More specifically:


for image classification, most networks expect a specific fixed input size;


for object detection and instance segmentation, networks have more flexibility but the image should have approximately the same size as the training set images.


Furthermore large images can be much slower to process than smaller images (the number of pixels varies quadratically with the height and width).
End of explanation
"""


lowres_image.dtype

"""
Explanation: The values of the pixels of the low resolution image are computed from by combining the values of the pixels in the high resolution image. The result is therefore represented as floating points.
End of explanation
"""


lowres_image.min(), lowres_image.max()

"""
Explanation: By conventions, both skimage.transform.imresize and plt.imshow assume that floating point values range from 0.0 to 1.0 when using floating points as opposed to 0 to 255 when using 8-bit integers:
End of explanation
"""


lowres_large_range_image = resize(image, (50, 50), mode='reflect',
                                  anti_aliasing=True, preserve_range=True)

lowres_large_range_image.shape

lowres_large_range_image.dtype

lowres_large_range_image.min(), lowres_large_range_image.max()

"""
Explanation: Note that keras on the other hand might expect images encoded with values in the [0.0 - 255.0] range irrespectively of the data type of the array. To avoid the implicit conversion to the [0.0 - 1.0] range we use the preserve_range=True option.
End of explanation
"""


plt.imshow(lowres_large_range_image, interpolation='nearest');

"""
Explanation: Warning: the behavior of plt.imshow depends on both the dtype and the dynamic range when displaying RGB images. In particular it does not work on RGB images with float64 values in the [0.0 - 255.0] range:
End of explanation
"""


# %load solutions/question_imshow_dtype_and_range.py

"""
Explanation: Question
Suggest two possible ways to correctly display an RGB array with floating point values in the [0.0 - 255.0] range:
End of explanation
"""


import tensorflow as tf
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing import image

model = ResNet50(include_top=True, weights='imagenet')

print(model.summary())

"""
Explanation: Using a pretrained model
Objectives:

Load a pre-trained ResNet50 pre-trained model using Keras Zoo
Build a headless model and compute representations of images
Explore the quality of representation with t-SNE
Retrain last layer on cat vs dog dataset
End of explanation
"""


from tensorflow.keras.applications.imagenet_utils import preprocess_input
from tensorflow.keras.applications.imagenet_utils import decode_predictions

path = "laptop.jpeg"

# TODO

# %load solutions/predict_image.py

"""
Explanation: Classification of an image
Exercise
- Open an image, preprocess it and build a batch of 1 image
- Use the model to classify this image
- Decode the predictions using decode_predictions from Keras
Notes:
- Test your code with "images_resize/000007.jpg"
- You may need preprocess_input for preprocessing the image. 
- The Keras resnet expects floating point images of size (224, 224) with a dynamic in [0, 255] before preprocessing. skimage's resize has a preserve_range flag that you might find useful.
End of explanation
"""


import cv2

def camera_grab(camera_id=0, fallback_filename=None):
    camera = cv2.VideoCapture(camera_id)
    try:
        # take 10 consecutive snapshots to let the camera automatically tune
        # itself and hope that the contrast and lighting of the last snapshot
        # is good enough.
        for i in range(10):
            snapshot_ok, image = camera.read()
        if snapshot_ok:
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        else:
            print("WARNING: could not access camera")
            if fallback_filename:
                image = imread(fallback_filename)
    finally:
        camera.release()
    return image

image = camera_grab(camera_id=0, fallback_filename='laptop.jpeg')
plt.imshow(image)
print("dtype: {}, shape: {}, range: {}".format(
    image.dtype, image.shape, (image.min(), image.max())))


"""
Explanation: Taking snapshots from the webcam
Let's use the python API of OpenCV to take pictures.
End of explanation
"""


def classify():
    # TODO: write me
    pass

    
classify()

# %load solutions/classify_webcam.py

"""
Explanation: Exercise


Write a function named classify that takes a snapshot of the webcam and displays it along with the decoded predictions of model and their confidence level.


If you don't have access to a webcam take a picture with your mobile phone or a photo of your choice from the web, store it as a JPEG file on the disk instead and pass that file to the neural network to make the prediction.


Try to classify a photo of your face. Look at the confidence level. Can you explain the results?


Try to classify photos of common objects such as a book, a mobile phone, a cup... Try to center the objects and remove clutter to get confidence higher than 0.5.
End of explanation
"""


import os.path as op
from zipfile import ZipFile

if not op.exists("images_resize"):
    print('Extracting image files...')
    zf = ZipFile('images_pascalVOC.zip')
    zf.extractall('.')

"""
Explanation: Computing the representations of images
First let's make sure that we have access to a subset of image files from the PASCAL VOC dataset we'll be using
End of explanation
"""


input = model.layers[0].input
output = model.layers[-2].output
base_model = Model(input, output)
base_model.output_shape

"""
Explanation: Let's build a new model that maps the image input space to the output of the layer before the last layer of the pretrained Resnet 50 model. We call this new model the "base model":
End of explanation
"""


representation = base_model.predict(img_batch)
print("Shape of representation:", representation.shape)

"""
Explanation: The base model can transform any image into a flat, high dimensional, semantic feature vector:
End of explanation
"""


import os
paths = ["images_resize/" + path
         for path in sorted(os.listdir("images_resize/"))]

import h5py

with h5py.File('img_emb.h5', 'r') as h5f:
    out_tensors = h5f['img_emb'][:]
    
out_tensors.shape

out_tensors.dtype

"""
Explanation: Computing representations of all images can be time consuming.
This is usually made by large batches on a GPU for massive performance gains.
For the remaining part, we will use pre-computed representations saved in h5 format.
For those interested, this is done using the process_images.py script
End of explanation
"""


# %load solutions/representations.py

"""
Explanation: Exercise
- What is the proportion of 0 values in this representation?
- Can you find any negative values?
- Why are there so many zero values?
- Are the zero always located in the same dimensions for different input images?
End of explanation
"""


from sklearn.manifold import TSNE

img_emb_tsne = TSNE(perplexity=30).fit_transform(out_tensors)

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 10))
plt.scatter(img_emb_tsne[:, 0], img_emb_tsne[:, 1]);
plt.xticks(()); plt.yticks(());
plt.show()

"""
Explanation: Let's find a 2D representation of that high dimensional feature space using T-SNE:
End of explanation
"""


import matplotlib
import matplotlib.pyplot as plt
from matplotlib.offsetbox import OffsetImage, AnnotationBbox
from skimage.io import imread
from skimage.transform import resize

def imscatter(x, y, paths, ax=None, zoom=1, linewidth=0):
    if ax is None:
        ax = plt.gca()
    x, y = np.atleast_1d(x, y)
    artists = []
    for x0, y0, p in zip(x, y, paths):
        try:
            im = imread(p)
        except:
            print(p)
            continue
        im = resize(im, (224, 224), preserve_range=False, mode='reflect')
        im = OffsetImage(im, zoom=zoom)
        ab = AnnotationBbox(im, (x0, y0), xycoords='data',
                            frameon=True, pad=0.1, 
                            bboxprops=dict(edgecolor='red',
                                           linewidth=linewidth))
        artists.append(ax.add_artist(ab))
    ax.update_datalim(np.column_stack([x, y]))
    ax.autoscale()
    return artists

fig, ax = plt.subplots(figsize=(50, 50))
imscatter(img_emb_tsne[:, 0], img_emb_tsne[:, 1], paths, zoom=0.5, ax=ax)
plt.savefig('tsne.png')

"""
Explanation: Let's add thumnails of the original images at their TSNE locations:
End of explanation
"""


def display(img):
    plt.figure()
    img = imread(img)
    plt.imshow(img)

idx = 57

def most_similar(idx, top_n=5):
    dists = np.linalg.norm(out_tensors - out_tensors[idx], axis = 1)
    sorted_dists = np.argsort(dists)
    return sorted_dists[:top_n]

sim = most_similar(idx)
[display(paths[s]) for s in sim];

"""
Explanation: Visual Search: finding similar images
End of explanation
"""


out_norms = np.linalg.norm(out_tensors, axis=1, keepdims=True)
normed_out_tensors = out_tensors / out_norms

item_idx = 208
dists_to_item = np.linalg.norm(out_tensors - out_tensors[item_idx],
                               axis=1)
cos_to_item = np.dot(normed_out_tensors, normed_out_tensors[item_idx]) 
plt.hist(cos_to_item, bins=30)
display(paths[item_idx])

"""
Explanation: Bonus: Classification from Nearest Neighbors?
Using these representations, it may be possible to build a nearest neighbor classifier. However, the representations are learnt on ImageNet, which are centered images, when we input images from PascalVOC, more plausible inputs of a real world system.
The next section explores this possibility by computing the histogram of similarities between one image and the others.
End of explanation
"""


items = np.where(cos_to_item > 0.5)
print(items)
[display(paths[s]) for s in items[0]];

"""
Explanation: Unfortunately there is no clear separation of class boundaries visible in the histogram of similarities alone. We need some supervision to be able to classify images.
With a labeled dataset, even with very little labels per class, one would be able to:
- build a k-Nearest Neighbor model,
- build a classification model such as a SVM.
These approximate classifiers are useful in practice.
See the cat vs dog home assignment with GPU for another example of this approach.
End of explanation
"""


