# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="RHA88Oo_1JEz" colab_type="text"
# # Data Processing
#
# - Load image data using PyTorch
# - Image transformations
# - Preprocess images (resize, crop, normalize)

# + [markdown] id="MAerAjY6aeYg" colab_type="text"
# ### Setup drive

# + [markdown] id="BY8phsHYdicZ" colab_type="text"
# Run the following cell to mount your Drive onto Colab. Go to the given URL and once you login and copy and paste the authorization code, you should see "drive" pop up in the files tab on the left.

# + id="hurAYstF1TVc" colab_type="code" colab={}
from google.colab import drive
drive.mount('/content/drive')

# + [markdown] id="RzP9vRERdmst" colab_type="text"
# Click the little triangle next to "drive" and navigate to the "AI4All Chest X-Ray Project" folder. Hover over the folder and click the 3 dots that appear on the right. Select "copy path" and replace `PASTE PATH HERE` with the path to your folder.

# + id="9kac1-X-cXhZ" colab_type="code" colab={}
# cd "PASTE PATH HERE"

# + [markdown] id="YoRfQsJE493A" colab_type="text"
# ### Import necessary libraries
# Torchvision, or the PyTorch package, consists of popular datasets, model architectures, and common image transformations for computer vision.

# + id="y7qYM0a61JE2" colab_type="code" colab={}
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import random

from torch.utils.data import random_split, Subset

import torchvision
from torchvision import datasets, transforms

from utils.plotting import imshow_dataset
from utils.datahelper import calc_dataset_stats, get_random_image

# + [markdown] id="FlNKHL6V1JFD" colab_type="text"
# ### Setup paths
# Define paths and load metadata

# + id="JurkCoKd1JFF" colab_type="code" colab={}
path_to_dataset = os.path.join('data')

path_to_images = os.path.join(path_to_dataset, 'images')

metadata = pd.read_csv(os.path.join(path_to_dataset, 'metadata_train.csv'))

# + [markdown] id="RDefUUni1JFO" colab_type="text"
# ### Load images

# + [markdown] id="NWu981cD1JFb" colab_type="text"
# **Pytorch loads the data using sub-folder names as class labels**
#
# Navigate to the "images" folder to see what this means.
#

# + id="7K9vxOog8phL" colab_type="code" colab={}
dataset = datasets.ImageFolder(path_to_images, transform=None)
dataset

# + id="_jl4WNZ41JFo" colab_type="code" colab={}
# EXERCISE: Use the function .class_to_idx to see what our classes are


# + [markdown] id="WpUeGWTj83pJ" colab_type="text"
# **Now let's take a look at the images themselves!**
#
# Note: The `imshow_dataset` function is defined in the file `utils/plotting.py`.

# + id="4-5t6y8x1JFP" colab_type="code" colab={}
# plots the first 5 images
imshow_dataset(dataset, n=5)

# + id="4-eKm8oweTiY" colab_type="code" colab={}
# plots 5 random images
imshow_dataset(dataset, n=5, rand=True)

# + [markdown] id="fng1sPXc9Jak" colab_type="text"
# > **Discuss with each other**
# >
# > What do you notice about the images? What are their dimensions?
#
#

# + [markdown] id="aC0ylzSJ_ilu" colab_type="text"
# ### Transformations
# The transforms module in PyTorch defines various transformations that can be performed on an image. 
#
# Image transformations are used to pre-process images
# as well as to "augment" the data. (We will discuss data augmentation in another section.)
#

# + [markdown] id="E77sTYbmhxUY" colab_type="text"
# **Resize the image using transforms**

# + id="TxVrO7is_tnn" colab_type="code" colab={}
# get a random image from the dataset and resize it
im = get_random_image(dataset)
im = transforms.Resize(100)(im)
im

# + id="IzaCwPplAnS_" colab_type="code" colab={}
transforms.Resize(50)(im)

# + [markdown] id="9kgklEULi8ac" colab_type="text"
# **Try out other transformations**
#
# How do these transformations alter the image?
# - `transforms.ColorJitter`
# - `transforms.RandomAffine`
# - `transforms.RandomHorizontalFlip`  
#
# You can [read more about these transformations here](https://pytorch.org/docs/stable/torchvision/transforms.html)
#
#
#

# + id="wUfx08Koi6SV" colab_type="code" colab={}
# EXERCISE: Apply different transformations to images and check out the output
#
# HINT: Use the code above as an example and try transforms functions such as RandomAffine



# + [markdown] id="UPdzvLOgljle" colab_type="text"
# > **Discuss with each other**
# > 
# > Which transformations could be useful to normalize the dataset? Which transformations could be useful to add diversity to data set?

# + [markdown] id="nvscfnfU1JFu" colab_type="text"
# ### Examine image dimensions

# + [markdown] id="bDKayhybj9wJ" colab_type="text"
# Run the code below to calculate the image dimension.
#
# > **Discuss with each other**
# >
# > Based on the image dimension, are the images greyscale or color images?

# + id="tKuIS_OF1JFv" colab_type="code" colab={}
im_sizes = [d[0].size for d in dataset]

dimensions = set([len(s) for s in im_sizes])

print(f'Dimensions in dataset: {dimensions}')

# + [markdown] id="CkWTitgFkXjq" colab_type="text"
# Compare x-ray images to another image

# + id="kfqdczpxbOLq" colab_type="code" colab={}
# Answer the above question before running this block!

from skimage import io
color_image = io.imread('https://unsplash.com/photos/twukN12EN7c/download')
io.imshow(color_image)
print(f'Random color image shape: {color_image.shape}')
print(f'Random xray image shape: {get_random_image(dataset).size}')

# + [markdown] id="lwBVMapR1JF5" colab_type="text"
# **How much do image shapes and sizes vary in the dataset?**
#
# Run the code below to print the image dimensions for a set of random images

# + id="EeawbFJC1JF6" colab_type="code" colab={}
im_num = 10
rand_indices = random.sample(range(len(dataset)), im_num)
subset = Subset(dataset, rand_indices)

print(f'Image dimensions for {im_num} random images')
for d in subset:
    print(d[0].size)

# + [markdown] id="UO6YUmYBrt5N" colab_type="text"
# **Smallest dimension measurements**
#
# Calculate the smallest image width and height in the dataset.

# + id="NwR2WGDir1tp" colab_type="code" colab={}
# EXERCISE: calculate the smallest image width and smallest image height in the
# dataset
#
# HINT: look at blocks above for useful code, use min() to find minimum in a list


# + [markdown] id="SFMboM51mNWD" colab_type="text"
# > **Discuss with each other**
# > 
# > How should we resize and crop the images? How do the smallest image width and smallest image height constrain our strategy?

# + [markdown] id="w2zW9QWB1JGN" colab_type="text"
# ### Resize and crop

# + [markdown] id="83D3utI1mFGh" colab_type="text"
# To make the images the same shape and size for the learning model, we can apply image transformations when loading the data.
#
# The `transforms.Compose` function puts together a list of image transformations, which are applied in order to the images.

# + id="DzyqD6yJ1JGO" colab_type="code" colab={}
# EXERCISE: set resize and crop parameters based on your observations above

resize_value = # HERE #
crop_value = # HERE #


# compose transformations
data_transforms = transforms.Compose([
        transforms.Resize(resize_value),
        transforms.CenterCrop(crop_value)])

dataset = datasets.ImageFolder(path_to_images, transform=data_transforms)

# + id="X4ZiIi7Lmv4e" colab_type="code" colab={}
# EXERCISE: compare the images with and without transformation applied. 



# + id="0C8N8lRZm9FC" colab_type="code" colab={}
# EXERCISE: try applying another list of transformations and compare the results



# + [markdown] id="u856kmgx1JGT" colab_type="text"
# ### Normalize images

# + [markdown] id="q7UI9MUn1JGW" colab_type="text"
# **Calculate the pixel intensity mean and standard deviation across all images in the dataset.**
#
# Note: This code takes some time to run. The output is 
#
# - Mean: 0.544
# - Standard Deviation: 0.237

# + id="tORr1xbH1JGX" colab_type="code" colab={}
data_transforms = transforms.Compose([
        transforms.Resize(resize_value),
        transforms.CenterCrop(crop_value),
        transforms.ToTensor()])

dataset = datasets.ImageFolder(path_to_images, transform=data_transforms)

data_mean, data_std = calc_dataset_stats(dataset)
print(f'Mean: {data_mean:.3f}, Standard Deviation: {data_std:.3f}')

# + [markdown] id="t-PKVJ1R1JGg" colab_type="text"
# **Add normalization to the transformation list**
#
# The normalization step is applied on tensors and so is added after the `transforms.ToTensor` step. 

# + id="nW9X6mR31JGh" colab_type="code" colab={}
data_transforms = transforms.Compose([
        transforms.Grayscale(),
        transforms.Resize(resize_value),
        transforms.CenterCrop(crop_value),
        transforms.ToTensor(),
        transforms.Normalize(mean=[data_mean], std=[data_std])])

dataset = datasets.ImageFolder(path_to_images, transform=data_transforms)

# + id="ElIWlR4dp7zf" colab_type="code" colab={}
# EXERCISE: compare the images with all the transformations applied.



# + id="MP4a2Uly1JGs" colab_type="code" colab={}


StarcoderdataJupyterScriptsDedupFiltered