"""
All data from book **Deep Learning with PyTorch** https://pytorch.org/deep-learning-with-pytorch
"""
"""
https://github.com/deep-learning-with-pytorch/dlwpt-code
"""
"""
* https://www.kaggle.com/dmisky/dlwpt-p1ch2-dog-detection

* https://www.kaggle.com/dmisky/dlwpt-p1ch2-gan-horse-zebra

* https://www.kaggle.com/dmisky/dlwpt-p1ch3-tensors

* https://www.kaggle.com/dmisky/dlwpt-p1ch4-working-with-images

* https://www.kaggle.com/dmisky/dlwpt-p1ch4-3d-images-volumetric-data
"""
import numpy as np

import csv

import torch
torch.set_printoptions(edgeitems=2, precision=2, linewidth=75)
"""
# Representing tabular data
"""
"""
The simplest form of data we’ll encounter on a machine learning job is sitting in a
spreadsheet, CSV file, or database. Whatever the medium, it’s a table containing one
row per sample (or record), where columns contain one piece of information about
our sample.
"""
"""
At first we are going to assume there’s no meaning to the order in which samples
appear in the table: such a table is a collection of independent samples, unlike a time
series, for instance, in which samples are related by a time dimension.
"""
"""
Columns may contain numerical values, like temperatures at specific locations; or
labels, like a string expressing an attribute of the sample, like “blue.” Therefore, tabu-
lar data is typically not homogeneous: different columns don’t have the same type. We
might have a column showing the weight of apples and another encoding their color
in a label.
"""
"""
PyTorch tensors, on the other hand, are homogeneous. Information in PyTorch is
typically encoded as a number, typically floating-point (though integer types and
Boolean are supported as well). This numeric encoding is deliberate, since neural
networks are mathematical entities that take real numbers as inputs and produce real
numbers as output through successive application of matrix multiplications and
nonlinear functions.
"""
"""
## Using a real-world dataset
"""
"""
Our first job as deep learning practitioners is to encode heterogeneous, real-world
data into a tensor of floating-point numbers, ready for consumption by a neural network. A large number of tabular datasets are freely available on the internet; see, for
instance, https://github.com/caesar0301/awesome-public-datasets. Let’s start with
something fun: wine! The Wine Quality dataset is a freely available table containing
chemical characterizations of samples of vinho verde, a wine from north Portugal,
together with a sensory quality score. The dataset for white wines can be downloaded
here: http://mng.bz/90Ol. For convenience, we also created a copy of the dataset on
the Deep Learning with PyTorch Git repository, under data/p1ch4/tabular-wine.
"""
"""
The file contains a comma-separated collection of values organized in 12 columnsmpreceded by a header line containing the column names. The first 11 columns contain values of chemical variables, and the last column contains the sensory quality score from 0 (very bad) to 10 (excellent). These are the column names in the order they appear in the dataset:
"""
"""
* fixed acidity
* volatile acidity
* citric acid
* residual sugar
* chlorides
* free sulfur dioxide
* total sulfur dioxide
* density
* pH
* sulphates
* alcohol
* quality
"""
"""
A possible machine learning task on this dataset is predicting the quality score from chemical characterization alone. Don’t worry, though; machine learning is not going to kill wine tasting anytime soon. We have to get the training data from somewhere! As we can see in figure 4.3, we’re hoping to find a relationship between one of the chemical columns in our data and the quality column. Here, we’re expecting to see quality increase as sulfur decreases.
"""
"""
![image.png](attachment:image.png)
"""
"""
## Loading a wine data tensor
"""
"""
* The csv module that ships with Python
* NumPy
* Pandas
"""
"""
The third option is the most time- and memory-efficient. However, we’ll avoid introducing an additional library in our learning trajectory just because we need to load a file. Since we already introduced NumPy in the previous section, and PyTorch has excellent NumPy interoperability, we’ll go with that. Let’s load our file and turn the resulting NumPy array into a PyTorch tensor (code/p1ch4/3_tabular_wine.ipynb).
"""
wine_path = "/kaggle/input/winequality-white.csv"
wineq_numpy = np.loadtxt(wine_path, dtype=np.float32, delimiter=";", skiprows=1)

wineq_numpy
"""
Here we just prescribe what the type of the 2D array should be (32-bit floating-point), the delimiter used to separate values in each row, and the fact that the first line should not be read since it contains the column names. Let’s check that all the data has been read
"""
col_list = next(csv.reader(open(wine_path), delimiter=';'))

wineq_numpy.shape, col_list
"""
and proceed to convert the NumPy array to a PyTorch tensor:
"""
wineq = torch.from_numpy(wineq_numpy)

wineq.shape, wineq.dtype
"""
At this point, we have a floating-point torch.Tensor containing all the columns, including the last, which refers to the quality score.
"""
"""
**Continuous, ordinal, and categorical values**

"""
"""
We should be aware of three different kinds of numerical values as we attempt to make sense of our data. The first kind is continuous values. These are the most intuitive when represented as numbers. They are strictly ordered, and a difference between various values has a strict meaning. Stating that package A is 2 kilograms heavier than package B, or that package B came from 100 miles farther away than A has a fixed meaning, regardless of whether package A is 3 kilograms or 10, or if B came from 200 miles away or 2,000. If you’re counting or measuring something with units, it’s probably a continuous value. The literature actually divides continuous values further: in the previous examples, it makes sense to say something is twice as heavy or three times farther away, so those values are said to be on a ratio scale. The time of day, on the other hand, does have the notion of difference, but it is not reasonable to claim that 6:00 is twice as late as 3:00; so time of day only offers an interval scale.
"""
"""
Next we have ordinal values. The strict ordering we have with continuous values remains, but the fixed relationship between values no longer applies. A good example of this is ordering a small, medium, or large drink, with small mapped to the value 1, medium 2, and large 3. The large drink is bigger than the medium, in the same way that 3 is bigger than 2, but it doesn’t tell us anything about how much bigger. If we were to convert our 1, 2, and 3 to the actual volumes (say, 8, 12, and 24 fluid ounces), then they would switch to being interval values. It’s important to remember that we can’t “do math” on the values outside of ordering them; trying to average large = 3 and small = 1 does not result in a medium drink!
"""
"""
Finally, categorical values have neither ordering nor numerical meaning to their values. These are often just enumerations of possibilities assigned arbitrary numbers. Assigning water to 1, coffee to 2, soda to 3, and milk to 4 is a good example. There’s no real logic to placing water first and milk last; they simply need distinct values to differentiate them. We could assign coffee to 10 and milk to –3, and there would be no significant change (though assigning values in the range 0.. N – 1 will have advantages for one-hot encoding and the embeddings we’ll discuss in section 4.5.4.) Because the numerical values bear no meaning, they are said to be on a nominal scale.
"""
"""
As a starting point for a more in-depth discussion, refer to https://en.wikipedia.org/wiki/Level_of_measurement.
"""
"""
## Representing scores
"""
"""
We could treat the score as a continuous variable, keep it as a real number, and perform a regression task, or treat it as a label and try to guess the label from the chemical analysis in a classification task. In both approaches, we will typically remove the score from the tensor of input data and keep it in a separate tensor, so that we can use the score as the ground truth without it being input to our model:
"""
data = wineq[:, :-1]  # Selects all rows and all columns except the last
data, data.shape
target = wineq[:, -1]  # Selects all rows and the last column
target, target.shape
"""
If we want to transform the target tensor in a tensor of labels, we have two options, depending on the strategy or what we use the categorical data for. One is simply to treat labels as an integer vector of scores:
"""
target = wineq[:, -1].long()
target
"""
If targets were string labels, like wine color, assigning an integer number to each string
would let us follow the same approach.
"""
"""
## One-hot encoding
"""
"""
The other approach is to build a one-hot encoding of the scores: that is, encode each of the 10 scores in a vector of 10 elements, with all elements set to 0 but one, at a different index for each score. This way, a score of 1 could be mapped onto the vector (1,0,0,0,0,0,0,0,0,0) , a score of 5 onto (0,0,0,0,1,0,0,0,0,0) , and so on. Note that the fact that the score corresponds to the index of the nonzero element is purely incidental: we could shuffle the assignment, and nothing would change from a classification standpoint.
"""
"""
There’s a marked difference between the two approaches. Keeping wine quality scores in an integer vector of scores induces an ordering on the scores—which might be totally appropriate in this case, since a score of 1 is lower than a score of 4. It also induces some sort of distance between scores: that is, the distance between 1 and 3 is the same as the distance between 2 and 4. If this holds for our quantity, then great. If, on the other hand, scores are purely discrete, like grape variety, one-hot encoding will be a much better fit, as there’s no implied ordering or distance. One-hot encoding is also appropriate for quantitative scores when fractional values in between integer scores, like 2.4, make no sense for the application—for when the score is either this or that.
"""
"""
We can achieve one-hot encoding using the scatter_ method, which fills the ten-
sor with values from a source tensor along the indices provided as arguments:
"""
target_onehot = torch.zeros(target.shape[0], 10)

target_onehot.scatter_(1, target.unsqueeze(1), 1.0)
"""
Let’s see what scatter_ does. First, we notice that its name ends with an underscore.
As you learned in the previous chapter, this is a convention in PyTorch that indicates
the method will not return a new tensor, but will instead modify the tensor in place.
The arguments for scatter_ are as follows:
* The dimension along which the following two arguments are specified
* A column tensor indicating the indices of the elements to scatter
* A tensor containing the elements to scatter or a single scalar to scatter (1, in
this case)
"""
"""
In other words, the previous invocation reads, “For each row, take the index of the target label (which coincides with the score in our case) and use it as the column index to set the value 1.0.” The end result is a tensor encoding categorical information.
"""
"""
The second argument of scatter_ , the index tensor, is required to have the same
number of dimensions as the tensor we scatter into. Since target_onehot has two
dimensions (4,898 × 10), we need to add an extra dummy dimension to target using
unsqueeze :
"""
target_unsqueezed = target.unsqueeze(1)
target_unsqueezed
"""
The call to unsqueeze adds a singleton dimension, from a 1D tensor of 4,898 elements
to a 2D tensor of size (4,898 × 1), without changing its contents—no extra elements
are added; we just decided to use an extra index to access the elements. That is, we
access the first element of target as target[0] and the first element of its
unsqueezed counterpart as target_unsqueezed[0,0] .
"""
"""
PyTorch allows us to use class indices directly as targets while training neural networks. However, if we wanted to use the score as a categorical input to the network, we would have to transform it to a one-hot-encoded tensor.
"""
"""
## When to categorize
"""
"""
Now we have seen ways to deal with both continuous and categorical data. You may wonder what the deal is with the ordinal case discussed in the earlier sidebar. There is no general recipe for it; most commonly, such data is either treated as categorical (losing the ordering part, and hoping that maybe our model will pick it up during training if we only have a few categories) or continuous (introducing an arbitrary notion of distance). We will do the latter for the weather situation in figure 4.5. We summarize our data mapping in a small flow chart in figure 4.4.
"""
"""
![image.png](attachment:image.png)
"""
"""
Let’s go back to our data tensor, containing the 11 variables associated with the chemical analysis. We can use the functions in the PyTorch Tensor API to manipulate our data in tensor form. Let’s first obtain the mean and standard deviations for each column:
"""
data_mean = torch.mean(data, dim=0)
data_mean
data_var = torch.var(data, dim=0)
data_var
"""
In this case, dim=0 indicates that the reduction is performed along dimension 0. At this point, we can normalize the data by subtracting the mean and dividing by the standard deviation, which helps with the learning process (we’ll discuss this in more detail in chapter 5, in section 5.4.4):
"""
data_normalized = (data - data_mean) / torch.sqrt(data_var)
data_normalized
"""
## Finding thresholds
"""
"""
Next, let’s start to look at the data with an eye to seeing if there is an easy way to tell good and bad wines apart at a glance. First, we’re going to determine which rows in target correspond to a score less than or equal to 3:
"""
bad_indexes = target <= 3  # PyTorch also provides comparison functions, 
                           # here torch.le(target, 3), but using operators
                           # seems to be a good standard.
bad_indexes.shape, bad_indexes.dtype, bad_indexes.sum()
"""
Note that only 20 of the bad_indexes entries are set to True ! By using a feature in PyTorch called advanced indexing, we can use a tensor with data type torch.bool to index the data tensor. This will essentially filter data to be only items (or rows) corresponding to True in the indexing tensor. The bad_indexes tensor has the same shape as target , with values of False or True depending on the outcome of the comparison between our threshold and each element in the original target tensor:
"""
bad_data = data[bad_indexes]
bad_data.shape
"""
Note that the new bad_data tensor has 20 rows, the same as the number of rows with True in the bad_indexes tensor. It retains all 11 columns. Now we can start to get information about wines grouped into good, middling, and bad categories. Let’s take the .mean() of each column:
"""
bad_data = data[target <= 3]
mid_data = data[(target > 3) & (target < 7)]  # For Boolean NumPy arrays and
                                              # PyTorch tensors, the & operator
                                              # does a logical “and” operation.
good_data = data[target >= 7]
bad_mean = torch.mean(bad_data, dim=0)
mid_mean = torch.mean(mid_data, dim=0)
good_mean = torch.mean(good_data, dim=0)
for i, args in enumerate(zip(col_list, bad_mean, mid_mean, good_mean)):
    print('{:2} {:20} {:6.2f} {:6.2f} {:6.2f}'.format(i, *args))
"""
It looks like we’re on to something here: at first glance, the bad wines seem to have higher total sulfur dioxide, among other differences. We could use a threshold on total sulfur dioxide as a crude criterion for discriminating good wines from bad ones. Let’s get the indexes where the total sulfur dioxide column is below the midpoint we calculated earlier, like so:
"""
total_sulfur_threshold = 141.83
total_sulfur_data = data[:,6]
predicted_indexes = torch.lt(total_sulfur_data, total_sulfur_threshold)

predicted_indexes.shape, predicted_indexes.dtype, predicted_indexes.sum()
"""
This means our threshold implies that just over half of all the wines are going to be high quality. Next, we’ll need to get the indexes of the actually good wines:
"""
actual_indexes = target > 5

actual_indexes.shape, actual_indexes.dtype, actual_indexes.sum()
"""
Since there are about 500 more actually good wines than our threshold predicted, we already have hard evidence that it’s not perfect. Now we need to see how well our predictions line up with the actual rankings. We will perform a logical “and” between our prediction indexes and the actual good indexes (remember that each is just an array of zeros and ones) and use that intersection of wines-in-agreement to determine how well we did:
"""
n_matches = torch.sum(actual_indexes & predicted_indexes).item()
n_predicted = torch.sum(predicted_indexes).item()
n_actual = torch.sum(actual_indexes).item()

n_matches, n_matches / n_predicted, n_matches / n_actual
"""
We got around 2,000 wines right! Since we predicted 2,700 wines, this gives us a 74% chance that if we predict a wine to be high quality, it actually is. Unfortunately, there are 3,200 good wines, and we only identified 61% of them. Well, we got what we signed up for; that’s barely better than random! Of course, this is all very naive: we know for sure that multiple variables contribute to wine quality, and the relationships between the values of these variables and the outcome (which could be the actual score, rather than a binarized version of it) is likely more complicated than a simple threshold on a single value.
"""
"""
Indeed, a simple neural network would overcome all of these limitations, as would a lot of other basic machine learning methods. We’ll have the tools to tackle this problem after the next two chapters, once we have learned how to build our first neural network from scratch. We will also revisit how to better grade our results in chapter 12. Let’s move on to other data types for now.
"""