import tensorflow as tf
import sys

print("Python Version:",sys.version.split(" ")[0])
print("TensorFlow Version:",tf.VERSION)

"""
Explanation: <table>
    <tr>
        <td style="text-align:left;"><div style="font-family: monospace; font-size: 2em; display: inline-block; width:60%">2. Tensors</div><img src="images/roshan.png" style="width:30%; display: inline; text-align: left; float:right;"></td>
        <td></td>
    </tr>
</table>

Before we go into tensors and programming in TensorFlow, let's take a look at how does it work.
TensorFlow Basics
TensorFlow has a few concepts that you should be familiar with. TensorFlow executes your code inside an execution engine that you communicate with using an API. In TensorFlow your data is called a Tensor that you can apply operations (OPs) to. Your code is converted into a Graph that is executed in an execution engine called a Session. So your python code is just a representation of your graph that can be executed in a session.

You can see how your data (or tensors) flow from one operation to the next, hence the name TensorFlow.
Since version 1.5 and as of version 1.8 there are two methods to execute code in TensorFlow:

Graph Execution
Eager Execution

The main difference is graph execution is a type of declarative programming and eager execution is a type of imperative programming. In plain English, the difference is graph execution defines your code as a graph and executes in a session. Your objects in python are not the actual objects inside the session, they are only a reference to them. Eager execution executes your code as you run it giving you a better control of your program while it is running so you are not stuck with a predefined graph.
So why do we even bother with graph execution?
Performance is a big issue in machine learning and a small difference in execution time can save you in a long project weeks of your time and 1000s of hours of GPU time. Support is another issue for now where some features of TensorFlow do not work in Eager Execution like high level estimators.
We will focus in the following tutorials only on graph execution and we will cover eager execution later in this series.
Tensors
Tensors represent your data. It a scalar variable or an array of any dimension. Tensors are the main object to store and pass data between operations, the input and output of all operations are always tensors.
Tensor Shape and Rank
Tensors have a rank and a shape so for scalar values, we use rank-0 tensors of a shape () which is an empty shape
Assuming we need a variable or a constant number to use in our software, we can represent it as a tensor of rank-0.

A rank-1 tensor can be though of as an vector or a one dimensional array. creating a rank-1 tensor with shape (3) will create a tensor that can hold three values in a one-dimensional array.

A rank-2 is a matrix or a two dimensional array. This can be used to hold two dimensional data like a black and white image. The shape of the tensor can match the shape of the image so to hold a 256x256 pixel image in a tensor, you can create a rank-2 tensor of shape (256,256).

A rank-3 tensor is a three dimensional array. It can be used to hold three dimensional data like a color image represented in (RGB). To create a tensor to hold an color image of size 256x256, you can create a rank-3 tensor of shape (256,256,3).

TensorFlow allows tensors in higher dimensions but you will very rarely see tensors of a rank exceeding 5 (batch size, width, height, RGB, frames) for representing a batch of video clips.
Importing Tensor Flow
Let's import TensorFlow and start working with some tensors.
End of explanation
"""


sess = tf.InteractiveSession()

"""
Explanation: Graph Execution
TensorFlow executes your code inside a C++ program and returns the results through the TensorFlow API. Since we are using Python, we will be using TensorFlow Python API which is the most documented and most used API of TensorFlow.
Since we are using graph execution, there are two ways to create a session:

Session
Interactive Session

Sessions and interactive sessions, use your code to build a "Graph" which is a representation of your code inside TensorFlow's execution engine. The main difference between them is, an interactive session makes itself the default session. Since we are using only one session for our code, we will use that.
For now let's start an interactive session and start flowing some tensors!
End of explanation
"""


a = tf.zeros(())

"""
Explanation: Generating new Tensors
Since we have an interactive session, let's create a tensor. There are two common ways to create a tensor tf.zeros() and tf.ones(). Each one of them takes a python tuple or an array as the shape of the tensor.
Let's start be creating a rank-0 tensor.
End of explanation
"""


a

"""
Explanation: We create a tensor and assigned it to a local variable named a. When we check the value of a this is what we get.
End of explanation
"""


a.eval()

"""
Explanation: Notice there is no value. You need to call eval() method of the tensor to get the actual value. This method takes an optional parameter where you can pass your session. Since we are using interactive session, we don't have ato pass anything.
End of explanation
"""


a.shape

"""
Explanation: You should know that eval() method returns a numpy.float32 (or what ever the type of the tensor is) if the rank of the tensor is 0 and numpy.ndarray if the tensor has rank 1 or higher.

Numpy: is a multi-dimensional array library for python that runs the operations in a C program and interfaces back with python to ensure fast array operations.

We can also check the rank and shape of the tensor.
End of explanation
"""


a.shape.ndims

"""
Explanation: the rank would be the number of dimensions.
End of explanation
"""


a.name

"""
Explanation: Notice the name inside the TensorFlow execution engine is not a. It is zeros:0 which is an auto generated name for the variable. The auto generated name is the name of the operation that generated the tensor and then an the index of the of the tensor in the output of the operation.
End of explanation
"""


tf.zeros(())

"""
Explanation: If you created another variable using the same operation, it will be named zeros_1:0.
End of explanation
"""


b = tf.zeros((3), name="b")
b

"""
Explanation: Now let's create a second tensor of shape (3) which is going to be a rank-1 tensor. This time we will name it b and store it in a local variable named b.
End of explanation
"""


type(b.eval())

"""
Explanation: Notice the name of the variable now is b:0 which is the name that we gave it and the index of 0 of the return of the operation. We can also get the value in the same way using eval() method.
End of explanation
"""


sess.run(b)

"""
Explanation: You can also get the value of a tensor by executing the tensor using your session.
End of explanation
"""


tf.fill((2,2), 5).eval()

"""
Explanation: You can also fill the tensor with any other value you want other than 0 and 1 using fill() function.
End of explanation
"""


tf.zeros((10,3)).eval()

"""
Explanation: Notice that the data type of this tensor is int32 and not float32 because you initialized the tensor with an integer 5 and not 5.0.
Tensor Shape
For multi dimensional tensors, the shape is passed as a tuple or an array. The way this array is arranged is from the outer most dimension to the inner dimensions. So for a tensor that should represents 10 items and each item has three numbers, the shape would be (10,3).
End of explanation
"""


tf.zeros((2,3,4)).eval()

"""
Explanation: Note: This is the opposite of the how matrix shape notation is written in mathematics. A matrix of shape $A_{(3,10)}$ can be represented in TensorFlow as (10,3). The reason for that is in mathematics the shape is $(Columns,Rows)$ and TensorFlow uses (Outer,Inner) which translates in 2-D tensor as (Rows,Columns).
For higher dimensions the same rules applies. Let say we have 2 items and each item has 3 parts and each part consists of 4 numbers the shape would be (2,3,4)
End of explanation
"""


arr1 = tf.random_normal((1000,))
arr1

"""
Explanation: Generating Tensors with Random Values
In many cases, you want to generate a new tensor but we want to start with random values stored in the tensor. The way we do that is using one the random generators of TensorFlow.
Normal Distribution
tf.random_normal(shape, mean=0.0, stddev=1.0, dtype=tf.float32, seed=None, name=None)
End of explanation
"""


import matplotlib.pyplot as plt

%matplotlib inline

plt.hist(arr1.eval(), bins=15);

"""
Explanation: This function returns random values using normal distribution which is also known as Gaussian distribution or informally called a "Bell Curve". To better understand it, let's first look at a graph showing this distribution. In Mathematics, a normal distribution of mean $\mu$ and standard deviation $\sigma$ is denoted $N(\mu, \sigma)$ (More about that in "The Math Behind It").
To do that we will import Matplotlib which is a very common charting library for Python.
End of explanation
"""


arr1 = tf.random_normal((1000,), mean=20.0)

plt.hist(arr1.eval(), bins=15);

"""
Explanation: Notice the bill shape of the curve where you get more values around your mean a fewer values as you move away from the mean.
You can also change the mean.
End of explanation
"""


arr1 = tf.random_normal((1000,), stddev=2, name="arr1")
arr2 = tf.random_normal((1000,), stddev=1, name="arr2")

plt.hist([arr1.eval(), arr2.eval()], bins=15);

"""
Explanation: You can also control how concentrated your random numbers will be around the mean by controlling the standard deviation. Higher standard deviation means less values around the mean and wider distribution.
End of explanation
"""


plt.hist(tf.truncated_normal((1000,)).eval(), bins=15);

"""
Explanation: One more note on normal distribution, if you created a large tensor with millions or tens of millions of random values, some of these values will fall really far from the mean. With some machine learning algorithms this might create instability. You can avoid that by using truncated_normal() function instead of random_normal(). This will re-sample any values that falls more than 2 standard deviations from the mean.
End of explanation
"""


arr1 = tf.random_uniform((1000,))
plt.hist(arr1.eval(), bins=15);

"""
Explanation: Uniform Distribution
The other common distribution is the uniform one. This will generate values with equal probability of falling anywhere between two numbers.
tf.random_uniform(shape, minval=0, maxval=None, dtype=tf.float32, seed=None, name=None)
End of explanation
"""


tf.range(5).eval()

"""
Explanation: Generating Tensors with Sequence Values
You can generate tensors with sequence values using the following function:
tf.range(start, limit=None, delta=1, dtype=None, name='range')
End of explanation
"""


tf.range(0, 5).eval()

"""
Explanation: Which is equivalent to:
End of explanation
"""


tf.range(0, 5, 2).eval()

"""
Explanation: Notice that the output of range() function will never reach the limit parameter.
You can also control the delta which is the spacing between the tensor's elements.
End of explanation
"""


a = tf.range(6)
tf.reshape(a, (3,2)).eval()

"""
Explanation: Reshaping Tensors
You can reshape tensors using this function:
tf.reshape(tensor, shape, name=None)
End of explanation
"""


a = tf.ones((2,2))
b = tf.fill((2,2), 10.0) # Notice we used 10.0 and not 10 to ensure the data type will be float32

c = a + b
c.eval()

"""
Explanation: Tensor Arithmetics
You can use standard python arithmetics on tensors and get results in a new tensor.
Addition
End of explanation
"""


d = c * 2.0
d.eval()

(d + 3).eval()

"""
Explanation: Element-wise Operations
End of explanation
"""


i = tf.eye(3,3)
i.eval()

"""
Explanation: Matrix Operations
TensorFlow support a variety of matrix operations.
Identity Matrix
An identity matrix is a matrix where all the values are zeros exception diagonally where it has values of 1. This is an example of an identity matrix of shape (3,3).
$$\begin{bmatrix}
{1} & {0} & {0} \
{0} & {1} & {0} \
{0} & {0} & {1} \
\end{bmatrix}$$
To do that in TensorFlow use the tf.eye() function.
End of explanation
"""


a = tf.range(1,9)
i = tf.reshape(a, (2,4))
i.eval()

it = tf.matrix_transpose(i)
it.eval()

"""
Explanation: Transpose
Transpose is another operation that is commonly used in matrix calculations. Transpose converts rows to columns. Assume you have a matrix $\mathbf{A}$, a transpose operation over this matrix produces $\mathbf{A^T}$ pronounced $\mathbf{A}$ transpose.
End of explanation
"""


a = tf.ones((2,3))
b = tf.ones((3,4))

c = tf.matmul(a ,b)

print("c has the shape of:", c.shape)
c.eval()

"""
Explanation: Matrix Multiplication
One of the most common operations for matrices in deep learning in matrix multiplication. Matrix multiplication is not an element-wise operation. The exact math will be discussed in the last section of this tutorial "The Math Behind It". But for now to give you the basics you should know the following:
Assume we have two matrices $\mathbf{A}$ and $\mathbf{B}$. The shape of $\mathbf{A}$ is (m,n) and the shape of $\mathbf{B}$ is (o,p) we can write these two matrices with their shape as $\mathbf{A}{(m,n)}$ and $\mathbf{B}{(o,p)}$. Multiplying these two matrices produces a matrix of the shape (m,p) IF $n=o$ like this:
$\mathbf{A}{(m,n)} . \mathbf{B}{(o,p)}=\mathbf{C}_{(m,p)} \leftarrow n=o$
Notice the inner shape of these two matrices is the same and the output matrix has the shape of the outer shape of these two matrices. If the inner shape of the matrices does not match the product doesn’t exist.
We can use tf.matmul() function to do that.
End of explanation
"""


g = [88, 94, 71, 97, 84, 82, 80, 98, 91, 93]

total = sum(g)
count = len(g)

mean = total/count
mean

"""
Explanation: The Math Behind It
Standard Deviation $\sigma$ or $s$
Standard deviation is the measure of how elements in a set vary from the mean. So a sample with most data points close to the mean has low standard deviation and it gets higher as the data points start moving away from the mean. In statistics, standard deviation is denoted as the small letter sigma $\sigma$ or $s$.
The formula to calculate standard deviation for a whole population is:
$$\sigma={\sqrt {\frac {\sum_{i=1}^N(x_{i}-{\overline {x}})^{2}}{N}}}$$
Let's break it down and see how to calculate standard deviation.
Assume we have exam grades of 10 students and the grades are so follow:
| ID    | Grade  |
| ----- |:------:|
| 1     |  88    |
| 2     |  94    |
| 3     |  71    |
| 4     |  97    |
| 5     |  84    |
| 6     |  82    |
| 7     |  80    |
| 8     |  98    |
| 9     |  91    |
| 10    |  93    |
First thing we need to do is calculate the mean. The mean is denoted as $\overline {x}$ (pronouned "x bar"). To calculate the mean (or average) get the sum all numbers and divide it by their count. It is also commonly denoted as a small letter mu $\mu$. Assume you have $N$ values, this will be the formula to calculate the mean:
$$\overline {x} = \frac{x_1 + x_2 + ... + x_N}{N} = \frac{\sum_{i=1}^N x_i}{N}$$
So let's calculate that.
End of explanation
"""


from math import sqrt

σ = sqrt(sum([(x-mean)**2 for x in g]) / count)
σ

"""
Explanation: Now that we know the mean, we can go back to the original equation and calculate the standard deviation.
So looking at the equation one more:
$$\sigma={\sqrt {\frac {\sum_{i=1}^N(x_{i}-{\overline {x}})^{2}}{N}}}$$
First we need to get each element in our grades $x_i$ and subtract it from the mean $\overline {x}$ then square it and take the sum of that.
python
a = [(x-mean)**2 for x in g]
b = sum(a)
Divide that by the number of elements $N$ then take the square root
python
variance = b / count
σ = sqrt(variance)
We can write the whole thing in one like this:
End of explanation
"""


import numpy as np

np.std(g)

"""
Explanation: Note that standard deviation is a build in function in NumPy, TensorFlow and many other languages and libraries.
End of explanation
"""


t = tf.constant(g, dtype=tf.float64)

mean_t, var_t = tf.nn.moments(t, axes=0)
sqrt(var_t.eval())

"""
Explanation: In TensorFlow
End of explanation
"""


variance = sum([(x-mean)**2 for x in g]) / count
variance

"""
Explanation: Variance $\sigma^2$, $s^2$ or $Var(X)$
It is just the square of the standard deviation.
End of explanation
"""


a = [[1,0],
     [3,2],
     [1,4],
    ]

b = [[2,1,2],
     [1,2,3],
    ]

"""
Explanation: Matrix Multiplication
Arguably, the most common matrix operation you will perform in deep learning is multiplying matrices. Understanding this operation is a good start to understanding the math behind neural networks. This operation is also known as "dot product".
Assume we have two matrices $\mathbf{A}{(2,3)}$ and $\mathbf{B}{(3,2)}$. The dot product of these two matrices $\mathbf{A}{(2,3)} . \mathbf{B}{(3,2)}$ is calculated as follows:
$\mathbf{A}_{(2,3)} = \begin{bmatrix}
{1} & {0} \
{3} & {2} \
{1} & {4} \
\end{bmatrix}$
$\mathbf{B}_{(3,2)} = \begin{bmatrix}
{2} & {1} & {2} \
{1} & {2} & {3} \
\end{bmatrix}$
$\mathbf{C}{(2,2)} = \mathbf{A}{(2,3)} . \mathbf{B}_{(3,2)}$
$\mathbf{C}_{(2,2)} = \begin{bmatrix}
{2\times1 + 3\times1 + 1\times2} & {0\times2 + 2\times1 + 4\times2} \
{1\times1 + 3\times2 + 1\times3} & {0\times1 + 2\times2 + 4\times3} \
\end{bmatrix} = \begin{bmatrix}
{2 + 3 + 2} & {0 + 2 + 8} \
{1 + 6 + 3} & {0 + 4 + 12} \
\end{bmatrix}= \begin{bmatrix}
{7} & {10} \
{10} & {16} \
\end{bmatrix}$
This is an animation to shows how that is done step by step.

Now let's confirm it with TensorFlow.
End of explanation
"""


a = tf.constant(a)
b = tf.constant(b)

c = tf.matmul(tf.matrix_transpose(a), tf.matrix_transpose(b))

c.eval()

"""
Explanation: Remember from before, that mathematical shape of a matrix is opposite of TensorFlow shape of a tensor. So instead of rewriting our arrays, we will just use transpose you make rows into columns and columns into rows.
End of explanation
"""


c = tf.matmul(a,b, transpose_a=True, transpose_b=True)

c.eval()

"""
Explanation: Luckily there is also a easier way to to that.
End of explanation
"""


