# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Programming for Data Analysis Assignment 

# ## Semester 2, October 2018, <NAME>

# **Problem statement**
#
# The following assignment concerns the `numpy. random` package in Python. You are required to create a Jupyter notebook explaining the use of the package, including detailed explanations of at least five of the distributions provided for in the package.
#
# There are four distinct tasks to be carried out in your Jupyter notebook.
# 1. Explain the overall purpose of the package.
# 2. Explain the use of the “Simple random data” and “Permutations” functions.
# 3. Explain the use and purpose of at least five “Distributions” functions.
# 4. Explain the use of seeds in generating pseudorandom numbers.

# ## PART 1: Overall Purpose of `numpy.random` Package

# The overall purpose of the `numpy.random` package is to generate random numbers for the user. These can be used, for example, to create games, test hypotheses and code or even as a useful teaching aid. 
#
# Sounds simple, right? Not so simple actually! At the most basic level, the package must at least account for the shape of the array. 
#
# Here for example, using the most basic command `random.rand` the package will generate numbers defaulting to a uniform distribution between 0 and 1 (not including 1 itself)<sup>1</sup>:  

import numpy as np #instruct the notebook to utilize numpy functionality, it can now be used this throughout the numerous commands in this notebook 
np.random.rand(1) #generates an array of 1 random number within the set value parameters

# An example with a more complex 2 dimensional array of random numbers is also seen below:

eg = np.random.rand(5,3) #this time we request a 5 x 3 shaped array of random numbers
eg

import matplotlib.pyplot as plt #import matplotlib functionalities
plt.hist(eg)
plt.show() #wecan see that the random numbers are chosen quite uniformly

# A light change to the code (`.randn`) alters the request to numbers within the "Standard Distribution" where the mean of the data is 0 and variance is 1. "Standard Distribution" implies the data will fit to a bell shaped curve around 0<sup>1</sup>.
#
# For example:

x = np.random.randn(10000, 1) #note the change of command to .randn

# To show the distribution of these random numbers, we need to plot them on a histogram to see that they are indeed centered around 0<sup>2</sup>. 

plt.hist(x) #command to represent x as a histogram
plt.show() #and show it inline

# **Note 1:** `random.randn` only accepts integers as inputs, this is a convenience function. The example above uses a simple array to produce a simple example of the data transposed into a histogram. `random.normal` however is capable of performing the same action but with floats and more parameters making it a more functional tool.   

# **Note 2:** Python actually has its own random package but this is significantly slower and less functional than `numpy.random`<sup>3</sup>.

# As mentioned earlier, the random package can be used for example in simple games where the user must select an option and python must randomly selected the "winner". For example:

# +
import random # Python can use this functionality to provides a library called random to generate pseudo-random numbers

available_choices = ['Dublin', 'New York', 'Sydney'] #this establishes the possibilities for the contestant and python to choose from

winning_city = random.choice(available_choices) # request python to pick a random city from the list.

#based on Ian McLoughlan's notebook (https://github.com/ianmcloughlin/jupyter-teaching-notebooks/raw/master/win-a-car.ipynb)

# +
#Insert Contestant Guess here: 
Guess = 'Dublin'

Guess == winning_city #ask python if the contestant's guess is right (i.e. True) or wrong (i.e. False)  
# -

# It can also be used to test Hypotheses, particularly a "Null" Hypothesis, where the more skeptical assumption is made and then tested to see how likely it is that the person/object/function etc. is doing the thing it is claiming to do. 
#
# For example, the Null Hypothesis could be that from 6 playing cards, 3 red, 3 black, a magician does **NOT** have the skill to pick the red cards first. In order to test the theory that the magician is just guessing, we can perform the following analysis:

# We create a list of six cards numbered from 0 to 5.
cards = list(range(6))
cards

# We calculate that the nNumber of ways of selecting 3 red cards is as follows: 

# +
#1 chance divided by the different number of possible outcomes for each card draw i.e. 6 is the first as there are 6 cards to draw from on the first go, then 5 and 4. 
#They are then divided by the possibilities for the 3 remaining cards not selected i.e. 3, then 2, then only 1. 

1/((6*5*4)/(3*2*1)) #1 in 20 chance or 5% 
# -

# This is confirmed by using the `itertools.combinations` function to list out all the possible combinations of choices the magician could make. It again confirms there is a 1 in 20 chance of getting the only right answer.   

# +
import itertools #import itertools functionality

poss = list(itertools.combinations(cards, 3))
len(poss)

# +
import seaborn as sns #import seaborn functionality to produce graphs

# we then use random.choice to select a correct answer from the list of 20 to simulate the experiment.
red_cards = set(random.choice(poss))

# this function then counts the overlap between the correct answer, and each of the 20 possiblities.
counts = [len(red_cards & set(i)) for i in itertools.combinations(cards, 3)]

# This function then draws a plot so that we may analyse the data visually.
sns.countplot(counts)

#based on Ian McLoughlan's notebook (https://github.com/ianmcloughlin/jupyter-teaching-notebooks/raw/master/lady-tasting-tea.ipynb)
# -

# We can see that apart from the 5% chance the magician will get the red cards, there is also a 50% chance that the magician will get at least 2 red cards if only randomly guessing the pick.
#
# The above examples show how random data can be used within the Python package.

# ## PART 2: Use of "Simple Random Data" and "Permutations" Functions

# The use of the Simple Random Data versus Permutations Functions is best seen through examples. 
#
# Functions listed as Simple Random Data in Scipy.org<sup>4</sup> include `random.rand` and `random.randn` as demonstrated above, as well as the following:
#
# ### randint and random_integers
# `random.randint` and `random.random_integers` are similar in that they accept boundary intervals of integers but are half-open interval and closed interval respectively. If no high integer is assigned for `random.randint`, 0 is the lowest value assigned, while for `random.random_integers`, the low parameter becomes the highest such integer and 1. 
# Data type (dtype) is also an optional input for `random.randint`. 

np.random.randint(0, 60, size=17) #request to produce an array of 20 random integers from 0 to 59. Half-open interval includes 0 but not 60.

# Interestingly, ipython is moving away from `random.random_integers` for the preferable `random.randint`and actively suggests to use it. 

np.random.random_integers(5, size=17) #closed interval includes 1 and 5 

# ### random_sample, random, ranf, sample

# Reviewing these 4 functions they all appear to be producing the same results using a “continuous uniform” distribution. A quick review of the characteristics of the functions show identical results. See table below created from data in Scipy.org<sup>4</sup>.
#
#
# |**Function** |**Returns** | **Interval** | **Parameters**
# |:---|:---|:---|:---
# |`random_sample` | float | half-open interval [0.0, 1.0) | size (optional)
# |`random` | float | half-open interval [0.0, 1.0) | size (optional)
# |`ranf` | float | half-open interval [0.0, 1.0) | size (optional)
# |`sample` | float | half-open interval [0.0, 1.0) | size (optional)
#
# After some research, it indeed appears to be correct that they are fundamentally the same and all produce the same random numbers<sup>5</sup>. Examples are as follows: 

np.random.random_sample((8,2)) #Defaults to a range between 0 and 1.0, not including 0 itself. 

# +
a = 3 #Lowest value
b = 10 #Highest value 
x = 3 #Array Size x axis 
y = 4 #Array Size y axis

(b - a) * np.random.random_sample((y, x)) + a 
#Requests an array of random numbers from 3 (but not including 3 itself) to 10
# -

# ### random.choice

# We have already seen `random.choice` in action earlier where the game was created to select a city. Python would choose a "winner" from the established list. More examples of this function in use are below:

np.random.choice(20, 15) 
# NOTE: this is equivalent to np.random.randint(0,20,15) 
# 15 random integers are selected in the output array between 0 and 20 without including 20

# This function also allows you to influence the probability of each entry too using a *p* entry to create non-uniform output, for example:

np.random.choice(5, 5, p=[0, 0, 0.4, 0.5, 0.1]) #9 of every 10 selections should result in a 2 or a 3 as each p location corresponds to the array's index position

# Using the previous example of the game to choose a city, we can now create a preference for one city over the others.

# +
available_choices = ['Dublin', 'New York', 'Sydney'] #add comment

# Pick a random city from the list.
winning_city = np.random.choice(available_choices, p=[0.1, 0.3, 0.6]) # NOTE: I did need to add np. to use the numpy functionality to influence probability as this is not a feature of the basic python random function 
winning_city #the results are definitely preferable for Syndey now
# -

# ### Bytes

# By inputting the desired length, this function will return random number of bytes, for example:

np.random.bytes(5) #request 5 bytes

# ### Shuffle and Permutation

# Moving onto the two Permutations Functions, the first is shuffle and as you might expect, its function is to shuffle the sequence of information fed into it. for example:

numbers = np.arange(17) #creates list of numbers from 0-16
numbers

np.random.shuffle(numbers) #shuffles their order but set is still lenght of 17 with values of 0-16
#does not give output array without calling for "numbers" again

numbers

# When we call `numbers` the second time, the resulting array is the original list except it now has shuffled numbers i.e. the numbers are shuffled *in situ* and this is now `numbers`. 
#
# However, with a `permutation`, although the result looks similar, a new copy of the array is produced containing the shuffled numbers. The original array can still be called as it ramains unchanged. 

x = np.arange(10) #create an array called x containing a range of numbers from 0 to 9
x

np.random.permutation(x) #create a permutated version of x (a copy of x except now shuffled)

x #we can see when we call x again, it is the original array as it has not been changes, only a copy made above

# ### Summary

# In summary, the difference between the "Simple Random Data" and "Permutations" Functions is that Simple Random Data is selecting data randomly from a range of data input into the function, along with potentially size, shape or probability. Permutations however are instead mixing the existing array of data randomly, either within the original array itself (`shuffle`) or by creating a copy of the array (`permutation`).  

# ## PART 3: Use and Purpose of Five "Distributions" Functions

# In this section I will examine the use and purpose of the following 5 "Distributions" Functions<sup>4</sup> :
#
# - `random.normal`
# - `random.standard_normal`
# - `random.standard_t`
# - `random.binomial`
# - `random.laplace`
#

# ### random.normal

# `random.normal` draws its selection of numbers from the normal (Gaussian) distribution curve A.K.A Bell curve due to its shape. Normal distribution is designed to capture the variation often seen in nature. The percentage of values reduce as you move away from the mean of the curve and are measured along the x axis in units of standard deviation. The percentage for each standard deviation away from the centre is 
# - approximately 68% of values are within one standard deviation (σ) away from the mean (x̅), 
# - approximately 95% of values are within two standard deviations (σ) away from the mean (x̅),
# - approximately 99.7% of values are within three standard deviations (σ) away from the mean (x̅)
#
#   or 95% of the area of a normal distribution is within 1.96 standard deviations (σ) of the mean (x̅)

mu, sigma = 0, 0.1 #takes arguements for the mean and standard deviation
x = np.random.normal(mu, sigma, 8000) #creates a data set that is of normal distribution
x #shows us a preview of that data set

plt.hist(x) #plot the 8000 values on a simple histogram.
plt.show()

# +
#source to add normal curve to data: https://stackoverflow.com/questions/35544233/fit-a-curve-to-a-histogram-in-python

from scipy.optimize import curve_fit 

def gaussian(x, mean, amplitude, standard_deviation): #define the parameters of a normal distribution curve* 
    return amplitude * np.exp( - ((x - mean) / standard_deviation) ** 2)

bin_heights, bin_borders, _ = plt.hist(x, bins='auto', label='histogram') #define bin numbers & heights (more bins means better visualisation of curve unlike example above that only has 10 bins)

bin_centers = bin_borders[:-1] + np.diff(bin_borders) / 2
popt, _ = curve_fit(gaussian, bin_centers, bin_heights, p0=[1., 0., 1.])

x_interval_for_fit = np.linspace(bin_borders[0], bin_borders[-1], 8000)
plt.plot(x_interval_for_fit, gaussian(x_interval_for_fit, *popt), label='fit')
plt.legend()
# -

# Imagine overlaying the above curve with the textbook Bell Shaped Curve and you can see how they are very close in shape.
#
# ![alt text](https://upload.wikimedia.org/wikipedia/commons/8/8c/Standard_deviation_diagram.svg)
#
# Source: http://landsurvival.com/schools-wikipedia/images/209/20970.png.htm

# ### random.standard_normal

# The `standard_normal` function is therefore a simplified version of the `normal` function where no mean or standard deviation figures need to be added, only the number of values to be randomly produced.

s = np.random.standard_normal(8000) # is a simplified version of normal and just takes one arguement of size producing similar results
s

import matplotlib.pyplot as plt
bin_heights, bin_borders, _ = plt.hist(s, bins='auto', label='histogram') #define bin numbers & heights (more bins means better visualisation of curve unlike example above that only has 10 bins)
plt.hist(s) #plot the 8000 values on a simple histogram shows the similar outcome when using this function.
plt.show()

# ### random.standard_t (t-test)

#
# Scipy.org describes the t-test well as "a significance test for the mean value of a normal distribution" <sup>4</sup>. It is a complex but very useful test to compare the mean of populations that meet certain requirements and assumptions<sup>6</sup>. It has a hyperbolic distribution and requires *degrees of freedom* as an input, and size is optional. An example<sup>4</sup> of the implementation of this test is below:   

# +
intake = np.array([5260, 5470, 5640, 6180, 6390, 6515, 6805, 7515, 7515, 8230, 8770]) #example of women's intake of calories per day in Kj
intake

#NOTE: recommended value is 7725Kj and there are 10 degrees of freedom
# -

np.mean(intake) #establish the mean of the intake array which will remain fixed 

# +
intake.std(ddof=1) 

#where “Delta Degrees of Freedom” is the divisor used in the calculation is N - ddof, 
#where N represents the number of elements. By default ddof is zero.
# -

s = np.random.standard_t(10, size=100000) 
s
#the function is used to create a second array of random numbers to compare to the "intake" array. 
#the function tests if the sample mean is within 95% of the recommended value.

# +
t = (np.mean(intake)-7725)/(intake.std(ddof=1)/np.sqrt(len(intake)))

np.sum(s<t) / float(len(s)) 

#the two set of data are then compared using the one-sided t-test, to answer how far out in the distribution does the t statistic appear?
# -

# The final figure is the p-value and in this case is approx. 0.009, which tells us that the null hypothesis has a probability of about 99% of being true.

# ### random.binomial

# Binomial distribution follows the following rules<sup>7</sup>: 
#
# - the experiment consists of *n* repeated trials
# - each trial can result in just two possible outcomes i.e. success or failure
# - the probability of success (*P*) is the same on every trial
# - the trials are independent i.e. the outcome of one trial does not affect the outcome of other trials
#
# The parameters for the numpy function are therefore *n*, *P*, and the size of the sample nd it returns a binomial distributed array.
#
# An example is outlined below: 

# +
# 10,000 trials of equal probability of getting an even number on 3 dice throws.  
p = sum(np.random.binomial(3, 0.5, 10000) == 0)/10000  

p * 100
# -

# The output is the % probability of a True outcome

# ### random.power

# This function creates random values between 0 to 1 with a power distribution with positive exponent (a - 1). This function is also known as the power function distribution. It requires a float greater than 0 or array and can also accept a shape which defaults to null with a single number returned if no shape is specified.

a = 5. # where a is the size/shape of the data
samples = 1000 # the number of values to sample
s = np.random.power(a, samples) 

plt.hist(s, bins=30) #specify 30 bins to be used
plt.show()

# ### random.laplace

# The Laplace A.K.A. double exponential distribution works with the following (optional) parameters:
# - float -  in this case is the mean of the distribution peak
# - the exponential decay
# - a shape which defaults to null with a single number returned if no shape is specified, with specified location (or mean) and scale (decay).
#
# Overall it is similar to the Gaussian/normal distribution, but is sharper at the peak and has fatter tails. It examines the difference between two independent, identically distributed exponential random variables<sup>4</sup>.

loc, scale = 0, 1 #where loc is the position of the distribution peak and scale is the exponential decay 
s = np.random.laplace(loc, scale, 1000) #1000 is the size of the dataset

count, bins, ignored = plt.hist(s, 30, density=True) #random dataset, 30 bins requested and density=Trye works to normalize our inputs 

count, bins, ignored = plt.hist(s, 30, density=True) #random dataset, 30 bins requested and density=Trye works to normalize our inputs 
x = np.arange(-8., 8., .01) #creates x axis values from -8 to 8 for creation of curve
pdf = np.exp(-abs(x-loc)/scale)/(2.*scale) #creates y axis values for creation of curve
plt.plot(x, pdf) # plots x and y to create laplace (exponenmtial decay) curve

# ## PART 4: Use of Seeds in Generating Pseudorandom Numbers

# In <NAME>'s "Python for Data Analysis"<sup>3</sup>, he explains why we refer to these python/numpy generated nbumbers as pseudo-random numbers. 
#
# >"We say that these are pseudorandom numbers because they are generated by an algorithm with deterministic behavior based on the seed of the random number generator. You can change NumPy’s random number generation seed using `np.random.seed`"
#
# In otherwords, these numbers appear random but there is an algorithim driving their creation. To be truely random, there needs to be little or no external influence on the outcome, for example, flipping a coin, rolling a dice, dealng shufffeled cards. 

# +
np. random. seed(1234) #example of where the seed for the random number generation can be set and therefore controlled 
#Note: if no seed is specified, the current system time is used or other seed determined by the local operating system
w = np.random.rand(10)
w #rerunning the cell to produce a new output does not change the random numbers that were first generated as the seed producing them remains the same. 

#if no seed was specified, new random numbers woudl appear each time I ran the cell as the time would be different and therefore the seed different 
# -

# McKinney goes on to say that: 
#
# >"The data generation functions in `numpy.random` use a global random seed. To avoid global state, you can use `numpy.random.RandomState` to create a random number generator isolated from others"

rng = np. random. RandomState(1234)
rng. randn(10)

rng #this command requests information on the current random generator in use by python/numpy (https://blogs.mathworks.com/cleve/2015/04/17/random-number-generator-mersenne-twister/)

# The "mt" prefix indicates that the Mersenne Twister pseudo-random number generator is in use<sup>8</sup>.  The creators of the generator, Matsumoto and Nishimura, describe it as a;
# > "A 623 dimentionally equidistributed uniform pseudorandom number generator <sup>9</sup>."
#
# It is a generator that is in use widely in data programs but is not suitable for modern cryptology. Its strength is in its ability to have a massive 2<sup>19937</sup>-1 instances before it begins repeating sequences* but its weakness lies in the seed. If the seed is revealed or guessed, the entire suquence can then be discovered<sup>10</sup>.  
#
# \* this is a Mersenne prime number meaning that it is a prime number that is one less than a power of two (M<sub>n</sub>=2<sup>n</sup>−1 where n is an integer).

# In summary, the generation of pseudorandom numbers is useful for programming and cryptography (although Marsenne Twister specifically is not recommnended). Pseudorandomness at its most basic level can be seeded (determined) by the time meaning that random data sets can be produced at will to create, test or play with.   

# ## References

# 1. https://docs.scipy.org/doc/numpy/reference/routines.random.html
# 2. Ian McLoughlan Programming for Data Analysis, GMIT Lectures Week 2 "PyPlot_Examples_Master" 
# 3. Python for Data Analysis, <NAME>
#     2.6 Pseudorandom Number Generation (page 118/119)
# 4. https://docs.scipy.org/doc/numpy/reference/routines.random.html 
# 5. https://stackoverflow.com/questions/18829185/difference-between-various-numpy-random-functions
# 6. https://www.encyclopediaofmath.org/index.php/Student_test
# 7. https://stattrek.com/probability-distributions/binomial.aspx
# 8. https://docs.scipy.org/doc/numpy-1.6.0/reference/generated/numpy.random.mtrand.RandomState.html)
# 9. Marsenne Twister: A 623 dimentionally equidistributed uniform pseudorandom number generator, Matsumoto and Nishimura, 1997 http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/ARTICLES/mt.pdf
# 10. https://asecuritysite.com/encryption/twister

# ## Back Up/Work in Progress

# +
import numpy as np
import random
import seaborn as sns
import itertools
# %matplotlib inline

lotto_numbers = list(range(47))
winning_numbers = list(np.random.permutation(48))
y = winning_numbers[0:6]
y

poss = list(itertools.combinations(lotto_numbers, 6))
x = len(poss)
x

# -

poss = list(itertools.combinations(lotto_numbers, 6))
x = len(poss)
x

# +

#1 in 10,737,573 chances of getting 6 numbers from a 47 ball draw.

#So if I play the same numbers every Saturday for a year (52 weeks) 
favourite_numbers = [1, 3, 11, 16, 29, 37]
z1 = list(np.random.permutation(poss))
z = z1[0:52]
z

#needs work
if z.count(favourite_numbers) == 1:
    print(Yes)a
    
# Count the overlap between the correct answer, and each of the 70 possiblities.
counts = [len(favourite_numbers & set(z)) for i in poss]
counts
# Creat the plot.
sns.countplot(counts)

StarcoderdataJupyterScriptsDedupFiltered