# Class 4

Note:  The notes that follow are largely those of Mark Krumholz (ANU) who led the Bootcamp
last in 2015.  You can find the 2015 lectures [here](https://sites.google.com/a/ucsc.edu/krumholz/teaching-and-courses/python-15)

```python
# These are to display images in-line
from IPython.display import Image
from IPython.core.display import HTML

#Imports
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

# Files: The Basics

## Writing text files

Thus far we have mostly dealt with inputs and outputs by keyboard and screen, with the exception of using files to define functions and to write out plots. However, for the vast majority of scientific (and other) applications we want to be able to handle files. Handling files allows us to read in and process data in an automated fashion, and to write out results in a form that is easy to save, and is suitable for something too complex to just print to the screen. Thus half of today's class is devoted to file handling.

Files are opened with the open() command, which works as follows:

```python
f = open('testfile.txt', 'w')
```

The first argument is just the name of the file to be opened. The second argument gives the mode in which it is to be opened. We'll learn about more modes as we go, but a quick summary is that files can be opened for writing, reading, appending, or both reading and writing. A file that is opened for writing is created as an empty file; if a file of the same name already exists, it is automatically deleted. Append mode is the same, except that material is added to the end of the existing file instead of replacing it. Files that are opened for writing or appending can only be written, not read as inputs. Conversely, files that are opened for reading can be read in, but cannot be altered. Files that are opened for both reading and writing can be interacted with in both ways.

Once a file has been opened, we can write text into it by invoking the write function that is associated with that file:

```python
f.write('Output written to a file\n')
a = 2
b = 3.0
f.write(str(a)+' '+str(b)+'\n')
```

    6

There are two subtle things to notice here. First, we've ended the lines we're writing out with '\n'. The \n is the code for a line break. If we don't include that, the output will all be on a single line. By default the write command does not add a line break after writing something, so if you want line breaks, you need to put them in yourself. Second, the write command takes a string as an argument, and only a string: in this example, f.write(a) would produce an error, because a is an integer not a string. To write it out, we must turn a into a string first by doing str(a).

Once we're done with the file, we can close it by invoking the close() method:

```python
f.close()
```

Now let's look at the file from the command line using the more command:

```python
# more is a bash (terminal) command, using the sign ! allows you to write bash comands in the notebook ( try !ls)
!more testfile.txt
```

    Output written to a file
    2 3.0

We've successfully written to a file.

## String formatting

When writing numerical output to a file, it is often helpful to have a bit more control about how numbers are converted into text. This capability is provided by the format command. Here's a very simple example of how string formatting works:

```python
s = "Here is an integer: {}; here is a float: {}".format(a, b)
print(s)
```

    Here is an integer: 2; here is a float: 3.0

The basic usage of the format command is as follows: we put a string in quotes, which contains one or more pairs of curly brackets {}. At the end of the string, we put .format(arg1, arg2, arg3,...) where arg1, arg2, arg3, ... are some number of arguments that are to be formatted. These arguments will be matched up with the curly brackets, which are used to specify where the data are to be placed in the output. Each of these arguments will be converted to a string and inserted in the appropriate location. In this example, the variable a was converted to the string "2" and the variable b was converted to the string "3.0". The resulting string is returned, and may be assigned to a variable, printed on the screen, or given to the write command, as we like.

The power of the format command comes in the fact that we can put codes inside the brackets to control how the numerical values are converted to strings. For example:

```python
print("Here is an integer: {:5}; here is a float: {:5.3f}".format(a, b))
```

    Here is an integer:     2; here is a float: 3.000

Let's break down what appeared inside the brackets. First, there is a colon, which says that we are now going to specify a format. Then in both cases we put the number 5. This specifies that the number is to be converted into a space at least 5 characters wide, padding it with spaces if necessary. Then for the second argument, we had ".3f". The period followed by a number and f says that we want a floating point number to be printed out using a certain minimum number of decimal places, padding with zeros if necessary.

Some other options:

```python
print("Here is an integer: {:05}; here is a float: {:^10.2e}".format(a, b))
```

    Here is an integer: 00002; here is a float:  3.00e+00

In the first location, we put 05 instead of 5. The leading 0 says that the extra spaces are to be filled with zeros instead of spaces. In the second argument, the ^10 says that the string is to be 10 spaces wide, and that the results are to be centered within it, as opposed to right-justified (which is the default). The ".2e" says that we want two decimal places, and that we want the output to be printed out in exponential (scientific) notation instead of floating point notation.

Numerous other options for string formatting exist: see http://docs.python.org/2.7/library/string.html#formatspec.

Let's use this capability to print out some data describing a sin function:

```python
x = np.arange(0,2*np.pi,0.01)
f = open('sin.txt', 'w')
for val in x:
    f.write("{:.4f} {:7.4f}\n".format(val, np.sin(val)))
f.close()
```

Now we can look at the file from the command line, since we have created a large file, we will only print the first 20 lines using the bash command head:

```python
!head -20 sin.txt
```

    0.0000  0.0000
    0.0100  0.0100
    0.0200  0.0200
    0.0300  0.0300
    0.0400  0.0400
    0.0500  0.0500
    0.0600  0.0600
    0.0700  0.0699
    0.0800  0.0799
    0.0900  0.0899
    0.1000  0.0998
    0.1100  0.1098
    0.1200  0.1197
    0.1300  0.1296
    0.1400  0.1395
    0.1500  0.1494
    0.1600  0.1593
    0.1700  0.1692
    0.1800  0.1790
    0.1900  0.1889

We've printed out the sin function to four digits of precision in both the x and y values. For the y values, we specified that the column was to be 7 spaces wide, so that the decimal points would line up.

## Reading text files

Next we come to the inverse operation: reading in a text file. Let's open up the file we just wrote in read mode:

```python
 f = open('sin.txt', 'r')
```

We can read a single line from the file in as a string with the readline() command:

```python
f.readline()
```

    '0.0000  0.0000\n'

```python
f.readline()
```

    '0.0100  0.0100\n'

```python
f.readline()
```

    '0.0200  0.0200\n'

Note that each call to readline() automatically moves the pointer in the file forward, so that the next invocation of readline() reads in the next line. If one reaches the end of the file, then calling readline() just produces an empty string.

(Note to those familiar with languages like C++: there is no EOF-checking function in python, and reading beyond the end of a file does not produce an error. Python's general philosophy about things like this is "It is better to ask forgiveness than permission", meaning that we don't handle things like the ends of files by testing for them, we handle them by trying the operation we want to perform, and seeing if it succeeds.)

If you want to go back to the beginning, you can use the seek() function:

```python
f.seek(0)
```

    0

The argument is the offset from the start of the file, measured in bytes. Thus a value of 0 points back to the start of the file.

One can also read the file contents in a couple of other ways. The function readlines(), as opposed to readline(), reads all the lines of the file into a list, with each item in the list corresponding to a separate line. For example:

```python
contents = f.readlines()
contents[0]
```

    '0.0000  0.0000\n'

```python
contents[1]
```

    '0.0100  0.0100\n'

```python
contents[2]
```

    '0.0200  0.0200\n'

```python
len(contents)
```

    629

One can also iterate over a file using a for loop, which reads in the lines one by one. Here's an example of using this approach to read our sin function back into a pair of x and y arrays that we can then plot:

```python
f.seek(0)        # Back to the beginning
xinput = []      # Create an empty list to receive the x values
yinput = []      # Ditto for the y values
for line in f:   # Read the file
    spl = line.split()              # Break up the line into two parts
    xinput.append(float(spl[0]))    # Add first part to xinput list
    yinput.append(float(spl[1]))    # Add second part to yinput list

f.close()                   # We're done with this file
xinput = np.array(xinput)   # Convert x list to array
yinput = np.array(yinput)   # Same for y list
plt.plot( xinput, yinput )  # Plot
```

Let's break down this block of code to understand what it does. The first line just returns us to the start of the file. The next two lines create empty lists into which we will place the x and y values we're reading. The last three lines convert the x and y lists into arrays and plot them.

The fourth line is the meat of the operation. The construct "for line in f:" iterates over the file just like we can iterate over a list, with each line of file acting like an element of the list. (We would have gotten exactly the same result by doing "for line in contents:", but the construct "for line in f:" is generally preferable, because it will work even for files that are too large to hold the entire thing in memory at once.) Inside the loop, we use the string function split() to break the string up into two pieces separated by whitespace, and then we append those two pieces to the x and y lists, respectively. When we're done, we have all the lines of the file read into those lists.

Reading text files is a common enough operation that, not surprisingly, people have written code to perform operations of this sort before. One particularly flexible bit of code for reading in tables of data stored in text form is the ASCII table reader that is included in the [astropy](http://www.astropy.org/) package of python routines for astronomy. To read the data using astropy, we can do the following:

```python
from astropy.io import ascii

data = ascii.read('sin.txt')
type(data)
```

    astropy.table.table.Table

The first line imports that ascii package from astropy.io (the input/output part of astropy). The second reads the data table. Astropy is smart enough to guess many common formats, and has no trouble with the simple one we've constructed. The resulting object is stored in a variable called data, and we can see that it is of type table.

In a table like ours without any headers, the columns of the table are by default named col1, col2, etc. We can access these from the data object using an interface like dict, where we give the name of the column. The data can then be plotted. For example, we can do

```python
plt.clf()
plt.plot( data['col1'], data['col2'] )
```

## Binary files: writing

The files we've dealt with so far are text files, consisting of sets of characters. While text files have the virtue of being mostly human-readable, text format is very inefficient in terms of the number of bytes required to store data. For example, let's think about a standard python floating point value. On most modern architectures a float is 64 bits long. While a float is 64 bits (= 8 bytes) of information, how much space does it take to store that information as text?

For the way floats are stored, 64 bits translates to 15 digits decimal places of precision and 3 digits in the base-10 exponent. For example, on my laptop, the largest representable float (obtained by typing sys.float_info at the python prompt, for those interested) is 1.7976931348623157e+308. (For the astute, notice that this is 16 decimal places; the last one is not precise because we cannot accurately place arbitrary values in it.) Thus in text format, to represent an arbitrary a floating point number in exponential notation we need something of the form X.XXXXXXXXXXXXXXXe+XXX. This is 22 characters, and each ASCII text character requires 1 byte to represent it, so representing this 8 byte number in ASCII requires 22 bytes of storage -- and that's not counting the extra spaces, carriage returns, or other formatting stuff we might need to render a list of such numbers legible.

Thus storing floats as ASCII requires a minimum of 2.75 times as much space as the actual data itself. Clearly this is a waste of disk space, and will also unnecessarily slow down file reading, transfer, etc. For this reason, data sets of any substantial size are always stored as binary rather than text data.

Python allows one to handle binary as well as text data. To deal with binary data, when opening a file one must specify that it is to be opened in binary mode. The main difference between binary and text mode is that, in text mode, one can write data other than strings, and it will not be converted to any sort of string representation before it is written. (There are other, subtle, platform-dependent differences as well, which we will not get into.) To open a file in binary mode, just append a "b" after the "r", "w", or "a" specifying the read/write/append mode.

As an example, we can write out our sin function data to a binary file as follows:

```python
f = open('sin.dat', 'wb')
f.write(x)
f.write(np.sin(x))
f.close()
```

These commands will create a binary file called sin.dat, which contains the values stored in the array x, followed by the values for sin(x). We can't look at this data directly using the more command, because it's not stored in a format that is human-readable, but we can verify that the right amount of data is there by doing the following at the command line (note that this assumes mac or unix):

```python
!ls -l sin.dat
```

    -rw-r--r-- 1 bruno bruno 10064 Jan 12 17:06 sin.dat

The number 10064 is the number of bytes in the file, and you can very quickly verify that this is 629 * 2 * 8 -- here 629 is the number of elements in x, 2 is for two arrays (x and sin(x)), and 8 is the number of bytes for each float. So the file is the right size at least. If you compare this to the size of the text file sin.txt, you will notice that sin.txt is actually almost exactly the same size (about 5% smaller), but this is just because we only wrote out the text data to 4 decimal places, and thereby threw away a huge amount of information. The binary data file contains 15 digits of precision, not 4, in the same amount of disk space.

## Binary files: reading

The downside to the compactness of binary format is that reading the data back in is considerably more complex. The problem is that a binary file is nothing but a string of bytes. There's no formatting information specifying whether they are floats, integers, or something else. Thus you have to know how to interpret them in order to turn them into something useful. In python, this procedure is done in two steps.

The first step is to read in the raw data using the read() command. Let's read in the array of x values for our example file:

```python
f = open('sin.dat', 'rb')
xraw = f.read(629*8)
type(xraw)
```

    bytes

Note that the read command takes as an argument the number of bytes to read in; if one omits this argument, the entire file is read. In this example, we know that the file contains two arrays of 629 numbers each, corresponding to the array of x values, and then the array of sin(x) values. Thus 629*8 is the number of bytes used to store the x array, and the command f.read(629*8) reads in this data.

The data are placed into the variable xraw. This variable is a string, but really it's just a representation of the stream of bits that represents our array. To turn it into a set of floating point numbers that we can actually use, we need to tell python explicitly how to interpret this stream of bits. Python provides a module called struct for performing this sort of operation. Here's how we can use it:

```python
import struct

xin = np.array(struct.unpack('d'*629, xraw))
```

After importing the struct module, we call the function it provides called unpack(). The unpack() function converts streams of raw bits like we just read into numbers. The second argument to unpack() is the stream of bits to be converted, while the first argument is a string specifying how they are to be interpreted. In this example, the letter d means that they are to be interpreted as double precision numbers (meaning 64-bit floats; this is called double precision for [historical reasons](http://en.wikipedia.org/wiki/Double-precision_floating-point_format) having to do with computer architecture), while the *629 makes a string of 629 d's, since there are 629 such numbers. Finally, the array() command says to take these 629 numbers, which come out as a tuple, and convert them to a numpy array suitable for plotting.

To finish up, we can do the same for the sin(x) array, and then plot:

```python
yraw = f.read(629*8)
f.close()
yin = np.array(struct.unpack('d'*629, yraw))
plt.clf()
plt.plot( xin, yin )
```

The result is the same as when we stored the data in text format (or at least indistinguishable by eye -- the text data is much less precise because we only stored 4 decimal places, but the differences too small an amount to see):

# Specialized File Formats

The choice between binary and text files is somewhat unappetizing. Text files are easy to read by humans, but are impractically-large and inefficient for substantial amounts of data. Binary files are compact, but you have to know exactly what they contain to get anything useful out of them. Is there an alternative? Yes!

This is what specialized file formats are for. They are file formats that consist (usually) of all or mostly binary data, but in an agreed-upon format that is self-documenting, meaning that, if you know the format, the file itself contains enough information to interpret it the rest of it. A simple example of self-documentation would be if we started our example binary file by writing out the number of elements in the x array, and then writing the data, so that we wouldn't have to know in advance how long the arrays of x and sin(x) values are. This would only add a very small amount to the file size, but would make the data much more useful.

The number of file formats in the world is immense, but there are a few that are particularly useful for astronomical and/or python applications, which we'll discuss briefly.

## Numpy array files

The numpy library provides a standard tool for saving the contents of numpy arrays, and reading them back in: the save() and load() functions. Usage is very simple. To save an array, just do

```python
np.save( 'x.npy', x )
np.save( 'sinx.npy', np.sin(x) )
```

The first argument is the file name to save to. If you don't type in the extension .npy manually, one will be added for you. The second argument is the array to save.

Then to read the arrays back in, just do

```python
xin = np.load( 'x.npy' )
sinxin = np. load( 'sinx.npy' )
```

That's it.

## Pickling and pickle files

The numpy array format is fine for saving numpy arrays, but suppose that we want to save something else. Is there a way to do that? Yes!

Python provides a technique called pickling. The [pickle](http://docs.python.org/2/library/pickle.html) module provides tools to take an arbitrary variable or object in python, turn it into a stream of bytes that are then saved to a file, and reconstruct the object from a file. The procedure of taking an object and turning it into a file that can be saved is called pickling, and the procedure of unpacking the file to get back a python object that can be manipulated in a python program or session is called unpickling. Any built-in python variable can be pickled and unpickled, as can most user-defined objects, leaving out some subtle complications.

Here's an example. Suppose we make a dict that contains an array of x values and then its sin, cosine, and tangent:

```python
trigfuncs = {'x': x, 'sin': np.sin(x), 'cos': np.cos(x), 'tan': np.tan(x)}
```

We can save this object using pickle as follows:

```python
import pickle

f = open('trigfuncs.pkl', 'wb')
pickle.dump(trigfuncs, f)
f.close()
```

The first command here imports the pickle module. The second opens a file for writing in binary format; the standard extension for pickle files is .pkl. The third line invokes the pickle.dump() method, which writes the data to the file. The first argument is the data to be written, and the second is the file where it is to go. The final line closes the file.

To reconstruct the object, we just open the file again, and read it in using the pickle.load() command:

```python
f = open('trigfuncs.pkl', 'rb')
data=pickle.load(f)
f.close()
```

The first command opens the file, the second loads its contents into an object called data, and the third closes the file. We can readily verify that the object we've created, called data, is a dict that contains the same information as the original object trigfuncs:

```python
type(data)
```

    dict

```python
plt.plot( data['x'], data['cos'] )
```

## FITS image files

A second very common format in astronomical applications is FITS, which stands for Flexible Image Transport System. FITS is a file format that was originally designed to store astronomical images. It differs from most standard image formats in that it was set up to store, in addition to the image itself, a lot of metadata describing things like what instrument the image came from, how that instrument was configured, where in the sky it was pointed, etc. FITS is supported these days by the Goddard Space Flight Center (GSFC); more information about FITS can be found at http://fits.gsfc.nasa.gov/.

To experiment with FITS data, we need some to play with. You can download an example file [here](https://sites.google.com/a/ucsc.edu/krumholz/teaching-and-courses/python14/class-4/w0ck0101t_c0h.fit?attredirects=0&d=1). This particular file is an example from the GSFC website, at http://fits.gsfc.nasa.gov/fits_nraodata.html.

Once we've got the file downloaded and moved into the directory where we're running python, we're ready to look at it. The first step is to load the required libraries, which are part of the [astropy](http://www.astropy.org/) package of python routines for astronomy:

```python
from astropy.io import fits
```

Then we open the data file.

```python
hdulist = fits.open('w0ck0101t_c0h.fit')
```

The open command returns a list of HDUs, short for header data units. An HDU is a block of data. The most common format for FITS files, which this example follows, is that there are two HDUs, one containing an image, and one containing metadata describing that image. To see a summary of what the HDUs contain, we can do

```python
hdulist.info()
```

    Filename: w0ck0101t_c0h.fit
    No.    Name      Ver    Type      Cards   Dimensions   Format
      0  PRIMARY       1 PrimaryHDU     196   (800, 800, 4)   int16 (rescales to float32)
      1  w0ck0101t_cvt.c0h.tab    1 TableHDU       194   4R x 37C   [D25.16, D25.16, D25.16, E15.7, E15.7, E15.7, E15.7, E15.7, E15.7, E15.7, E15.7, I11, E15.7, I11, I11, A24, A24, A8, A8, A8, I11, E15.7, E15.7, E15.7, E15.7, I11, I11, I11, I11, I11, I11, I11, A24, E15.7, E15.7, E15.7, E15.7]

This output tells us that there are two HDUs. The first one, called PRIMARY, consists of an array of 800 x 800 x 4 floating point numbers. The type of HDU is PrimaryHDU, which is a type that corresponds to an astronomical image, or, in this case, four images. (There are four because this particular data is from an instrument on the Hubble Space Telescope that had four CCD chips, and the data are recorded separately for each one.) The second HDU, called w0ck0101t_cvt.c0h.tab, is of type TableHDU. It is a table consisting of 194 entries. That's the metadata.

We can get a look at the metadata by printing out the header information associated with the table:

```python
hdulist[1].header
```

    XTENSION= 'TABLE   '           / FITS STANDARD
    BITPIX  =                    8 /  8-bits per 'pixels'
    NAXIS   =                    2 /  Simple 2-D matrix
    NAXIS1  =                  584 /  No of characters per row
    NAXIS2  =                    4 /  The number of rows
    PCOUNT  =                    0 /  No 'random' parameters
    GCOUNT  =                    1 /  Only one group
    TFIELDS =                   37 /  Number of fields per row
    EXTNAME = 'w0ck0101t_cvt.c0h.tab' / Name of table
    TTYPE1  = 'CRVAL1  '           /
    TBCOL1  =                    1 /
    TFORM1  = 'D25.16  '           / %25.16g
    TUNIT1  = '        '           /
    TTYPE2  = 'CRVAL2  '           /
    TBCOL2  =                   27 /
    TFORM2  = 'D25.16  '           / %25.16g
    TUNIT2  = '        '           /
    TTYPE3  = 'CRVAL3  '           /
    TBCOL3  =                   53 /
    TFORM3  = 'D25.16  '           / %25.16g
    TUNIT3  = '        '           /
    TTYPE4  = 'CRPIX1  '           /
    TBCOL4  =                   79 /
    TFORM4  = 'E15.7   '           / %15.7g
    TUNIT4  = '        '           /
    TTYPE5  = 'CRPIX2  '           /
    TBCOL5  =                   95 /
    TFORM5  = 'E15.7   '           / %15.7g
    TUNIT5  = '        '           /
    TTYPE6  = 'CD1_1   '           /
    TBCOL6  =                  111 /
    TFORM6  = 'E15.7   '           / %15.7g
    TUNIT6  = '        '           /
    TTYPE7  = 'CD1_2   '           /
    TBCOL7  =                  127 /
    TFORM7  = 'E15.7   '           / %15.7g
    TUNIT7  = '        '           /
    TTYPE8  = 'CD2_1   '           /
    TBCOL8  =                  143 /
    TFORM8  = 'E15.7   '           / %15.7g
    TUNIT8  = '        '           /
    TTYPE9  = 'CD2_2   '           /
    TBCOL9  =                  159 /
    TFORM9  = 'E15.7   '           / %15.7g
    TUNIT9  = '        '           /
    TTYPE10 = 'DATAMIN '           /
    TBCOL10 =                  175 /
    TFORM10 = 'E15.7   '           / %15.7g
    TUNIT10 = '        '           /
    TTYPE11 = 'DATAMAX '           /
    TBCOL11 =                  191 /
    TFORM11 = 'E15.7   '           / %15.7g
    TUNIT11 = '        '           /
    TTYPE12 = 'MIR_REVR'           /
    TBCOL12 =                  207 /
    TFORM12 = 'I11     '           / %11d
    TUNIT12 = '        '           /
    TTYPE13 = 'ORIENTAT'           /
    TBCOL13 =                  219 /
    TFORM13 = 'E15.7   '           / %15.7g
    TUNIT13 = '        '           /
    TTYPE14 = 'FILLCNT '           /
    TBCOL14 =                  235 /
    TFORM14 = 'I11     '           / %11d
    TUNIT14 = '        '           /
    TTYPE15 = 'ERRCNT  '           /
    TBCOL15 =                  247 /
    TFORM15 = 'I11     '           / %11d
    TUNIT15 = '        '           /
    TTYPE16 = 'FPKTTIME'           /
    TBCOL16 =                  259 /
    TFORM16 = 'A24     '           / %-24s
    TUNIT16 = 'CH*24   '           /
    TTYPE17 = 'LPKTTIME'           /
    TBCOL17 =                  284 /
    TFORM17 = 'A24     '           / %-24s
    TUNIT17 = 'CH*24   '           /
    TTYPE18 = 'CTYPE1  '           /
    TBCOL18 =                  309 /
    TFORM18 = 'A8      '           / %-8s
    TUNIT18 = 'CH*8    '           /
    TTYPE19 = 'CTYPE2  '           /
    TBCOL19 =                  318 /
    TFORM19 = 'A8      '           / %-8s
    TUNIT19 = 'CH*8    '           /
    TTYPE20 = 'CTYPE3  '           /
    TBCOL20 =                  327 /
    TFORM20 = 'A8      '           / %-8s
    TUNIT20 = 'CH*8    '           /
    TTYPE21 = 'DETECTOR'           /
    TBCOL21 =                  336 /
    TFORM21 = 'I11     '           / %11d
    TUNIT21 = '        '           /
    TTYPE22 = 'DEZERO  '           /
    TBCOL22 =                  348 /
    TFORM22 = 'E15.7   '           / %15.7g
    TUNIT22 = '        '           /
    TTYPE23 = 'GOODMIN '           /
    TBCOL23 =                  364 /
    TFORM23 = 'E15.7   '           / %15.7g
    TUNIT23 = '        '           /
    TTYPE24 = 'GOODMAX '           /
    TBCOL24 =                  380 /
    TFORM24 = 'E15.7   '           / %15.7g
    TUNIT24 = '        '           /
    TTYPE25 = 'DATAMEAN'           /
    TBCOL25 =                  396 /
    TFORM25 = 'E15.7   '           / %15.7g
    TUNIT25 = '        '           /
    TTYPE26 = 'GPIXELS '           /
    TBCOL26 =                  412 /
    TFORM26 = 'I11     '           / %11d
    TUNIT26 = '        '           /
    TTYPE27 = 'SOFTERRS'           /
    TBCOL27 =                  424 /
    TFORM27 = 'I11     '           / %11d
    TUNIT27 = '        '           /
    TTYPE28 = 'CALIBDEF'           /
    TBCOL28 =                  436 /
    TFORM28 = 'I11     '           / %11d
    TUNIT28 = '        '           /
    TTYPE29 = 'STATICD '           /
    TBCOL29 =                  448 /
    TFORM29 = 'I11     '           / %11d
    TUNIT29 = '        '           /
    TTYPE30 = 'ATODSAT '           /
    TBCOL30 =                  460 /
    TFORM30 = 'I11     '           / %11d
    TUNIT30 = '        '           /
    TTYPE31 = 'DATALOST'           /
    TBCOL31 =                  472 /
    TFORM31 = 'I11     '           / %11d
    TUNIT31 = '        '           /
    TTYPE32 = 'BADPIXEL'           /
    TBCOL32 =                  484 /
    TFORM32 = 'I11     '           / %11d
    TUNIT32 = '        '           /
    TTYPE33 = 'PHOTMODE'           /
    TBCOL33 =                  496 /
    TFORM33 = 'A24     '           / %-24s
    TUNIT33 = 'CH*24   '           /
    TTYPE34 = 'PHOTFLAM'           /
    TBCOL34 =                  521 /
    TFORM34 = 'E15.7   '           / %15.7g
    TUNIT34 = '        '           /
    TTYPE35 = 'PHOTZPT '           /
    TBCOL35 =                  537 /
    TFORM35 = 'E15.7   '           / %15.7g
    TUNIT35 = '        '           /
    TTYPE36 = 'PHOTPLAM'           /
    TBCOL36 =                  553 /
    TFORM36 = 'E15.7   '           / %15.7g
    TUNIT36 = '        '           /
    TTYPE37 = 'PHOTBW  '           /
    TBCOL37 =                  569 /
    TFORM37 = 'E15.7   '           / %15.7g
    TUNIT37 = '        '           /
    CRVAL1  = ' right ascension of reference pixel' /
    CRVAL2  = ' declination of reference pixel' /
    CRVAL3  = ' first packet time' /
    CRPIX1  = ' x-coordinate of reference pixel' /
    CRPIX2  = ' y-coordinate of reference pixel' /
    CD1_1   = ' partial of the right ascension w.r.t. x' /
    CD1_2   = ' partial of the right ascension w.r.t. y' /
    CD2_1   = ' partial of the declination w.r.t. x' /
    CD2_2   = ' partial of the declination w.r.t. y' /
    DATAMIN = ' minimum value of the data' /
    DATAMAX = ' maximum value of the data' /
    MIR_REVR= ' is the image mirror reversed?' /
    ORIENTAT= ' orientation of the image in degrees' /
    FILLCNT = ' number of segments containing fill' /
    ERRCNT  = ' number of segments containing errors' /
    FPKTTIME= ' time of the first packet' /
    LPKTTIME= ' time of the last packet' /
    CTYPE1  = ' first coordinate type' /
    CTYPE2  = ' second coordinate type' /
    CTYPE3  = ' third coordinate type' /
    DETECTOR= ' CCD detector: WFC 1-4, PC 5-8' /
    DEZERO  = ' Bias level from EED extended register' /
    GOODMIN = ' minumum value of the "good" pixels' /
    GOODMAX = ' maximum value of the "good" pixels' /
    DATAMEAN= ' mean value of the "good" pixels' /
    GPIXELS = ' number of "good" pixels               (DQF=0)' /
    SOFTERRS= ' number of "soft error" pixels         (DQF=1)' /
    CALIBDEF= ' number of "calibration defect" pixels (DQF=2)' /
    STATICD = ' number of "static defect" pixels      (DQF=4)' /
    ATODSAT = ' number of "AtoD saturated" pixels     (DQF=8)' /
    DATALOST= ' number of "data lost" pixels          (DQF=16)' /
    BADPIXEL= ' number of "generic bad" pixels        (DQF=32)' /
    PHOTMODE= ' Photometry mode'   /
    PHOTFLAM= ' Inverse Sensitivity' /
    PHOTZPT = ' Zero point'        /
    PHOTPLAM= ' Pivot wavelength'  /
    PHOTBW  = ' RMS bandwidth of the filter' /

The output here tells us about the entries in the table. We can access them using the attribute data, and asking about particular data values. For example, two of the entries are CRVAL1 and CRVAL2, which give the right ascension and declinations (celestial coordinates) of a reference pixel. We can print those out as follows:

```python
hdulist[1].data['CRVAL1']
```

    array([291.02797853, 291.02797853, 291.02797853, 291.02797853])

```python
hdulist[1].data['CRVAL2']
```

    array([-22.01122139, -22.01122139, -22.01122139, -22.01122139])

There are four entries because there are four images, though the reference pixel positions are the same for all of them in this example. This data says that the right ascension of the reference pixel is 291.02797853 degrees, and the declination is -22.01122139 degrees.

The image itself is stored in the first HDU, and we can use the imshow command to display it. Let's do that, using a logarithmic color scale:

```python
plt.clf()
plt.imshow( np.log(hdulist[0].data[0,:,:]) )
```

Note that the [0,:,:] is to specify that we want to see the first of the four images. Also notice the warning; that occurred because some of the pixel values are negative due to instrument noise, and taking the log of a negative number if of course undefined. However, we don't have to worry about that.

This particular example is an image of Saturn, one of the first ever made by HST.

When we're done with the FITS file, we close it via

```python
hdulist.close()
```

# Basic Statistics

## Descriptive statistics

We now move to the second topic of today's class. Now that we know how to get data in and out of a python session, what can we do with it? We can do many things of course, but one of the most basic is to perform some statistical analysis of it.

Let's start by making ourselves some data to play with. Since we want this to be like real data, we'll be sure to add some noise to it.

```python
noise = np.random.normal( scale=0.25, size=len(x) )
y = np.sin(x) + noise
plt.clf()
plt.plot( x, y, 'o' )
```

To make the noise, we used the numpy function random.normal(). This picks random numbers from a normal (Gaussian) distribution with a dispersion sigma given by the keyword scale. The keyword size specifies how many random values to pick, and in this case we set it to as many elements as there were in x. Then we made a y value by setting it equal to sin(x) plus the noise.

Note that there are other numpy random routines that will generate random numbers with different distributions.

Once we have this data, there are numerous numpy routines that we can use to analyze it. The functions below all come from the [numpy statistics package](http://docs.scipy.org/doc/numpy/reference/routines.statistics.html).

We've already encountered the amin and amax routines, which find the minimum and maximum value. We can also find the mean, median, and standard deviation just as easily:

```python
np.mean(y)
```

    2.2357252709125184e-05

```python
np.median(y)
```

    -0.013349835524341103

```python
np.std(y)
```

    0.7653375596903697

We can also find percentiles, i.e., the value below which p percent of the sample lies. For example,

```python
np.percentile( y, [25,50,70] )
```

    array([-0.64566341, -0.01334984,  0.57541746])

This gives the value below which 25%, 50%, and 75% of the values in y lie. Note that the 50th percentile is the same as the median.

Finally, we can put the data in bins and count occurrences. The command to do this is called [histogram()](http://docs.scipy.org/doc/numpy/reference/generated/numpy.histogram.html), and it works as follows:

```python
hist, edges = np.histogram( y, range=(-1.5, 1.5), bins=30 )
```

```python
edges
```

    array([-1.5, -1.4, -1.3, -1.2, -1.1, -1. , -0.9, -0.8, -0.7, -0.6, -0.5,
           -0.4, -0.3, -0.2, -0.1,  0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,
            0.7,  0.8,  0.9,  1. ,  1.1,  1.2,  1.3,  1.4,  1.5])

```python
hist
```

    array([ 4,  8, 16, 20, 22, 18, 22, 30, 36, 31, 29, 23, 22, 19, 20, 20, 15,
           20, 18, 22, 32, 30, 30, 26, 40, 16, 14, 11, 10,  3])

The histogram command takes as input the data to be histogrammed, and then keywords specifying the range for the bins (in this case -1.5 to 1.5) and the number of bins (in this case 30, so that each bin will be 0.1 wide). One can also specify the edges of the bins manually. The output is a tuple of two items. The first is the histogram itself, which we've called hist, and the second is the values of the bin edges, which we've called edges. Upon return, hist contains the number of array elements in each bin.

We can plot this data as a bar plot, using the bar command in matplotlib.

```python
plt.clf()
plt.bar( edges[:-1], hist, width=0.1 )
```

## Curve fitting

What we've been doing thus far is descriptive statistics: taking a set of data and calculating various quantities that describe it. The final topic for today is model fitting: using a data set to derive a model for how the underlying phenomenon behaves. The simplest application of this is to fit curves through data. Before discussing how to do this in python, we must review the basic theory of curve fitting first.

Curve fitting is a problem of finding the minimum of a function. Suppose that we have a set of measured data points (x1, y1), (x2, y2), (x3, y3), .... The data are not perfect. They consist of a measurement plus some amount of noise. We have some model of the data, which we can describe by a function y(x; p), where p is a set of one or more parameters describing the model. For example, we might wish to measure the strength of the surface gravitational field on a planet. We know that a point mass dropped in a constant gravitational field in vacuum will fall a distance

\begin{equation}
d = \frac{1}{2} g t^2
\end{equation}

in a time $t$. To measure $g$, we can drop and object and record the distance $d$ it has travelled at a variety of times $t$. Given these measurements, we want to find the value of $g$ that we should infer from the data; here $g$ is the parameter to be fit given the ($t$, $d$) data.

Under fairly general assumptions about the nature of the error in the measurements, one can show that the the "best" fit is given by the value of $g$ that minimizes the squared distance between the model and the data. That is, we can define a function for the error by

\begin{equation}
e^2(g) = \sum \left[ \frac{1}{2} g t_i^2 - d_i \right]^2
\end{equation}

where the sum runs over all the measurements $i$. The best value of $g$ is the one for which $e^2(g)$ reaches its minimum value.

More generally, for a set of measurements (x1, y1), (x2, y2), (x3, y3), .... and a model y(x; **p**), we wish to find the set of values **p** that minimizes the multi-dimensional function

\begin{equation}
e^2(\mathbf{p}) = \sum \left[ y_i - y(x_i; \mathbf{p}) \right]^2.
\end{equation}

If the data points do not all have the same error, so that some are more reliable than others, this can be generalize to minimizing

\begin{equation}
e^2(\mathbf{p}) = \sum \frac{ \left[ y_i - y(x_i; \mathbf{p}) \right]^2 }{ \sigma_i^2},
\end{equation}

where $\sigma_i$ is the error on data point $i$.

In the case where $\mathbf{p}$ is only a single parameter, this problem is fairly straightforward, although for sufficiently complex models it can still be tricky as the function may have more than one minimum, and our goal is to find the global minimum. In multiple dimensions the problem is considerably trickier, and the problem that there might be multiple local minima that we must sort through to find the global minimum is significantly worse.

The operation of attempting to find the best-fitting parameter is sufficiently common that numpy and scipy provide routines to do it. The simplest of these fits to a particular function $y(x; \mathbf{p})$: a polynomial. That is, a function of the form

\begin{equation}
y(x; \mathbf{p}) = p_0 + p_1 x + p_2 x^2 + p_3 x^3 + ... + p_N x^N.
\end{equation}

Here the value $N$ is the degree of the polynomial. The numpy routine that performs this operation is called [polyfit](http://docs.scipy.org/doc/numpy/reference/generated/numpy.polyfit.html). Here is an example of how it can be used:

```python
fit = np.polyfit( x, y, 1 )
fit
```

    array([-0.31065589,  0.97548184])

The first argument is the array of x values, the second is the array of y values, and the third is the degree of the polynomial to be fit. In this case we have fit a first-order polynomial, i.e., a straight line. The output is the array of polynomial coefficients, starting with the highest-order one and proceeding to the lowest. Thus polyfit has returned a best-fitting function

```python
function = 'y = {} + {} x '.format( fit[1], fit[0] )
print( function )
```

    y = 0.9754818447791317 + -0.3106558877472683 x

To see how this compares to the data, we can do

```python
plt.clf()
plt.plot( x, y, 'o' )
plt.plot( x, fit[0]*x + fit[1], lw=5 )
```

Clearly this is not a very good fit, which illustrates an important point: just because you can find a best fit, doesn't mean it's a good one

We can try a somewhat more complex function, say a 12th order polynomial, and see if that does better:

```python
fit_2 = np.polyfit( x, y, 12 )
y_fit = np.zeros( len(x) )
for i in range( 13 ):
    y_fit = y_fit + fit_2[i]*x**(12-i)

plt.clf()
plt.plot( x, y, 'o' )
plt.plot( x, y_fit, lw=5 )
```

Clearly this is a better fit.

We can also fit functional forms other than polynomials. For example, suppose we knew, perhaps based on some theoretical model, that the data we are looking at should follow a sin function, but one with unknown wavelength, phase, amplitude, and offset from zero. That is, we want to fit to a functional form

\begin{equation}
y(x; \mathbf{p}) = p_0 + p_1 sin[p_2 (x - p_3)].
\end{equation}

The scipy routine curve_fit provides this capability. We use this routine in two steps. First, we must define the function we want to fit:

```python
def sinfunc(x, p0, p1, p2, p3):
    return p0 + p1*np.sin(p2*(x-p3))
```

This function must take an argument x giving the value of the independent variable, and then some arbitrary number of additional parameters. Then we import the curve_fit function from the scipy.optimize module, and call it:

```python
from scipy.optimize import curve_fit

p, pcov = curve_fit( sinfunc, x, y )
```

The curve_fit function returns a tuple of two values, which here we have stored to p and pcov. The first is the set of optimized parameters p. The second is what is called the [covariance matrix](http://en.wikipedia.org/wiki/Covariance_matrix), which is a 2d array the describes the variance in the parameter estimate.

```python
p
```

    array([ 2.12779163e-05,  1.02433660e+00,  9.99284812e-01, -1.15036741e-02])

We can see $\mathbf{p}$ is a very good fit to what we put in: the offset is almost zero, the amplitude is almost one, the wavelength is almost $2 \pi$ (corresponding to $p_2 = 1$), and the phase shift is almost zero. A plot of the fit looks very good too:

```python
plt.clf()
plt.plot( x, y, 'o' )
plt.plot(x, sinfunc(x, p[0], p[1], p[2], p[3]), lw=5)
```


Probabilistic Programming
=====
and Bayesian Methods for Hackers
========

##### Version 0.1

`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`
___

Welcome to *Bayesian Methods for Hackers*. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code, but still not certain.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

### The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by preserving *uncertainty*. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: *Frequentist*, known as the more *classical* version of statistics, assume that probability is the long-run frequency of events (hence the bestowed title). For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of *belief*, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate *A* will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an *individual*, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is *your* belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability*.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being Heads. $P(A | X):\;\;$ You look at the coin, observe a Heads has landed, denote this information $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

### Bayesian Inference in Practice

 If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return *probabilities*.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a *YES*. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of *YES* and *NO*. The function might return:

>    *YES*, with probability 0.8; *NO*, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument:  *"Often my code has bugs"*. This parameter is the *prior*. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

#### Incorporating evidence

As we acquire more and more instances of evidence, our prior belief is *washed out* by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an *infinite* amount of evidence, say as $N \rightarrow \infty$, our Bayesian results (often) align with frequentist results. Hence for large $N$, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more *unstable*: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we *preserve the uncertainty* that reflects the instability of statistical inference of a small $N$ dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[1], before making such a decision:

> Sample sizes are never large. If $N$ is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$ is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$ is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

### Are frequentist methods incorrect then?

**No.**

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

#### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?")

The much more difficult analytic problems involve *medium data* and, especially troublesome, *really small data*. Using a similar argument as  Gelman's above, if big data problems are *big enough* to be readily solved, then we should be more interested in the *not-quite-big enough* datasets.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to })
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```python
"""
The book uses a custom matplotlibrc file, which provides the unique styles for
matplotlib plots. If executing this book, and you wish to use the book's
styling, provided are two options:
    1. Overwrite your own matplotlibrc file with the rc-file provided in the
       book's styles/ dir. See http://matplotlib.org/users/customizing.html
    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to
       update the styles in only this notebook. Try running the following code:

        import json
        s = json.load(open("../styles/bmh_matplotlibrc.json"))
        matplotlib.rcParams.update(s)

"""

# The code below can be passed over, as it is currently not important, plus it
# uses advanced topics we have not covered yet. LOOK AT PICTURE, MICHAEL!
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials)/2, 2, k+1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head?). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

##### Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```python
figsize(12.5, 4)
p = np.linspace(0, 1, 50)
plt.plot(p, 2*p/(1+p), color="#348ABD", lw=3)
#plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2*(0.2)/1.2, s=140, c="#348ABD")
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title("Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```python
figsize(12.5, 4)
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior = [1./3, 2./3]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])

plt.bar([0+0.25, .7+0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label="posterior distribution",
        lw="3", edgecolor=colours[1])

plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title("Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

## Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability distribution function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

### Discrete Case
If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```python
figsize(12.5, 4)

import scipy.stats as stats
a = np.arange(16)
poi = stats.poisson
lambda_ = [1.5, 4.25]
colours = ["#348ABD", "#A60628"]

plt.bar(a, poi.pmf(a, lambda_[0]), color=colours[0],
        label="$\lambda = %.1f$" % lambda_[0], alpha=0.60,
        edgecolor=colours[0], lw="3")

plt.bar(a, poi.pmf(a, lambda_[1]), color=colours[1],
        label="$\lambda = %.1f$" % lambda_[1], alpha=0.60,
        edgecolor=colours[1], lw="3")

plt.xticks(a + 0.4, a)
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values");
```

### Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```python
a = np.linspace(0, 4, 100)
expo = stats.expon
lambda_ = [0.5, 1]

for l, c in zip(lambda_, colours):
    plt.plot(a, expo.pdf(a, scale=1./l), lw=3,
             color=c, label="$\lambda = %.1f$" % l)
    plt.fill_between(a, expo.pdf(a, scale=1./l), color=c, alpha=.33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0,1.2)
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

### But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```python
figsize(12.5, 3.5)
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
plt.bar(np.arange(n_count_data), count_data, color="#348ABD")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to PyMC3, a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

Introducing our first hammer: Pyro
-----

```python
from jax import random
import jax
import jax.numpy as np
import numpyro as pyro
import numpyro.distributions as dist
```

```python
count_data = jax.numpy.array(count_data)
```

```python
def model(data):
    alpha = (1. / data.mean())
    lambda1 = pyro.sample("lambda_1", dist.Exponential(rate=alpha))
    lambda2 = pyro.sample("lambda_2", dist.Exponential(rate=alpha))

    tau = pyro.sample("tau", dist.Uniform(0, 1))

    lambda12 = np.where(np.arange(len(data)) < tau * len(data), lambda1, lambda2)
    pyro.sample('obs', dist.Poisson(lambda12), obs=data)
```

```python
nuts_kernel = pyro.infer.NUTS(model)
posterior = pyro.infer.MCMC(nuts_kernel, num_samples=10000, num_warmup=5000) # 100x faster than pyro
rng_key = random.PRNGKey(0)
posterior.run(rng_key, count_data)
```

    sample: 100%|██████████| 15000/15000 [00:26<00:00, 569.63it/s, 1023 steps of size 2.41e-03. acc. prob=0.75]

```python
lambda_1_samples = posterior.get_samples()['lambda_1']
lambda_2_samples = posterior.get_samples()['lambda_2']
tau_samples = posterior.get_samples()['tau']
```

```python
figsize(12.5, 10)
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel("$\lambda_2$ value")

plt.subplot(313)
tau_samples = np.array((tau_samples * count_data.size + 1), dtype=np.int32)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data)-20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel("probability");
```

### Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```python
figsize(12.5, 5)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day = jax.ops.index_update(expected_texts_per_day, day, (lambda_1_samples[ix].sum() + lambda_2_samples[~ix].sum()) / N)

plt.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data)
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```python
#type your code here.
```

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```python
#type your code here.
```

3\. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45.  That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC3 part. Just consider all instances where `tau_samples < 45`.)

```python
#type your code here.
```

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg).
-  [2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).
- [3] Salvatier, J, Wiecki TV, and Fonnesbeck C. (2016) Probabilistic programming in Python using PyMC3. *PeerJ Computer Science* 2:e55 <https://doi.org/10.7717/peerj-cs.55>
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```