# Basic control structures and variables

## Nails for the hammer

# First experiment with Jupyter

* Click the blank box below saying `In [ ]`
* Press *Ctrl+Return*

```python
x = 1
```

Note that `#` is the comment symbol in Python - all text after it is ignored.

This lets you to make notes-to-self in your code. However, above, we want to Python to run that line, so don't forget to delete the `#` symbol before pressing _Ctrl+Return_

Click to edit the next `In [ ]` and press *Ctrl+Return* (without changing anything this time)

```python
2 * x
```

    2

Hopefully, you should notice that *x* is indeed what you input. This should all be pretty familiar, unless you come from statically-typed languages (C/C++/Java) where you would have to declare a variable. Python lets you effectively declare it, and its type, by assigning to it.

We can refer to this output as `Out[2]`, using it like any variable, without having to re-run everything so far.

Do the same trick in the next `In [ ]` (edit and *Ctrl+Return*)

```python
if 8 / 2 == x:
    print("Something else")
print("Second")
```

    Second

There are a couple of things here to dissect. Many languages need braces `{}`, brackets `[]` or parentheses `()` to draw boundaries around the argument of the `if` statement, or the body. In Python, symbols are reduced in favour of words and spaces, for readability. To achieve the same effect, Python separates the condition from the body with the colon `:` and the indentation (four spaces, by convention)

Indentation is Python's Marmite for other programmers. As they are keen to point out, most whitespace does *not* matter: e.g. I could write

`Out[2] / 2           ==    x  :`

and it's the same as

`Out[2] / 2 = x:`

However, indentation (whitespace at the start of a line) does. It is an extremely visually clear way to show where blocks of code, like the body of this `if` statement, start and end. For scientists, it forces a very good programming practice - basic code style - your code doesn't run without if it isn't visually clear (well... neatly indented). One of the reasons I got into Python was that my PhD C++ code was so unreadable, dodgily indented and unwieldy, it was easier to rewrite in Python than try and extend it - once I had modification was easy, I could see how everything was laid out, where functions started and ended, before even reading a character.

Another thing to note is that you can print with `print`. Here we are running Python3, the latest version. If you've played with Python2, you will notice this syntax is a bit different - it has parentheses (aka. parens) around the argument. A whole raft of changes came in with Python3 and, like XP, Python2 will be around for some time to come through sheer momentum. However by 2018, almost all important libraries are now Python3-ready and I, for one, develop new software in Python3 only, where possible. Good porting tools exist for moving old code.

To summarize those key points:

* Indentation separates bodies of functions, conditionals, loops from the outside code
* Basic boolean operators are `==`, `or`, `and`, `not`, `>`, `<`, etc.
* This is Python3 syntax (very slightly different to Python2)

# Debugging
## Ladder for the hole

Now we can edit and run code, the inevitable consequence is that we get a bug!

Try running the line below (without correcting it):

```python
if (x < 2) or (x > 100):
    print("Do I know you?')
else:
    print('Yes I do')
```

Great. Pretty colours! What do they mean?

In general, the most immediately useful information is on the very last line: `SyntaxError`. Above it, is the last "frame" called - that is, level of function. Say you call a function, and it calls another, and so forth until the innermost function hits an error. Here, it will list all the functions in that sequence, so you can track exactly how you got to the dodgy line.

In Jupyter, the filename is not very informative - but you can see the input number (e.g. `ipython-input-4` ⇆ `In [4]`).

Bonus mark if you spotted the error (not that we actually have marks, but hey) - have a look and see if you can fix it and re-run the command, the same way you ran it the first time. Feel free to discuss. Note that the `In [4]` changes to `In [5]`, and so forth, each time you run that "cell".

In general, "a string" and 'a string' are equivalent - Python does not mind whether single or double quotes are used, as long as they match at either end.

* Check the error at the bottom
* Check the caret (^) showing where Python sees a problem
* Look at the syntax highlighting in the cell

Like every language, Python has functions... they are defined like so:

```python
def tell_me_if_im_old(name, age):
    print("...checking if", name, "is old...")

    if age < 100:
        return False
    else:
        return True
```

Couple things to note: (i) this has nested blocks, we just indented twice for the inner ones (`if` and `else` bodies). (ii) this *didn't* specify a type for `age` (or `name`) - all it cares is that we can compare `age` to an *int*. Similarly, we don't specify a return type - it is up to us to return something intuitive and sensible. Let's use it:

```python
phils_age = 32
name = "Phil"

print("I am", phils_age, "- am I old?")

if tell_me_if_im_old(name, 150):
    print("Yup, it's a fact.")
else:
    print("Not at all, spring chicken.")
```

    I am 32 - am I old?
    ...checking if Phil is old...
    Yup, it's a fact.

This one I'm saving for future use. You can see that this is used pretty much like every function in many languages - name, parentheses (aka "parens") containing arguments.

There are other types of function - those on objects. Rather than cramming an introduction to object-oriented programming (OOP) now, especially as a number of you have done Computer Science, I have added an Appendix to these notes about my non-existent dogs, "Freddie" and "Nitwit". If OOP confuses you, we can use a bit of the break if you're brain isn't fried to go through those.

For the moment though, what matters is that you can add a dot to most things in Python and access a number of properties of that variable or things it can do. A couple of examples to make it clearer:

```python
"just a normal string".upper()
```

    'JUST A NORMAL STRING'

```python
if "anotheR string".islower():
    print("The string was lowercase")
```

# Lists and dicts
## Conveyor belts and storage units

Like any language, often need to be able to group things in Python.

```python
things_to_do = ['Learn Python', 'Finish PhD', 'Publish research',
                'Accept Nobel prize', 'Inspire a new generation']
```

The syntax here is the same as for the more basic types we saw so far, `variable = value`, and our list is a comma-separated series of *things*, bounded by brackets `[]`.

Once you execute the cell above, you should be able to try this:

```python
sorted(things_to_do)
```

    ['Accept Nobel prize',
     'Finish PhD',
     'Inspire a new generation',
     'Learn Python',
     'Publish research']

...and get an alphabetical list of tasks. Note that `sorted` doesn't update the original object, so if you re-run without `sorted`, it's back to the previous order.

To do this, you use a slightly different technique *or* you just assign: `things_to_do = sorted(things_to_do)`.

`sorted` guesses the ordering you want, based on the type of data. It doesn't all have to be strings, you could have mixture of strings and integers, say. `sorted` can take optional arguments, letting you force it to do things the way you want, or even provide your own ordering function.

# So how do we get things out of a list?

Lists are ordered, so you can say, "I'd like the fifth item, please". You put it in brackets after the list name:

```python
In[4]
```

    'if 8 / 2 == x:\n    print("Something else")\nprint(2/3)'

However, note that...

## PYTHON IS ZERO-INDEXED

Like C, but unlike MATLAB or FORTRAN. The first item in a list is "`things_to_do[0]`"

By the way, did you notice that `In[]` is a list?

```python
type(In)
```

You can try `len` as well!

```python
print(things_to_do)
things_to_do[1:4]
```

    ['Learn Python', 'Finish PhD', 'Publish research', 'Accept Nobel prize', 'Inspire a new generation']

    ['Finish PhD', 'Publish research', 'Accept Nobel prize']

...if you wonder why `things_to_do[4]` (the fifth item) didn't appear, note that that number after the colon means 'up-to-but-not-including'...

```python
things_to_do[:2]
```

    ['Learn Python', 'Finish PhD']

```python
things_to_do[3:]
```

    ['Accept Nobel prize', 'Inspire a new generation']

```python
things_to_do[-2:]
```

    ['Accept Nobel prize', 'Inspire a new generation']

We can use negative numbers to count back from the end instead...

Which is "start two from the end, and give me everything from there onwards"

This is fine - a list is like a conveyer belt, with a load of arbitrarily filled boxes passing by, one after the other. However, sometimes we care which item is which, and want to have a name for it, so we can drop in and grab it wherever it may be. That's where dictionaries (= dicts) come in. Try running this:

```python
my_meetup_dot_com_profile = {
    "first name": "Ignatius",
    "favourite number": 9,
    "favourite programming language": "FORTRAN66",
    3: "is the magic number"
}
```

The syntax is slightly different. Instead of brackets, we use braces `{}`.

Now each element has a name, followed by a colon and the element itself, which can still be basically anything. Like storage units, we now have a group of things we can address. That address could be a number, a string, or any other basic type. And keys can be different types - check the last entry.

```python
print("My favourite number is")
my_meetup_dot_com_profile['favourite number']
```

    My favourite number is

    9

Hmm... I'll let you fix that...

And even though it is unsorted, it can have integers or so forth as keys - note the very last entry in the definition...

```python
my_meetup_dot_com_profile[3]
```

    'is the magic number'

```python
my_meetup_dot_com_profile[2]
```

...but not...

Because it doesn't exist.

Extending lists is a little unwieldy, but dicts are more intuitive:

```python
things_to_do.append("Find a nice retirement village in the Galapagos Islands")

my_meetup_dot_com_profile["Interests"] = ["Python2", "Python3", "Scientific Python", "Pottery"]

print("TODO:", things_to_do, "\n\nMEETUP:", my_meetup_dot_com_profile)
```

    TODO: ['Learn Python', 'Finish PhD', 'Publish research', 'Accept Nobel prize', 'Inspire a new generation', 'Find a nice retirement village in the Galapagos Islands']

    MEETUP: {'first name': 'Ignatius', 'favourite programming language': 'FORTRAN66', 'favourite number': 9, 'Interests': ['Python2', 'Python3', 'Scientific Python', 'Pottery'], 3: 'is the magic number'}

Here, we added an `Interests` key to the `my_meetup_dot_com_profile` dictionary.

Note that anything can be an *item* in a dictionary or list - in this case, a list is an item in our dict, whose key is `Interests`. Also note that, as with other languages, `\n` signifies a newline.

# Join

## The most useful method in the str object

Different languages have different ways of collapsing a string with a separator - it is something you will need to do again and again. In Python, the approach is a little surprising - we have already seen that strings are objects (of type `str`), but it turns out Python uses this to provide a way of joining iterables (like `lists` or `dicts`):

```python
', '.join(my_meetup_dot_com_profile)
```

This way around, all the string needs to do is add itself before each item in the iterable (except the first) - as long as it can keep getting items, it doesn't care what the iterable is, a dict or anything (iterable == thing you iterate/step through).

Can you see the benefit of defining this method as part of `str`, being able to pass it any possible iterable to join up, instead of having to define a method on every type of iterable and passing it a string?

# Quick aside

## Save early, save often

Jupyter does do some automatic saving, but click the disk icon at the top left now to force an update...

# ReDebugging

## Bringing it together

Here's another one to see if you can fix - try running and then spot the issue:

```python
y = 120
for z in range(1, 5):
    y = y / z
    print("When z is", z, "then y is", y)
```

    When z is 1 then y is 120.0
    When z is 2 then y is 60.0
    When z is 3 then y is 20.0
    When z is 4 then y is 5.0

Output should be:
```
When z is 1 then y is 120.0
When z is 2 then y is 60.0
When z is 3 then y is 20.0
When z is 4 then y is 5.0
```

Couple of bits of useful information:

* `for` is, unsurprisingly, a `for` loop, as in other languages
* `for` doesn't have limit arguments, like in C, say - it takes a set/list/(anything iterable) and goes through each element
* `range` just returns a list of integers
* `range` can take one argument or two arguments (or more, but not so relevant now)
* `range` documentation is here : [Python3 range syntax](https://docs.python.org/3/library/functions.html#func-range) (link opens in new window, so click it!)

The easiest way, IMO, to info on any Python function or library is to Google "Python3 funcname" and click the first python.org link. If even something seems wrong, make sure you are looking at the Python3, not Python2 docs (or v.v.)

Note that the `in` operator can separately be used to return a boolean: e.g. "`list_item in alist`", "`key in adict`" can be used for `if` clauses

To see what range actually returns, you can run:

```python
type(range(5))
```

    range

Well, that wasn't very descriptive. It turns out the `range` function returns something with (a confusingly named) type `range` - this is what `for` gets handed. Don't worry about the ins-and-outs of that type just yet - what matters is: things of type `range` look like, act like and sound like a `list`.

As far as `for` is concerned, that's good enough - this is the [**duck test**](https://simple.wikipedia.org/wiki/Duck_typing), and is a key paradigm in Python - when writing code, don't require your input to be of a specific type `float`, or type `int` (for example), just complain if it doesn't *do* what you want (for example, it refuses to be added to something, or printed, by giving an error).

Here, `range` can iterate, like a list, which is all `for` wants - it doesn't care that it isn't _technically_ a list, as it can do what `for` needs. Similarly with, say, the `==` comparison operator - you can compare any objects and as long as they define their comparison behaviour, the operator itself doesn't care what they are.

To prove `range(5)` is what we expect it is, lets turn it into a list and see what happens...

```python
list(range(5))
```

    [0, 1, 2, 3, 4]

Essentially, `for` sees what we see - a sequence of five numbers.

Bear in mind that you can cast like this to various types. `str` will make something a string...

```python
str(range(5))
```

    'range(0, 5)'

```python
str(3.14) + " is almost pi"
```

    '3.14 is almost pi'

However, unlike some languages, Python does care about type, so you must concatenate strings with strings or add numerical types to numerical types. To check the type of any number or variable, you can use the built-in, `type`:

```python
type(str(3.14))
```

```python
my_meetup_dot_com_profile['favourite places'] = {
    'castle': 'Edinburgh Castle',
    'lake': 'Lake Superior'
}
def dict_to_string_dict(my_dict):
    new_dict = {}
    for key in my_dict:
        value = my_dict[key]
        if type(value) == type(['a', 'list']):
            new_dict[key] = ', '.join(value)
        else:
            new_dict[key] = str(value)
    return new_dict

adict = {'asd': ['foo', 'bar']}
dict_to_string_dict(adict)
```

    {'asd': 'foo, bar'}

Yes, yes it can: check out the '`for c in "YMCA"`'. We also snuck in a nested loop - you just keep indenting each time you nest - no more complicated than that.

And, admittedly, a new operator has appeared: "`+=`". Familiar to most languages it is equivalent to `d = d + blah`. Note that we therefore have to have `d` defined before the first `+=`, as "`d = d + anything`" doesn't make sense if `d` does not already exist. This is the reason for our first line: "`d = ""`".

Finally, note that we have looped twice using `range`, just as above, but our loop variable `i` is never used - it is effectively a placeholder. All we want from that line is to have the body below run twice.

*Sidenote*: some people use an underscore "`_`" as a loop variable in this case, to indicate to someone reading that the variable is nothing more than that and never used - as far as Python is concerned, "`_`" is as good a name for a variable as any other, so this is entirely about readability.

Here we have added one or two minor surprises. One is the *horrendously* misleading use of *len* (hint: you are unlikely to get a Nobel Prize by 2020), which in reality calculates the number of items in any "`iterable`". Reminder: this is the general term for something sort-of-list-like. According to my script, tasks take as long as the number of items in their "list", i.e. characters in the string.

Another aspect is the double slash. This is an important difference between Python3 and Python2 (and many other languages). [Dividing non-divisible integers](https://en.wikipedia.org/wiki/Division_%28mathematics%29#Of_integers) with one slash, as normal, gives a float in Python3. If you want to get an integer (that is, whatever the float answer is with the decimal chopped off), you can use double-slash. Try removing the second slash and re-running to see the exact non-integer answer. In Python2, using one slash *always* gives an integer, unless you cast the bottom or top to a float (as in C, say).

...if we have time...

Use the cell below to create a function that takes a dictionary as an argument (like `name` and `age` were in an earlier example), and returns a new dictionary, with the same keys, but where each original item has been converted to a string.

Any items of a ``list`` type should become a single string with entries separated by commas and no brackets at either side (e.g. "one, two, kumquat, four").

Call your function with "**`my_meetup_dot_com_profile`**" as an argument and print each entry on a separate line like so:

    FAVOURITE PROGRAMMING LANGUAGE : FORTRAN66
    FIRST NAME : Ignatius
    ...

where each key is capitalized and item value printed beside it. The order of lines is not important.

Note that you can create a blank dictionary with "`= {}`".
Extension: if you get that done quickly, try making your function call itself on any (nested) items it finds of `dict` type, and turn the output into the string-value you need for your new dictionary. This is "recursion".

Set
```
   my_meetup_dot_com_profile['dogs'] = {"Freddie": 5, "Nitwit": "also a dog"}
```
and run your function on "**`my_meetup_dot_com_profile`**" again to test your improvement.

```python
# Try the exercise in here
```

# Modules

## Gotta catch em all

```python

```

## Introduction to modules

* Where Python gets its power

Wealth of modules providing everything from full GUI toolkits to astrophysics simulations. Mature and reliable numerics libraries, and an evolving ecosystem that grows day by day

* Some modules come bundled, but you can install many more

Python has tools so you can just specify the name and it will get it from the appropriate online repository. On Windows, this can be tricky, but a recent Python distribution, Anaconda, makes it easy, and includes thousands of packages out of the box. This seems your best bet if on Windows or Mac and I have USB sticks here for you to get going at the end of the day.

* Some are part of Python, lots are third-party

It is good to have an idea of which is which, as you can get (normally free) help, often at pretty short notice, by heading to the project's forums - remember, always be polite and, bear in mind, it isn't a commercial service, so an answer isn't guaranteed. For reference, when someone is really desperate they can offer a bounty for a solution or a bit of code on certain websites, not to mention people offering consultancy support, so that doesn't entirely mean there are no other options, but most of the time what's freely available is more than adequate.

First off, you *import* a module. This tells Python that it should hoke it out of its cave of treasures for upcoming use...

```python
import math
```

This brings in a whole set of tools for dealing with basic mathematics. (so execute it!)

We reach into this with a dot...

```python
math.pi
```

    3.141592653589793

It has functions and constants...

```python
math.sin(math.pi / 2)
```

    1.0

So how do you find out what is available in `math`? Like before, Google `python3 math`... try it now:
https://www.google.ie/search?q=python3+math (this is just the Google search link)

For me, the closest version of Python was actually the second link (for Python 3.5) - normally, minor version differences are rarely a problem, but if what you see seems different to the manual, just add Google for "`python 3.4 math`", or whatever. To check the current version that Jupyter is using, go to `Help->About`.

Another useful one is `os`...

```python
import os
python_path = os.path.join('/usr', 'bin', 'python')
print(os.path.exists(python_path))
```

    True

This is in fact using a submodule, `os.path`, we can reach in twice to get functions inside that.

Also `sys`...

```python
import sys
print(sys.path)
# sys.exit(1)  # Exit with error code 1
```

    ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.5/dist-packages/IPython/extensions', '/home/jupyter/.ipython']

The last command tells your script to exit, with code 1. I don't really want to try that right now, as whatever happens won't be good.

```python
import math
print(math.e)
```

    2.718281828459045

This is the most succint as it just dumps everything in the `math` module into the global namespace, which saves much typing, but it can make debugging a headache, especially if there are some strange functions or variables in there that happen to have the same name as something else you later decide to use (this does happen from time to time)

```python
import math as m
print(m.e)
```

    2.718281828459045

...we've seen that...

```python
from math import e
print(e)
```

    2.718281828459045

...in other words, give `math` an alias...

```python
import math as m
import sympy as sy

print(m.e)
print(m.pi)
print(m.sin(3), sy.sin(3))
```

    2.718281828459045
    3.141592653589793
    0.1411200080598672 sin(3)

...which saves some typing if you're *sure* you don't want to use any other variable called `e`...

# Pythonic
## Walks like a Python...

This is a fundamental principle of Python and, unusually for something for fundamental, it isn't part of the code. Python is a bit like Ultimate Frisbee - in case you haven't had the pleasure, referees are not required, even in international competitions. This is on the basis that, if you're playing the sport, then you've decided you're there with the ethos, and the players can sort out those decisions amongst themselves and get on with the game. If you're trying to take advantage of that, then the fundamental question is, why play Ultimate Frisbee?

Good Python practice frequently diverges from what is commended in other languages. Succinctness or efficiency is not necessarily good - clarity comes before cleverness. However, using a pattern from previous experience when Python has a neater, more beautiful, more *Pythonic* way - that's something to try to avoid. The upshot of all this is that, when you come to what the community considers "good code", it is simple, elegant and clear.

## Not-so-Pythonic way:

## More Pythonic way:

```python
for task in things_to_do:
    print(task)
```

    Learn Python
    Finish PhD
    Publish research
    Accept Nobel prize
    Inspire a new generation
    Find a nice retirement village in the Galapagos Islands

Same output, but one feels more like pseudocode, more like how you would verbally convey the idea if somebody asked.

If you want to *get* what Python is about, try this (literally):

```python
import this
```

    The Zen of Python, by Tim Peters

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    Flat is better than nested.
    Sparse is better than dense.
    Readability counts.
    Special cases aren't special enough to break the rules.
    Although practicality beats purity.
    Errors should never pass silently.
    Unless explicitly silenced.
    In the face of ambiguity, refuse the temptation to guess.
    There should be one-- and preferably only one --obvious way to do it.
    Although that way may not be obvious at first unless you're Dutch.
    Now is better than never.
    Although never is often better than *right* now.
    If the implementation is hard to explain, it's a bad idea.
    If the implementation is easy to explain, it may be a good idea.
    Namespaces are one honking great idea -- let's do more of those!

This is one of the hardest things to get when switching to Python, particularly from compiled languages, but it is one of Python's greatest rewards. Nothing much beats opening up a script you hacked together two years ago to see zen-like transparency - no confusion, beautifully professional, and thanks to Python's design, no extra time spent writing it, just a feel for what Pythonic *is*.

# A couple of resources...

* If there is one link from this course to click: [What is Pythonic](http://blog.startifact.com/posts/older/what-is-pythonic.html)

* [PEP](https://www.python.org/dev/peps/pep-0008/) (Python Enhancement Proposal) 8 - code style

These are publicly accessible design documents and nearly everything new of importance passes through here. PEP8 is, in fact, a style guide; unlike many languages, there is more or less a definitive recommended style, and this spells it out. Since I discovered *flake8* and *pylint*, I have my IDE set to highlight style errors and it has reaped dividends to the moon and back - I strongly recommend you do so too, especially if we ever have to code together.

# Exceptions
## Are more than ordinary

In the words of one of early computing's most idiosyncratic legends, Rear Admiral Dr Grace Murray Hopper, "It is often easier to ask for forgiveness than to ask for permission". A distinguishing feature of Python is that it is built from the ground up with this maxim in mind, known as EAFP. In practice, this means preferring exceptions over tests, so using a `try-except` block (or, elsewhere, `try-catch`, etc.) instead of an `if` statement when checking whether you can perform an action.

Simple examples where you are more likely to use an exception in Python than another language:

* When making a directory

Can you see why this might be useful? If you use an `if` to check for non-existence, there is a potential race condition: by the time you reach the body, it could be made. If you catch the exception, you know it existed exactly when you tried to make it. (Although, the recommended routine, `os.makedirs`, has an optional don't-complain-if-dir-exists argument, which is probably even better)

* When testing file existence

Again with the race condition - this is in fact a security issue, as an attacker can create the file between you checking for existence and opening it for writing. If they create it, of course, then they set the permissions and can see the content regardless of your attempts to block reading.

* Checking type

```python

```

We talked a lot about duck-typing... the underlying principle is that you never reject input types *as long as they work*. Now imagine you have a routine with an argument `x`, where you want to do one thing if `x` is numeric type and another if it's, say, a string... if you use `if` and check their type, well what if this is some weird subclass of float that your routine has been sent, or the author of this type has carefully implemented all the necessary magic functions to make it quack like a float, but it's a completely unrelated class?

Push the boat out and see if it floats. Try casting to a float and if it doesn't work catch the exception. Then try casting to a string. Everything should cast to a string somehow. Now you have made checking numeric-ness a problem of `float`, which is infinitely more qualified to do this than you are.

* Checking for a dictionary key

Maybe what you think is a dictionary isn't - it's something that, when you request `thingy["something"]` will check whether `"something"` is something it might dynamically add, and, if so, will gladly return the result. If you check first (`if "something" in thingy:`), either the answer is misleading ("no"), or what looks like an innocent if-clause is modifying your dictionary. Moreover, even for an ordinary dictionary, having an if-clause followed by a retrieval hits your dictionary scan twice - try-catch only searches once.

Hopefully this motivates the idea of exceptions before tests - EAFP. Why then have you been repeatedly told not to do this in, as Python calls them, LBYL (Look Before You Leap) languages? The answer is usually that exceptions are horrendously slow and inefficient. In Python this isn't true, by design. They are so fundamental to the language that every loop in fact ends, not with a failing test, but when the iterable throws a specific exception (the `StopIteration` exception).

# Try a try

The actual syntax is similar to what you will have seen elsewhere:

```python
try:
    something_stupid()
except:
    print('Doh!')
```

    Doh!

Well, the first thing is that we didn't define *something_stupid*. So we get an exception and our extremely generic `except` provides no useful information.

## An improvement

```python
try:
    something_stupid()
except Exception as e:
    print('Doh! You forgot that', e)
```

    Doh! You forgot that name 'something_stupid' is not defined

Definitely better, and note that `print` will cast `e`, the exception, to a string - it is actually a more complex object:

```python
try:
    something_stupid()
except Exception as e:
    print(', '.join(dir(e)))
```

    __cause__, __class__, __context__, __delattr__, __dict__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __gt__, __hash__, __init__, __le__, __lt__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __setstate__, __sizeof__, __str__, __subclasshook__, __suppress_context__, __traceback__, args, with_traceback

Note in particular that `e` can have arguments when it is thrown - retrieved via `e.args`. Suppose we don't actually want to stop the exception bubbling up, just to some logging or tidy-up on the way through.

```python
try:
    something_stupid()
except Exception as e:
    print('I am *not* cleaning your mess for you, deal with it yourself!')
    raise e
```

Now we get the same exception we would have got the last time, but the `raise` keyword has kept it moving on past. This is quite useful, as Python exceptions tend to have lots of juicy info we wouldn't want to lose by ending our except block with a boring `print('oh noes')` and program exit.

In fact this is how we can throw a brand new exception - we take one of the standard types, e.g. "`RuntimeError`" and `raise` it with an explanatory argument:

```python
raise RuntimeError("Something quite generic went wrong")
```

# Better yet

This is a very broad error, and a parent class of most of the exceptions we have seen. It doesn't make it easy for calling functions to pin-point what went wrong...

Looking at it from the other end, just as we should be more specific about what we catch, we should be more specific about what we catch. This is a good idea, as, usually when we catch an exception, it is because we expect a particular thing to go wrong. As a case in point:

```python
try:
    os.makedirs(dirname)
except:
    # Great, someone has already created that directory
    # We can carry on!
    pass
# lalalalala...
print("just mucking about with my friend dirname")
```

First note `pass`. This is required because every block must have at least one non-comment line - if nothing else is there, we can use `pass` - it is Python's `no-op` if that helps.

Now ask yourself, "did we ever actually define dirname?" No? Then as soon as we use it after our supposed check, we will get an unhandled NameError exception.

We can specify what type of exceptions we catch... dir-already-exists have type `OSError`.

```python
try:
    os.makedirs(dirname)
except OSError:
    print("We had an error creating the directory")
# lalalalala...
print("just mucking about with my friend dirname")
```

Now we have stepped out of the way of genuine errors coming through. We can actually handle different exception types in different ways - there are plenty of cases this might be useful, if you want to do something awkward which could fail in five different directions.

```python
try:
    os.makedirs(dirname)
except OSError:
    # Great, someone has already created that directory
    # We can carry on!
    pass
except NameError:
    print("""
    This is the third code example
    where you haven't defined `dirname`.
    Seriously, catch yourself on.
    """)
# lalalalala...
print("just mucking about with my friend dirname")
```

Yeah. Great.

Useful tool just slotted in there: multi-line strings. If you start a string with three quotes, you can keep going on and on until you hit another three.

One final example to show how, for this particular case, you really can be a little more specific.

```python
dirname = "/etc/passwd"
try:
    os.makedirs(dirname)
except OSError:
    # Great, someone has already created that directory
    # We can carry on!
    pass
# lalalalala...
print("just mucking about with my friend", dirname)
```

Emm...we tried to make a directory with the same name as a key system file and are blithely assuming that every failure in doing so is simply because it is an existing directory. Not so good.

```python
import errno

dirname = "/etc/pass"

try:
    os.makedirs(dirname)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise
    # Great, someone has already created that directory
    # We can carry on!
# lalalalala...
print("just mucking about with my friend", dirname)
```

This emphasises the fact that subclasses of `Exception` often have additional contextually-relevant properties, which you should use. It also points out that `if` statements still have their place in error-handling!

When `raise` has no argument, it re-raises whichever exception it was that got us into this mess.

# A few important Exceptions
## That you might want to catch

* KeyError - `box_of_tricks["not here"]`
* TypeError - `1 + "banana"`
* IOError - `open('/etc/passwd', 'w')`

Exceptions are often more *pythonic* than pre-checking. [This Python2 doc](https://docs.python.org/2/howto/doanddont.html#exceptions) is a good start on that road. CRUCIAL READING!

## We are near the end of the walk-through

Feel free to go back and forward through this. I am going to give you one final task to get you started on this topic...

```python
print("I am on line number", x)
```

    I am on line number 1

Your task is to write a Jupyter cell to give you the $x^{th}$ digit of $pi$.

There are several ways to do this... I've given you enough tools above to find at least one.

```python
from math import pi
x = 3
print(str(pi)[x])
round(pi * 10 ** (x - 1)) % 10
```

    4

    4

So, hopefully, you now have a calculator for the $x^{th}$ digit of $\pi$. How can you use this without going back and editing the cell?

Well... copy and paste your text below the line in the cell below:

```python
alist = [1, 2, 36, 7]
another_list = [str(x) for x in alist]
another_list
```

    ['1', '2', '36', '7']

Now indent everything beneath the `def` line by four spaces and run it. You have just defined a function! If so, this should give you a `1`:

```python
get_xth_digit_of_pi(5)
```

If you had errors that don't make sense, try the chat window, try someone beside you (my normal approach in life) or ask me.

By the by, here are a couple of approaches - which is more Pythonic?
```
1.
    scaled_up = math.pi * (10 ** (x - 1))
    higher_digits = 10 * (scaled_up // 10)
    return int(scaled_up - higher_digits)

2.
    pi_as_str = str(math.pi)
    if x == 1:
        x = 0
    return pi_as_str[x]
```

# Classes

The basic syntax for classes is pretty succinct:

```python
class MyClass(object):
    some_attribute = "default_value"

    def __init__(self, some_attribute, arg2, arg3):
        # Some initialization (over-riding default)
        self.some_attribute = some_attribute

    def some_method(self):
        return self.some_attribute

    def another_method(self, arg1):
        if arg1 == "example":
            output = self.some_method()
            return output
```

A few points to note: `class` is just a block like any other. Default values can be supplied at its top level, but be careful - often it is better to set defaults in the initializer. Methods are defined as functions, but inside the class block. All of them have a first argument `self`, which is the object itself. Like, say, `this` in C++ (but without dereferencing required).

Python classes don't have a particularly strong approach to private/public access, but starting with an underscore is a conventional approach to indicate a member shouldn't be touched by anything outside the class. We subclass from `object` - additional classes may be added by comma-separation. Subclassing from `object` is actually the default in Python3 if no parentheses (or parent class) are provided - so-called *new-style classes*.

You also see your first magic method - the `__init__` function. This is the constructor, called when the class is created. Here we use it to override a default from the arguments it is passed.

```python
aninstance = MyClass("non-default value", 1, 2)
```

Here, you see the arguments to the `__init__` function.

```python
aninstance.another_method("example")
```

And, one method has called the next and returned the value we asked. We can check the type and dir of this like any other object.

```python
print(type(aninstance))
print(", ".join(dir(aninstance)))
```

Note that you can call a override a method in a parent class simply by defining a method with the same name. If you want to effectively wrap it, you can use the format

```python
class MyClass(myparentclass):
    ...
    def mymethod(self, arg1, arg2):
        val = myparentclass.mymethod(self, arg1, arg2)
        # Do something to val
        return val
```

Note that you must explicitly pass "`self`" in this case.

# Exercise

...if we have time...

Below, we subclass `str` and call it `MyString`. It has one method, called `join` to override the usual `join` functionality that strings have - otherwise, `MyString` strings behaves exactly as any other Python strings.

Exercise: make this overridden `join` method check if the iterable you are joining has "Pottery" in it and, if so, make it raise a RuntimeError.

Extension: combine this with your previous dictionary-to-strings function. Then update your dictionary-to-strings function to have a try-except block, which quietly swaps "Pottery" for "More Python" when the RuntimeError above error occurs (and then continues as normal).

```python
class MyString(str):
    def join(self, iterable):
        # Replace this with your Pottery-checker statement
        # ...
        # then continue as normal:
        return str.join(self, iterable)

delimiter = MyString(",")
delimiter.join(my_meetup_dot_com_profile["Interests"])
```

# What about all the other structures??

Well, they are not so hard once you get this far:

 but you'll come across them throughout today

* Loops can be `while [SOME BOOLEAN]:`, as well as for

* Generators put loops inside lists

I'll let that boggle your mind for a moment. Our future sessions are not going to be so closely led, partially because it *cannot* be as interesting experiment with my code as it is getting your hands dirty with your own. Now you've got plenty of tools at your disposal.

Now, I'm going to do a quick tour of scripting applications of Python, a few ideas you might want to ask me about, to give you an idea of what Python can do, so sit back relax and let me fiddle with projector technology at the front for a couple of minutes...

# PS: Tuples
## The Ice-List

Tuples are basically frozen lists...

```python
target_coordinates = (56, -5)
print(target_coordinates[0], "N", target_coordinates[1], "E")
```

...but it's fixed, so you can't do this...

```python
target_coordinates[1] = 38
```

If you alter it to set the whole variable (try it), that's fine though - you can alter what `target_coordinates` points to, you just can't alter the tuple itself.

Lists are constantly changing - length, content, type of content. Really, they aren't much like arrays in compiled languages like C or FORTRAN. They are far too flightly to be used as something like a dictionary index for example. However, supose I have a pair - exactly two elements, one a known `int` and one a known `string`, and it can't change. Well, if you stick two basic types together then why can't you do the same thing with it you can do with basic types? There's no weird changing behaviour going on, so if you freeze a list into something like that, can you use it as a dictionary index?

In short - yes you can!

```python
battleships = {}  # new dict
coordinates = [3, 2]
battleships[coordinates] = "HIT"
```

So it doesn't work with a list - try it now with a tuple - swap the square brackets on the *second* line for parens () and re-run

```python
print("At", coordinates, "we have a", battleships[coordinates])
```

**Aside**: A *hashable* type can be used as an index in a dict - tuples are, lists aren't (because they can change, aka are *mutable*)

# PS: an introduction to objects...

# Objects

## Totally class

If you haven't done object oriented coding before, I am afraid I am not going to give you the magic bullet now. It is an important concept to get to grips with, but for the moment, we will keep it fairly functional. Do take time to look through it later - Python is an excellent language for playing around with it and getting an intuition for what an object is. A number of you will be very familiar with this so please bear with me and the very rough introductory description I will give - for those who are not so familiar, this is worth getting the basics down early.

In real life, everything has properties, and many things have things they can do. In Python these properties are called attributes and the things an object can do are called methods. Together these are called members and, in Python, are accessed by putting a dot after the object and the attribute or method name. Everything else is just like any other variable or function, respectively.

For instance, my dog Freddie has properties - he has a colour, which is black (technically, he is imaginary, but in my head he is definitely black). In Python terms, this would be:

**Attribute**: `freddie.colour = black`

I can tell Freddie to roll-over. When I do so, I am, in some sense, calling Freddie's method:

**Method**: `freddie.roll_over()`

I have another dog, Nitwit. Nitwit can also roll over...

**Method**: `nitwit.roll_over()`

Both dogs can shake hands, but I need to tell them which paw...

**Method**: `freddie.shake_paw('left')`

At this point, you're probably wondering what exactly *is* the set of attributes and methods that my dogs have? This template, showing what attributes and methods a dog of mine can be expected to have, is called a class.

# A simple class

```python
class PhilsDog:
    name = ""
    colour = ""
    def shake_paw(self, side):
        print("My name is", self.name, "and I am shaking my", side, "paw like a good dog")
```

Here we create a class called `PhilsDog` that all of my dogs *implement* (that is, they are of that type). It indicates that they will have a colour and that they have a method called `shake_paw`. Think of this from the perspective of the dog - the first argument, `self`, is a little Python magic that refers to the dog itself. This lets the dog use it's name and colour (and any other methods) in the `shake_paw` method. The second argument is which paw I told my dog to shake. In response, any dog of mine says "Shaking left/right paw like a good dog". That's quite impressive, but I'd rather they just learned to zip it and actually shake their paws instead.

Now, that's just the template for one of my dogs, so how do I use it?

```python
freddie = PhilsDog()
freddie.name = "Freddie"
freddie.colour = "black"

nitwit = PhilsDog()
nitwit.name = "Nitwit"
nitwit.colour = "brown"

print("Freddie is", freddie.colour, "while Nitwit is", nitwit.colour)
```

I call the class like a function. This creates a new PhilsDog object - in computing terminology, I **instantiate** the class, creating a new **instance** of PhilsDog. Freddie is an instance of PhilsDog and so is Nitwit. But PhilsDog is just a template - as you can see, both Freddie and Nitwit have their own colour and name. I can update these just as any variable, and Freddie's doesn't affect Nitwit's, and I can read both back out again.

```python
freddie.shake_paw('left')
nitwit.shake_paw('right')
```

Good dogs. Here we see the `shake_paw` method being called for each dog and with a different `side` parameter. To remind you, the body of the method was:

```python
    def shake_paw(self, side):
        print("My name is", self.name, "and I am shaking my", side, "paw like a good dog")
```

You can see how the `self.name` matches the name I gave each dog on the previous slide.

If that was familiar to you, then that was probably a very unexciting few minutes - if not, it's probably come and gone very quickly. Thankfully, we only need to know that, if we have an object, we can get its attributes and methods by adding a dot and the attribute/method name.

Now why is that important...

# In Python
## Everything is an object (nearly)

```python
"just a normal string".upper()
```

```python
"Another string".islower()
```

# Two useful tools

```python
type(freddie)
```

"`type`" lets you examine what class an object is. Don't worry about that `__main__` just for the moment.

```python
dir(freddie)[-3:]
```

This is a list of all the members of `freddie`. If he hasn't had any added on the fly (which is possible in Python), this is the same as the members of the `PhilsDog` class. That syntax in the brackets is coming up in a couple of slides, but basically it means, the last three items. Python denotes somewhat magic functions with double-underscores on either side (they are all the previous items I'm hiding away) - there is good reason for these being here, but they aren't essential just now. Try removing the bit in brackets, including the brackets, to see what you get.


Probabilistic Programming
=====
and Bayesian Methods for Hackers
========

##### Version 0.1

`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`
___

Welcome to *Bayesian Methods for Hackers*. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code, but still not certain.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

### The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by preserving *uncertainty*. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: *Frequentist*, known as the more *classical* version of statistics, assume that probability is the long-run frequency of events (hence the bestowed title). For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of *belief*, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate *A* will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an *individual*, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is *your* belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability*.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being Heads. $P(A | X):\;\;$ You look at the coin, observe a Heads has landed, denote this information $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

### Bayesian Inference in Practice

 If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return *probabilities*.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a *YES*. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of *YES* and *NO*. The function might return:

>    *YES*, with probability 0.8; *NO*, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument:  *"Often my code has bugs"*. This parameter is the *prior*. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

#### Incorporating evidence

As we acquire more and more instances of evidence, our prior belief is *washed out* by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an *infinite* amount of evidence, say as $N \rightarrow \infty$, our Bayesian results (often) align with frequentist results. Hence for large $N$, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more *unstable*: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we *preserve the uncertainty* that reflects the instability of statistical inference of a small $N$ dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[1], before making such a decision:

> Sample sizes are never large. If $N$ is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$ is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$ is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

### Are frequentist methods incorrect then?

**No.**

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

#### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?")

The much more difficult analytic problems involve *medium data* and, especially troublesome, *really small data*. Using a similar argument as  Gelman's above, if big data problems are *big enough* to be readily solved, then we should be more interested in the *not-quite-big enough* datasets.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to })
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```python
# Loading setting from file

import matplotlib
import json

s = json.load(open("../styles/bmh_matplotlibrc.json"))
matplotlib.rcParams.update(s)
```

```python
"""
The book uses a custom matplotlibrc file, which provides the unique styles for
matplotlib plots. If executing this book, and you wish to use the book's
styling, provided are two options:
    1. Overwrite your own matplotlibrc file with the rc-file provided in the
       book's styles/ dir. See http://matplotlib.org/users/customizing.html
    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to
       update the styles in only this notebook. Try running the following code:

        import json
        s = json.load(open("../styles/bmh_matplotlibrc.json"))
        matplotlib.rcParams.update(s)

"""

# The code below can be passed over, as it is currently not important, plus it
# uses advanced topics we have not covered yet. LOOK AT PICTURE, MICHAEL!
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials)/2, 2, k+1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head?). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

##### Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```python
figsize(12.5, 4)
p = np.linspace(0, 1, 50)
plt.plot(p, 2*p/(1+p), color="#348ABD", lw=3)
#plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2*(0.2)/1.2, s=140, c="#348ABD")
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title("Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```python
figsize(12.5, 4)
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior = [1./3, 2./3]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])

plt.bar([0+0.25, .7+0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label="posterior distribution",
        lw="3", edgecolor=colours[1])

plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title("Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

## Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability distribution function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

### Discrete Case
If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```python
figsize(12.5, 4)

import scipy.stats as stats
a = np.arange(16)
poi = stats.poisson
lambda_ = [1.5, 4.25]
colours = ["#348ABD", "#A60628"]

plt.bar(a, poi.pmf(a, lambda_[0]), color=colours[0],
        label="$\lambda = %.1f$" % lambda_[0], alpha=0.60,
        edgecolor=colours[0], lw="3")

plt.bar(a, poi.pmf(a, lambda_[1]), color=colours[1],
        label="$\lambda = %.1f$" % lambda_[1], alpha=0.60,
        edgecolor=colours[1], lw="3")

plt.xticks(a + 0.4, a)
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values");
```

### Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```python
a = np.linspace(0, 4, 100)
expo = stats.expon
lambda_ = [0.5, 1]

for l, c in zip(lambda_, colours):
    plt.plot(a, expo.pdf(a, scale=1./l), lw=3,
             color=c, label="$\lambda = %.1f$" % l)
    plt.fill_between(a, expo.pdf(a, scale=1./l), color=c, alpha=.33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0,1.2)
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

### But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```python
figsize(12.5, 3.5)
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
plt.bar(np.arange(n_count_data), count_data, color="#348ABD")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to PyMC3, a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

Introducing our first hammer: PyMC3
-----

PyMC3 is a Python library for programming Bayesian analysis [3]. It is a fast, well-maintained library. The only unfortunate part is that its documentation is lacking in certain areas, especially those that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why PyMC3 is so cool.

We will model the problem above using PyMC3. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components. Model components are first-class primitives within the PyMC3 framework.

B. Cronin [5] has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

PyMC3 code is easy to read. The only novel thing should be the syntax. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables.

```python
import pymc3 as pm
import theano.tensor as tt

with pm.Model() as model:
    alpha = 1.0/count_data.mean()  # Recall count_data is the
                                   # variable that holds our txt counts
    lambda_1 = pm.Exponential("lambda_1", alpha)
    lambda_2 = pm.Exponential("lambda_2", alpha)

    tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data - 1)
```

In the code above, we create the PyMC3 variables corresponding to $\lambda_1$ and $\lambda_2$. We assign them to PyMC3's *stochastic variables*, so-called because they are treated by the back end as random number generators.

```python
with model:
    idx = np.arange(n_count_data) # Index
    lambda_ = pm.math.switch(tau > idx, lambda_1, lambda_2)
```

This code creates a new function `lambda_`, but really we can think of it as a random variable: the random variable $\lambda$ from above. The `switch()` function assigns `lambda_1` or `lambda_2` as the value of `lambda_`, depending on what side of `tau` we are on. The values of `lambda_` up until `tau` are `lambda_1` and the values afterwards are `lambda_2`.

Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet.

```python
with model:
    observation = pm.Poisson("obs", lambda_, observed=count_data)
```

The variable `observation` combines our data, `count_data`, with our proposed data-generation scheme, given by the variable `lambda_`, through the `observed` keyword.

The code below will be explained in Chapter 3, but I show it here so you can see where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Markov Chain Monte Carlo* (MCMC), which I also delay explaining until Chapter 3. This technique returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distributions look like. Below, we collect the samples (called *traces* in the MCMC literature) into histograms.

```python
### Mysterious code to be explained in Chapter 3.
with model:
    step = pm.Metropolis()
    trace = pm.sample(10000, tune=5000,step=step)
```

    Multiprocess sampling (4 chains in 4 jobs)
    CompoundStep
    >Metropolis: [tau]
    >Metropolis: [lambda_2]
    >Metropolis: [lambda_1]
    Sampling 4 chains, 0 divergences: 100%|██████████| 60000/60000 [00:09<00:00, 6045.13draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
lambda_1_samples = trace['lambda_1']
lambda_2_samples = trace['lambda_2']
tau_samples = trace['tau']
```

```python
figsize(12.5, 10)
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel("$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data)-20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel("probability");
```

### Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```python
figsize(12.5, 5)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N

plt.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data)
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```python
#type your code here.
```

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```python
#type your code here.
```

3\. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45.  That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC3 part. Just consider all instances where `tau_samples < 45`.)

```python
#type your code here.
```

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg).
-  [2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).
- [3] Salvatier, J, Wiecki TV, and Fonnesbeck C. (2016) Probabilistic programming in Python using PyMC3. *PeerJ Computer Science* 2:e55 <https://doi.org/10.7717/peerj-cs.55>
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```python
# from IPython.core.display import HTML
# def css_styling():
#     styles = open("../styles/custom.css", "r").read()
#     return HTML(styles)
# css_styling()
```

```python

```