# Detect and Mitigate Unfairness in Models

Machine learning models can incorporate unintentional bias, which can lead to issues with *fairness*. For example, a model that predicts the likelihood of diabetes might work well for some age groups, but not for others - subjecting a subset of patients to unnecessary tests, or depriving them of tests that would confirm a diabetes diagnosis.

In this notebook, you'll use the **Fairlearn** package to analyze a model and explore disparity in prediction performance for different subsets of patients based on age.

> **Note**: Integration with the Fairlearn package is in preview at this time. You may experience some unexpected errors.

## Important - Considerations for fairness

> This notebook is designed as a practical exercise to help you explore the Fairlearn package and its integration with Azure Machine Learning. However, there are a great number of considerations that an organization or data science team must discuss related to fairness before using the tools. Fairness is a complex *sociotechnical* challenge that goes beyond simply running a tool to analyze models.
>
> Microsoft Research has co-developed a [fairness checklist](https://www.microsoft.com/en-us/research/publication/co-designing-checklists-to-understand-organizational-challenges-and-opportunities-around-fairness-in-ai/) that provides a great starting point for the important discussions that need to take place before a single line of code is written.

## Install the required SDKs

To use the Fairlearn package with Azure Machine Learning, you need the Azure Machine Learning and Fairlearn Python packages, so run the following cell verify that the **azureml-contrib-fairness** package is installed.

```python
!pip show azureml-contrib-fairness
```

    Name: azureml-contrib-fairness
    Version: 1.34.0
    Summary: Uploads fairness dashboards to AzureML (preview).
    Home-page: https://docs.microsoft.com/python/api/overview/azure/ml/?view=azure-ml-py
    Author: Microsoft Corp
    Author-email: None
    License: Proprietary https://aka.ms/azureml-preview-sdk-license
    Location: /anaconda/envs/azureml_py36/lib/python3.6/site-packages
    Requires: azureml-core, jsonschema
    Required-by:

You'll also need the **fairlearn** package itself, and the **raiwidgets** package (which is used by Fairlearn to visualize dashboards). Run the following cell to install them.

```python
!pip install --upgrade fairlearn==0.7.0 raiwidgets
```

    Requirement already up-to-date: fairlearn==0.7.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (0.7.0)
    Collecting raiwidgets
      Downloading raiwidgets-0.13.0-py3-none-any.whl (1.9 MB)
         |████████████████████████████████| 1.9 MB 14.1 MB/s eta 0:00:01     |███████████▍                    | 686 kB 14.1 MB/s eta 0:00:01
    Requirement already satisfied, skipping upgrade: numpy>=1.17.2 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from fairlearn==0.7.0) (1.18.5)
    Requirement already satisfied, skipping upgrade: scipy>=1.4.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from fairlearn==0.7.0) (1.5.2)
    Requirement already satisfied, skipping upgrade: scikit-learn>=0.22.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from fairlearn==0.7.0) (0.22.2.post1)
    Requirement already satisfied, skipping upgrade: pandas>=0.25.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from fairlearn==0.7.0) (0.25.3)
    Requirement already satisfied, skipping upgrade: ipython==7.16.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from raiwidgets) (7.16.1)
    Requirement already satisfied, skipping upgrade: rai-core-flask==0.2.4 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from raiwidgets) (0.2.4)
    Collecting responsibleai==0.13.0
      Downloading responsibleai-0.13.0-py3-none-any.whl (65 kB)
         |████████████████████████████████| 65 kB 3.7 MB/s  eta 0:00:01
    Requirement already satisfied, skipping upgrade: lightgbm>=2.0.11 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from raiwidgets) (2.3.0)
    Requirement already satisfied, skipping upgrade: ipykernel<6.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from raiwidgets) (5.5.5)
    Collecting jinja2==2.11.3
      Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
         |████████████████████████████████| 125 kB 79.8 MB/s eta 0:00:01
    Collecting erroranalysis>=0.1.24
      Downloading erroranalysis-0.1.26-py3-none-any.whl (19 kB)
    Requirement already satisfied, skipping upgrade: joblib>=0.11 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from scikit-learn>=0.22.1->fairlearn==0.7.0) (0.14.1)
    Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from pandas>=0.25.1->fairlearn==0.7.0) (2021.1)
    Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from pandas>=0.25.1->fairlearn==0.7.0) (2.8.2)
    Requirement already satisfied, skipping upgrade: traitlets>=4.2 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (4.3.3)
    Requirement already satisfied, skipping upgrade: setuptools>=18.5 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (50.3.0)
    Requirement already satisfied, skipping upgrade: pickleshare in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (0.7.5)
    Requirement already satisfied, skipping upgrade: backcall in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (0.2.0)
    Requirement already satisfied, skipping upgrade: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (3.0.20)
    Requirement already satisfied, skipping upgrade: pexpect; sys_platform != "win32" in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (4.8.0)
    Requirement already satisfied, skipping upgrade: jedi>=0.10 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (0.17.2)
    Requirement already satisfied, skipping upgrade: pygments in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (2.10.0)
    Requirement already satisfied, skipping upgrade: decorator in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipython==7.16.1->raiwidgets) (5.0.9)
    Collecting Flask-Cors==3.0.9
      Downloading Flask_Cors-3.0.9-py2.py3-none-any.whl (14 kB)
    Requirement already satisfied, skipping upgrade: Flask~=1.0.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from rai-core-flask==0.2.4->raiwidgets) (1.0.3)
    Collecting greenlet==0.4.17
      Downloading greenlet-0.4.17-cp36-cp36m-manylinux1_x86_64.whl (44 kB)
         |████████████████████████████████| 44 kB 2.4 MB/s  eta 0:00:01
    Collecting gevent==20.9.0
      Downloading gevent-20.9.0-cp36-cp36m-manylinux2010_x86_64.whl (5.3 MB)
         |████████████████████████████████| 5.3 MB 75.0 MB/s eta 0:00:01     |███▊                            | 614 kB 75.0 MB/s eta 0:00:01
    Requirement already satisfied, skipping upgrade: networkx<=2.5 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from responsibleai==0.13.0->raiwidgets) (2.5)
    Requirement already satisfied, skipping upgrade: econml~=0.12.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from responsibleai==0.13.0->raiwidgets) (0.12.0)
    Collecting interpret-community>=0.20.0
      Downloading interpret_community-0.21.0-py3-none-any.whl (136 kB)
         |████████████████████████████████| 136 kB 79.8 MB/s eta 0:00:01
    Requirement already satisfied, skipping upgrade: semver~=2.13.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from responsibleai==0.13.0->raiwidgets) (2.13.0)
    Collecting dice-ml<0.8,>=0.7.2
      Downloading dice_ml-0.7.2-py3-none-any.whl (242 kB)
         |████████████████████████████████| 242 kB 74.5 MB/s eta 0:00:01
    Requirement already satisfied, skipping upgrade: tornado>=4.2 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipykernel<6.0->raiwidgets) (6.1)
    Requirement already satisfied, skipping upgrade: jupyter-client in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from ipykernel<6.0->raiwidgets) (6.1.12)
    Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from jinja2==2.11.3->raiwidgets) (2.0.1)
    Requirement already satisfied, skipping upgrade: six>=1.5 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from python-dateutil>=2.6.1->pandas>=0.25.1->fairlearn==0.7.0) (1.16.0)
    Requirement already satisfied, skipping upgrade: ipython-genutils in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from traitlets>=4.2->ipython==7.16.1->raiwidgets) (0.2.0)
    Requirement already satisfied, skipping upgrade: wcwidth in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython==7.16.1->raiwidgets) (0.2.5)
    Requirement already satisfied, skipping upgrade: ptyprocess>=0.5 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from pexpect; sys_platform != "win32"->ipython==7.16.1->raiwidgets) (0.7.0)
    Collecting parso<0.8.0,>=0.7.0
      Downloading parso-0.7.1-py2.py3-none-any.whl (109 kB)
         |████████████████████████████████| 109 kB 101.7 MB/s eta 0:00:01
    Requirement already satisfied, skipping upgrade: click>=5.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from Flask~=1.0.0->rai-core-flask==0.2.4->raiwidgets) (8.0.1)
    Requirement already satisfied, skipping upgrade: Werkzeug>=0.14 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from Flask~=1.0.0->rai-core-flask==0.2.4->raiwidgets) (1.0.1)
    Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from Flask~=1.0.0->rai-core-flask==0.2.4->raiwidgets) (2.0.1)
    Requirement already satisfied, skipping upgrade: zope.interface in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from gevent==20.9.0->rai-core-flask==0.2.4->raiwidgets) (5.4.0)
    Requirement already satisfied, skipping upgrade: zope.event in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from gevent==20.9.0->rai-core-flask==0.2.4->raiwidgets) (4.5.0)
    Requirement already satisfied, skipping upgrade: dowhy in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.6)
    Requirement already satisfied, skipping upgrade: sparse in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.13.0)
    Requirement already satisfied, skipping upgrade: numba!=0.42.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.53.1)
    Requirement already satisfied, skipping upgrade: shap<0.40.0,>=0.38.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.39.0)
    Requirement already satisfied, skipping upgrade: statsmodels>=0.10 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.10.2)
    Requirement already satisfied, skipping upgrade: interpret-core[required]<=0.2.6,>=0.1.20 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from interpret-community>=0.20.0->responsibleai==0.13.0->raiwidgets) (0.2.5)
    Requirement already satisfied, skipping upgrade: packaging in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from interpret-community>=0.20.0->responsibleai==0.13.0->raiwidgets) (21.0)
    Requirement already satisfied, skipping upgrade: h5py in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from dice-ml<0.8,>=0.7.2->responsibleai==0.13.0->raiwidgets) (3.1.0)
    Requirement already satisfied, skipping upgrade: jsonschema in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from dice-ml<0.8,>=0.7.2->responsibleai==0.13.0->raiwidgets) (3.2.0)
    Requirement already satisfied, skipping upgrade: tqdm in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from dice-ml<0.8,>=0.7.2->responsibleai==0.13.0->raiwidgets) (4.62.2)
    Requirement already satisfied, skipping upgrade: jupyter-core>=4.6.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from jupyter-client->ipykernel<6.0->raiwidgets) (4.7.1)
    Requirement already satisfied, skipping upgrade: pyzmq>=13 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from jupyter-client->ipykernel<6.0->raiwidgets) (22.2.1)
    Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < "3.8" in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from click>=5.1->Flask~=1.0.0->rai-core-flask==0.2.4->raiwidgets) (4.8.1)
    Requirement already satisfied, skipping upgrade: sympy>=1.4 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from dowhy->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (1.8)
    Requirement already satisfied, skipping upgrade: pydot>=1.4 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from dowhy->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (1.4.2)
    Requirement already satisfied, skipping upgrade: llvmlite<0.37,>=0.36.0rc1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from numba!=0.42.1->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.36.0)
    Requirement already satisfied, skipping upgrade: cloudpickle in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from shap<0.40.0,>=0.38.1->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (1.6.0)
    Requirement already satisfied, skipping upgrade: slicer==0.0.7 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from shap<0.40.0,>=0.38.1->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.0.7)
    Requirement already satisfied, skipping upgrade: patsy>=0.4.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from statsmodels>=0.10->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (0.5.1)
    Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from packaging->interpret-community>=0.20.0->responsibleai==0.13.0->raiwidgets) (2.4.7)
    Requirement already satisfied, skipping upgrade: cached-property; python_version < "3.8" in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from h5py->dice-ml<0.8,>=0.7.2->responsibleai==0.13.0->raiwidgets) (1.5.2)
    Requirement already satisfied, skipping upgrade: pyrsistent>=0.14.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from jsonschema->dice-ml<0.8,>=0.7.2->responsibleai==0.13.0->raiwidgets) (0.18.0)
    Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from jsonschema->dice-ml<0.8,>=0.7.2->responsibleai==0.13.0->raiwidgets) (21.2.0)
    Requirement already satisfied, skipping upgrade: zipp>=0.5 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from importlib-metadata; python_version < "3.8"->click>=5.1->Flask~=1.0.0->rai-core-flask==0.2.4->raiwidgets) (3.5.0)
    Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.4; python_version < "3.8" in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from importlib-metadata; python_version < "3.8"->click>=5.1->Flask~=1.0.0->rai-core-flask==0.2.4->raiwidgets) (3.10.0.2)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from sympy>=1.4->dowhy->econml~=0.12.0->responsibleai==0.13.0->raiwidgets) (1.2.1)
    ERROR: sqlalchemy 1.4.26 has requirement greenlet!=0.4.17; python_version >= "3" and (platform_machine == "aarch64" or (platform_machine == "ppc64le" or (platform_machine == "x86_64" or (platform_machine == "amd64" or (platform_machine == "AMD64" or (platform_machine == "win32" or platform_machine == "WIN32")))))), but you'll have greenlet 0.4.17 which is incompatible.
    ERROR: azureml-train-automl-runtime 1.34.0 has requirement jinja2<=2.11.2, but you'll have jinja2 2.11.3 which is incompatible.
    ERROR: azureml-responsibleai 1.34.0 has requirement responsibleai==0.9.4, but you'll have responsibleai 0.13.0 which is incompatible.
    ERROR: azureml-interpret 1.34.0 has requirement interpret-community==0.19.*, but you'll have interpret-community 0.21.0 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement antlr4-python3-runtime~=4.7.2, but you'll have antlr4-python3-runtime 4.8 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement azure-graphrbac~=0.60.0, but you'll have azure-graphrbac 0.61.1 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement azure-mgmt-containerregistry==3.0.0rc17, but you'll have azure-mgmt-containerregistry 8.1.0 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement azure-mgmt-keyvault==9.0.0, but you'll have azure-mgmt-keyvault 9.1.0 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement azure-mgmt-resource==18.0.0, but you'll have azure-mgmt-resource 13.0.0 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement azure-mgmt-storage~=18.0.0, but you'll have azure-mgmt-storage 11.2.0 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement packaging~=20.9, but you'll have packaging 21.0 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement pytz==2019.1, but you'll have pytz 2021.1 which is incompatible.
    ERROR: azure-cli 2.24.0 has requirement websocket-client~=0.56.0, but you'll have websocket-client 1.2.1 which is incompatible.
    Installing collected packages: interpret-community, erroranalysis, dice-ml, responsibleai, jinja2, raiwidgets, Flask-Cors, greenlet, gevent, parso
      Attempting uninstall: interpret-community
        Found existing installation: interpret-community 0.19.3
        Uninstalling interpret-community-0.19.3:
          Successfully uninstalled interpret-community-0.19.3
      Attempting uninstall: erroranalysis
        Found existing installation: erroranalysis 0.1.17
        Uninstalling erroranalysis-0.1.17:
          Successfully uninstalled erroranalysis-0.1.17
      Attempting uninstall: dice-ml
        Found existing installation: dice-ml 0.6.1
        Uninstalling dice-ml-0.6.1:
          Successfully uninstalled dice-ml-0.6.1
      Attempting uninstall: responsibleai
        Found existing installation: responsibleai 0.10.0

## Train a model

You'll start by training a classification model to predict the likelihood of diabetes. In addition to splitting the data into training and test sets of features and labels, you'll extract *sensitive* features that are used to define subpopulations of the data for which you want to compare fairness. In this case, you'll use the **Age** column to define two categories of patient: those over 50 years old, and those 50 or younger.

```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

# load the diabetes dataset
print("Loading Data...")
data = pd.read_csv('data/diabetes.csv')

# Separate features and labels
features = ['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']
X, y = data[features].values, data['Diabetic'].values

# Get sensitive features
S = data[['Age']].astype(int)
# Change value to represent age groups
S['Age'] = np.where(S.Age > 50, 'Over 50', '50 or younger')

# Split data into training set and test set
X_train, X_test, y_train, y_test, S_train, S_test = train_test_split(X, y, S, test_size=0.20, random_state=0, stratify=y)

# Train a classification model
print("Training model...")
diabetes_model = DecisionTreeClassifier().fit(X_train, y_train)

print("Model trained.")
```

Now that you've trained a model, you can use the Fairlearn package to compare its behavior for different sensitive feature values. In this case, you'll:

- Use the fairlearn **selection_rate** function to return the selection rate (percentage of positive predictions) for the overall population.
- Use **scikit-learn** metric functions to calculate overall accuracy, recall, and precision metrics.
- Use a **MetricFrame** to calculate selection rate, accuracy, recall, and precision for each age group in the **Age** sensitive feature. Note that a mix of **fairlearn** and **scikit-learn** metric functions are used to calculate the performance values.

```python
from fairlearn.metrics import selection_rate, MetricFrame
from sklearn.metrics import accuracy_score, recall_score, precision_score

# Get predictions for the witheld test data
y_hat = diabetes_model.predict(X_test)

# Get overall metrics
print("Overall Metrics:")
# Get selection rate from fairlearn
overall_selection_rate = selection_rate(y_test, y_hat) # Get selection rate from fairlearn
print("\tSelection Rate:", overall_selection_rate)
# Get standard metrics from scikit-learn
overall_accuracy = accuracy_score(y_test, y_hat)
print("\tAccuracy:", overall_accuracy)
overall_recall = recall_score(y_test, y_hat)
print("\tRecall:", overall_recall)
overall_precision = precision_score(y_test, y_hat)
print("\tPrecision:", overall_precision)

# Get metrics by sensitive group from fairlearn
print('\nMetrics by Group:')
metrics = {'selection_rate': selection_rate,
           'accuracy': accuracy_score,
           'recall': recall_score,
           'precision': precision_score}

group_metrics = MetricFrame(metrics=metrics,
                             y_true=y_test,
                             y_pred=y_hat,
                             sensitive_features=S_test['Age'])

print(group_metrics.by_group)
```

From these metrics, you should be able to discern that a larger proportion of the older patients are predicted to be diabetic. *Accuracy* should be more or less equal for the two groups, but a closer inspection of *precision* and *recall* indicates some disparity in how well the model predicts for each age group.

In this scenario, consider *recall*. This metric indicates the proportion of positive cases that were correctly identified by the model. In other words, of all the patients who are actually diabetic, how many did the model find? The model does a better job of this for patients in the older age group than for younger patients.

It's often easier to compare metrics visually. To do this, you'll use the Fairlearn fairness dashboard:

1. Run the cell below to generate a dashboard from the model you created previously.
2. When the widget is displayed, use the **Get started** link to start configuring your visualization.
3. Select the sensitive features you want to compare (in this case, there's only one: **Age**).
4. Select the model performance metric you want to compare (in this case, it's a binary classification model so the options are *Accuracy*, *Balanced accuracy*, *Precision*, and *Recall*). Start with **Recall**.
5. Select the type of fairness comparison you want to view. Start with **Demographic parity difference**.
6. View the dashboard visualization, which shows:
    - **Disparity in performance** - how the selected performance metric compares for the subpopulations, including *underprediction* (false negatives) and *overprediction* (false positives).
    - **Disparity in predictions** - A comparison of the number of positive cases per subpopulation.
7. Edit the configuration to compare the predictions based on different performance and fairness metrics.

```python
from raiwidgets import FairnessDashboard

# View this model in Fairlearn's fairness dashboard, and see the disparities which appear:
FairnessDashboard(sensitive_features=S_test,
                   y_true=y_test,
                   y_pred={"diabetes_model": diabetes_model.predict(X_test)})
```

The results show a much higher selection rate for patients over 50 than for younger patients. However, in reality, age is a genuine factor in diabetes, so you would expect more positive cases among older patients.

If we base model performance on *accuracy* (in other words, the percentage of predictions the model gets right), then it seems to work more or less equally for both subpopulations. However, based on the *precision* and *recall* metrics, the model tends to perform better for patients who are over 50 years old.

Let's see what happens if we exclude the **Age** feature when training the model.

```python
# Separate features and labels
ageless = features.copy()
ageless.remove('Age')
X2, y2 = data[ageless].values, data['Diabetic'].values

# Split data into training set and test set
X_train2, X_test2, y_train2, y_test2, S_train2, S_test2 = train_test_split(X2, y2, S, test_size=0.20, random_state=0, stratify=y2)

# Train a classification model
print("Training model...")
ageless_model = DecisionTreeClassifier().fit(X_train2, y_train2)
print("Model trained.")

# View this model in Fairlearn's fairness dashboard, and see the disparities which appear:
FairnessDashboard(sensitive_features=S_test2,
                   y_true=y_test2,
                   y_pred={"ageless_diabetes_model": ageless_model.predict(X_test2)})
```

Explore the model in the dashboard.

When you review *recall*, note that the disparity has reduced, but the overall recall has also reduced because the model now significantly underpredicts positive cases for older patients. Even though **Age** was not a feature used in training, the model still exhibits some disparity in how well it predicts for older and younger patients.

In this scenario, simply removing the **Age** feature slightly reduces the disparity in *recall*, but increases the disparity in *precision* and *accuracy*. This underlines one the key difficulties in applying fairness to machine learning models - you must be clear about what *fairness* means in a particular context, and optimize for that.

## Register the model and upload the dashboard data to your workspace

You've trained the model and reviewed the dashboard locally in this notebook; but it might be useful to register the model in your Azure Machine Learning workspace and create an experiment to record the dashboard data so you can track and share your fairness analysis.

Let's start by registering the original model (which included **Age** as a feature).

> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure.

```python
from azureml.core import Workspace, Experiment, Model
import joblib
import os

# Load the Azure ML workspace from the saved config file
ws = Workspace.from_config()
print('Ready to work with', ws.name)

# Save the trained model
model_file = 'diabetes_model.pkl'
joblib.dump(value=diabetes_model, filename=model_file)

# Register the model
print('Registering model...')
registered_model = Model.register(model_path=model_file,
                                  model_name='diabetes_classifier',
                                  workspace=ws)
model_id= registered_model.id

print('Model registered.', model_id)
```

Now you can use the FairLearn package to create binary classification group metric sets for one or more models, and use an Azure Machine Learning experiment to upload the metrics.

> **Note**: This may take a while, and may result in some warning messages (which you can ignore). When the experiment has completed, the dashboard data will be downloaded and displayed to verify that it was uploaded successfully.

```python
from fairlearn.metrics._group_metric_set import _create_group_metric_set
from azureml.contrib.fairness import upload_dashboard_dictionary, download_dashboard_by_upload_id

#  Create a dictionary of model(s) you want to assess for fairness
sf = { 'Age': S_test.Age}
ys_pred = { model_id:diabetes_model.predict(X_test) }
dash_dict = _create_group_metric_set(y_true=y_test,
                                    predictions=ys_pred,
                                    sensitive_features=sf,
                                    prediction_type='binary_classification')

exp = Experiment(ws, 'mslearn-diabetes-fairness')
print(exp)

run = exp.start_logging()

# Upload the dashboard to Azure Machine Learning
try:
    dashboard_title = "Fairness insights of Diabetes Classifier"
    upload_id = upload_dashboard_dictionary(run,
                                            dash_dict,
                                            dashboard_name=dashboard_title)
    print("\nUploaded to id: {0}\n".format(upload_id))

    # To test the dashboard, you can download it
    downloaded_dict = download_dashboard_by_upload_id(run, upload_id)
    print(downloaded_dict)
finally:
    run.complete()
```

The preceding code downloaded the metrics generated in the experiement just to confirm it completed successfully. The real benefit of uploading the metrics to an experiement is that you can now view the FairLearn dashboard in Azure Machine Learning studio.

Run the cell below to see the experiment details, and click the **View Run details** link in the widget to see the run in Azure Machine Learning studio. Then view the **Fairness** tab of the experiment run to view the dashboard for the fairness ID assigned to the metrics you uploaded, which behaves the same way as the widget you viewed previously in this notebook.

```python
from azureml.widgets import RunDetails

RunDetails(run).show()
```

You can also find the fairness dashboard by selecting a model in the **Models** page of Azure Machine Learning studio and reviewing its **Fairness** tab. This enables your organization to maintain a log of fairness analysis for the models you train and register.

## Mitigate unfairness in the model

Now that you've analyzed the model for fairness, you can use any of the *mitigation* techniques supported by the FairLearn package to find a model that balances predictive performance and fairness.

In this exercise, you'll use the **GridSearch** feature, which trains multiple models in an attempt to minimize the disparity of predictive performance for the sensitive features in the dataset (in this case, the age groups). You'll optimize the models by applying the **EqualizedOdds** parity constraint, which tries to ensure that models that exhibit similar true and false positive rates for each sensitive feature grouping.

> *This may take some time to run*

```python
from fairlearn.reductions import GridSearch, EqualizedOdds
import joblib
import os

print('Finding mitigated models...')

# Train multiple models
sweep = GridSearch(DecisionTreeClassifier(),
                   constraints=EqualizedOdds(),
                   grid_size=20)

sweep.fit(X_train, y_train, sensitive_features=S_train.Age)
models = sweep.predictors_

# Save the models and get predictions from them (plus the original unmitigated one for comparison)
model_dir = 'mitigated_models'
os.makedirs(model_dir, exist_ok=True)
model_name = 'diabetes_unmitigated'
print(model_name)
joblib.dump(value=diabetes_model, filename=os.path.join(model_dir, '{0}.pkl'.format(model_name)))
predictions = {model_name: diabetes_model.predict(X_test)}
i = 0
for model in models:
    i += 1
    model_name = 'diabetes_mitigated_{0}'.format(i)
    print(model_name)
    joblib.dump(value=model, filename=os.path.join(model_dir, '{0}.pkl'.format(model_name)))
    predictions[model_name] = model.predict(X_test)

```

Now you can use the FairLearn dashboard to compare the mitigated models:

Run the following cell and then use the wizard to visualize **Age** by **Recall**.

```python
FairnessDashboard(sensitive_features=S_test,
                   y_true=y_test,
                   y_pred=predictions)
```

The models are shown on a scatter plot. You can compare the models by measuring the disparity in predictions (in other words, the selection rate) or the disparity in the selected performance metric (in this case, *recall*). In this scenario, we expect disparity in selection rates (because we know that age *is* a factor in diabetes, with more positive cases in the older age group). What we're interested in is the disparity in predictive performance, so select the option to measure **Disparity in recall**.

The chart shows clusters of models with the overall *recall* metric on the X axis, and the disparity in recall on the Y axis. Therefore, the ideal model (with high recall and low disparity) would be at the bottom right corner of the plot. You can choose the right balance of predictive performance and fairness for your particular needs, and select an appropriate model to see its details.

An important point to reinforce is that applying fairness mitigation to a model is a trade-off between overall predictive performance and disparity across sensitive feature groups - generally you must sacrifice some overall predictive performance to ensure that the model predicts fairly for all segments of the population.

> **Note**: Viewing the *precision* metric may result in a warning that precision is being set to 0.0 due to no predicted samples - you can ignore this.

## Upload the mitigation dashboard metrics to Azure Machine Learning

As before, you might want to keep track of your mitigation experimentation. To do this, you can:

1. Register the models found by the GridSearch process.
2. Compute the performance and disparity metrics for the models.
3. Upload the metrics in an Azure Machine Learning experiment.

```python
# Register the models
registered_model_predictions = dict()
for model_name, prediction_data in predictions.items():
    model_file = os.path.join(model_dir, model_name + ".pkl")
    registered_model = Model.register(model_path=model_file,
                                      model_name=model_name,
                                      workspace=ws)
    registered_model_predictions[registered_model.id] = prediction_data

#  Create a group metric set for binary classification based on the Age feature for all of the models
sf = { 'Age': S_test.Age}
dash_dict = _create_group_metric_set(y_true=y_test,
                                     predictions=registered_model_predictions,
                                     sensitive_features=sf,
                                     prediction_type='binary_classification')

exp = Experiment(ws, "mslearn-diabetes-fairness")
print(exp)

run = exp.start_logging()
RunDetails(run).show()

# Upload the dashboard to Azure Machine Learning
try:
    dashboard_title = "Fairness Comparison of Diabetes Models"
    upload_id = upload_dashboard_dictionary(run,
                                            dash_dict,
                                            dashboard_name=dashboard_title)
    print("\nUploaded to id: {0}\n".format(upload_id))
finally:
    run.complete()
```

> **Note**: A warning that precision is being set to 0.0 due to no predicted samples may be displayed - you can ignore this.

When the experiment has finished running, click the **View Run details** link in the widget to view the run in Azure Machine Learning studio (you may need to scroll past the initial output to see the widget), and view the FairLearn dashboard on the **fairness** tab.

.ipynb_checkpoints/15 - Detect Unfairness-checkpoint.ipynb


# Click "Edit App" to see the code

Python is a very powerful object oriented programming language, which is used in most fields of science both for computing and for data analysis.
This document is a very simplistic introduction to Python written by a non-Python expert, and is designed to be a quick introduction to Python for CHEM2000 students.
In many places the nomenclature is likely to be inaccurate and you are encouraged to consult more rigorous resources developed by real Python programmers.

This document is aimed at providing enough knowledge to solve all the numerical laboratories that are part of the CHEM2000 unit using Python, but it is far from comprehensive. There are many different ways to solve problems numerically, different libraries/packages can be used and here we are providing only one (or a few) possible way of doing things.

The entire document and the examples therein have been developed using a **Jupyter Notebook**, and work best there.

A Jupyter notebook consists of a series of cells that can be either _text_ or _code_, the cell can then be _executed_ by pressing the **run** button or _Shift+Return_ (if you want to look like a pro). If things go pear-shaped, restarting the kernel and rerunning all the cells may help; this can be done by pressing the _fast forward_ button.

Text cells accept _Markdown_ syntax and once executed will render the text. Markdown provides a simple way of producing formatted text and it accept _LaTeX_ commands for equations, which look much nicer that those made with the equation editor in MS Word. For a Markdown tutorial click [here](https://www.markdowntutorial.com)

The code cells contain Python 3 commands. Note that some Python 3 is different from Python 2, _e.g._ the **print** command works differently.

# My first Jupyter Notebook
In general you would use the first _code_ cell to import all the packages that we need to use in the remainder of the code. We can import entire packages, part of the packages and assign aliases to the package name, _e.g._

```python
# python packages
import pandas as pd # Dataframes and reading CSV files
import numpy as np # Numerical libraries
import matplotlib.pyplot as plt # Plotting library
from lmfit import Model # Least squares fitting library
```

this is particularly important because it ensures that if we define a variable that has the same name of a function belonging to that package there is no confusion about what the code does.
```python
# This is a variable
mean = 10

# This is the NumPy function to compute the mean of an array of numbers
np.mean([1,2,3])
```

Things to keep in mind when programming in Pythons are

1. It is Case sensitivE

```python
average = 0
```
is different from
```python
aVeRagE = 0
```

2. Variables' names cannot have spaces. This line would give you a _syntax error_
```ptyhon
number of values = 10
```

3. Spaces between operators are ignored, _1+2_ is the same as _1 + 2_

```python
print(1 + 2)
print(1+2)
```

4. \# is a comment and everything on its right is ignored

```python
# This is a comment
print("Hello world") # This is also a comment
```

5. A single command can spread over multiple lines. If we split the line between variables no continuation line character is required. Otherwise we can use the "\" character. Note the different results of the two commands below.

```python
print("Hello",
      "world")
print("Hello \
       world")
```

6. Indentation matters! Indentation is used to define the content of loops, functions... This will be more clear when we start using functions.

# Good programming practices

1. Use meaningful name for your variables so that you know what's inside, there is no character limit.
2. Use a consistent style and convention for your variables, the code will look neater. I like to use the _camel case_ style (numberOfValues) that allows me to have separation between words without using spaces. Alternatively you can use the _Pascal case_ style (NumberOfValues) or use the underscore character to separate words (number_of_variables) or create your own style.
3. Comment your code well, it may be obvious what it does when you write it, but it won't be so obvious after a year or more.

# Python as a simple calculator

Let's start by doing some simple mathematical operations using this Jupyter Notebook; addition, multiplication, division $\dots$, just to get familiar with the Jupyter Notebook.

```python
2 + 3
```

```python
4 * 3
```

```python
12 / 3
```

```python
2**3
```

The same operations can be done using variables. We can first define two variables _a_ and _b_ and then use them in in the following cells

```python
a = 12
b = 3
a + b
```

This is not a very efficient way of working because the result of the operation is not available to the rest of the code after the cell has been executed. Typically, we want to create new variables and than write the result using the _print_ command.

```python
a = 10
b = 20
c = a + b
print("Result :",c)
```

# Python as a scientific calculator
Not all scientific operators are natively available in Python, but they can be accessed through optional packages that are loaded at the beginning of your notebook, _e.g._ **NumPy**.
NumPy is one of the most commonly used Python libraries, it contains all the operators for square root, logarithm, exponential, the trigonometric function, a suite of constants and much more.
For more information see [https://numpy.org](https://numpy.org).

In the examples below we show how to use NumPy to access some of these functions.
* Note that we access the NumPy function through using the _np_ prefix because of the way we imported the NumPy library at the beginning of this notebook.

```python
print("The approximate value of pi is                     :",np.pi)
print("The approximate value of the Euler constant (e) is :",np.e)
print("The square root of two is                          :",np.sqrt(2))
print("The natural logarithm of two is                    :",np.log(2))
print("The logarithm base 10 of two is                    :",np.log10(2))
print("The sine of pi is                                  :",np.sin(np.pi))
print("The cosine of pi is                                :",np.cos(np.pi))
```

Even more useful than variables are arrays, which allow us to store many values in one place. Arrays can be created by hand or be the output of other Python functions.
They can contain numbers, strings or other variables, or mixed types

```python
arrayOfNumbers = [300, 2, 3.2]
print("Three dimensional array of numbers :",arrayOfNumbers)
arrayOfStrings = ["Temperature" , "Pressure" , "Volume"]
print("Three dimensional array of strings :",arrayOfStrings)
mixedArray = ["temperature" , 300]
print("The mixed array is                 :",mixedArray)
```

We can easily access one of the elements of the array, by specifying its location in the array.
* Note that Python starts counting from zero!

```python
print("Second element of the array of numbers :",arrayOfNumbers[1])
print("Second element of the array of strings :",arrayOfStrings[1])
```

We can also construct a loop to cycle over the elements of the array.
If we are interested in cycling over one array only we can use the **in** operator.
* It's important to note here that the indentation of the code determines where the loop finished

```python
for value in arrayOfNumbers:
    print("--- This is inside the loop ----",value)
print("--- This is outside the loop ---")
```

Alternatively we can create a loop over the indices of the array using the **range** iterator.
* Note the use of the function **len** to compute the size of the array!
* Note that the upper limit of the **range** iterator is not included!

```python
numberOfElements = len(arrayOfNumbers)
print("Number of elements :",numberOfElements)
for index in range(0,numberOfElements):
    print(index,                   # index
          arrayOfStrings[index],   # sting
          arrayOfNumbers[index])   # number
```

**range** is a special thing of python 3, and it produces a list of indices only when part of a loop. This is at variance with the NumPy **arange** function, which instead will produce an array that we can use normally.
Both the **range** and **arange** function typically take three arguments, the lower limit of the range (included) the upper limit of the range (not included) and the step. The main difference is that **range** being an iterator works only with integer numbers, while **arange** can also work with floating point numbers.
If the step is omitted, one is assumed.
Let's have a look at a couple of examples.

```python
for i in range(0,4):
    print(i)
```

```python
for i in range(0,4,2):
    print(i)
```

```python
for i in np.arange(0,4,2):
    print(i)
```

```python
for i in np.arange(0,4,0.5):
    print(i)
```

Let's now see how we can use the **range** iterator and the **np.arange** function to generate arrays of equally spaced values. While this is straightforward with the NumPy function, it is more complicated with **range** iterator, and we need to _recast_ the output into a list.

```python
values = np.arange(1.1,4,0.7)
print(values)

values = list(range(0,5,2))
print(values)
```

# Operations with arrays
As an illustrative example of using arrays and variables we can now compute the sum and average of all the elements in an array of numbers. In order to do that we initialise a variable to zero, and progressively add the array elemnts to it.
* Note how we can increment the value of the variable using the += operator. These two commands are equivalent
```python
summ += value
summ = summ + value
```
Those two line are also equivalent to the following, where we use a temporary variable, to explicitly show what the code does
```python
tmp = summ + value
summ = tmp
```
The operators -=, \*= and /= have analogous meanings.

```python
summ = 0.
for value in arrayOfNumbers:
    summ += value

print("Result of += :",summ)
```

The average can then be computed directly inside the print statement.

```python
print("Average :",summ/len(arrayOfNumbers))
```

Many simple operations on arrays can however be more efficiently performed using libraries such as NumPy, _e.g._ summation, average, standard deviation, etc.
Using these function will also make your code slimmer and easier to read.

```python
tally = np.sum(arrayOfNumbers)
average = np.mean(arrayOfNumbers)
StDev = np.std(arrayOfNumbers)

print("Sum :",tally)
print("Average :",average)
print("Standard Deviation :",StDev)
```

Unfortunately there is no NumPy function for computing the standard error, but we can easily compute that from its definition

\begin{equation}
StdErr = \frac{\sigma}{\sqrt{N}}
\end{equation}
where $\sigma$ is the standard deviation and $N$ the number of values used in the calculation.

```python
print("Standard Error :",StDev/np.sqrt(len(arrayOfNumbers)))
```

# Using DataFrames
Dataframes are powerful objects that are part of the _pandas_ package. The most simplistic description of a dataframe is that it is a multi-dimension mixed array. Dataframes are more than that, as they also include functions that operate on the DataFrame content.
This definition of DataFrames would probably horrify a Python programmer, but it would suffice for the purpose of this course.

Dataframes can be defined by hand, or created by other functions, _e.g._ by reading a Comma Separated Values file (.csv). Let's see first how we can create a an empty DataFrame, with three columns named "Temperature", "Volume" and "Pressure"; with their units.

```python
# This array is used to define the names of the columns
header = ["Temperature (K)" , "Volume (L)" , "Pressure (bar)"]

# This is our new dataframe
df = pd.DataFrame(data=None, columns=header)
print(df)
```

Let's now fill the dataframe using the ideal gas law

\begin{equation}
pV = nRT
\end{equation}

where $p$ is the pressure, $V$ the volume, $T$ the temperature and $R=8.314\ J/mol/K$ is the ideal gas constant, each expressed with the units specified in the header of the dataframe.

Let's compute the volume of an ideal gas at different pressures and temperatures.

For simplicity we fix the number of moles to 1.
* Note how we use the **range** function to create an array of integers and the NumPy **arange** function to create a an array of _floating point numbers_.
* The variable _index_ is used to count the number of elements that we already have in the DataFrame, and to add the next one. This works because Python starts counting from zero.
* Note we used the **loc** function to added an array to the DataFrame at a specific position.
* Note how we also created an array to store all the temperatures we generate, the array is created empty using **= []** and then we append elements to it using the **.append()** function

```python
R = 8.314 # kJ/mol
n = 1
# Conversion factor between J/bar to litre
conversionFactor = 0.01

listOfTemperatures = []
for T in range(100,301,50):
    listOfTemperatures.append(T)

    for p in np.arange(0.1,1,0.02):
        V = (n * R * T / p) * conversionFactor
        index = len(df.index)
        df.loc[index]  = [T , V , p] # a vector is added to the DataFrame

print(df)
```

There are many ways to access the data in a DataFrame. Here we'll show you two; one to quickly get an entire column of the array, and one to get selected chunks of data using the **iloc** function.

```python
print(df["Temperature (K)"])
```

For the DataFrame that we have, the **iloc** function takes two arguments, the row and columns indices.

```python
print(df.iloc[0,1])
```

We can then use "**:**" to specify a range of elements that we want to use.
* Note that the lower limit of the range is included while the upper limit is not !
* Note that if one limit of the range is missing, the start/end of the array is assumed
* Note how we have used **.values** to cast the output data in an array.

```python
print(df.iloc[0:3 , 1  ].values) # the first three volumes
print(df.iloc[0   , 0:3].values) # the first row
print(df.iloc[0   ,  : ].values) # the first row
print(df.iloc[0   , 1: ].values) # the last two elements of the first row
print(df.iloc[0   ,  :2].values) # the first two elements of the first row
```

# Making a graph
Let's now make a graph with these data using the **mathplotlib** library.
As a start we can make a plot of the entire DataFrame.

The **subplots** function creates two objects, the _figure_ and the _axes_ of the figure itself.
Each of those objects contain functions that we can use to customise the final plot. More in another tutorial.

```python
# Create the figure and axes objects
fig , ax = plt.subplots()
# Add the data to the plot from the DataFrame
ax.scatter(df["Pressure (bar)"] , df["Volume (L)"])
# Diplay the figure
plt.show()
```

Let's now do some data manipulation to make a better plot, using a line for each isotherm.
There are many ways of doing this, but here we'll take an educational approach and use _conditional_ statements to select parts of the dataframe, add them to arrays and plot them.

What we are going to do is to choose a temperature, _e.g._ 100K, and create two arrays (p,v) with the corresponding pressures and volumes, than we will use them for plotting.
* Note that we used the **scatter** and **plot** functions to plot the data as circles with a line overlaid.
* Note that we set the **set** function for to add the labels to the axes (_ax_)

```python
T=100
pressure = []
volume = []
for index in range(0,len(df.index)):
    if df.iloc[index,0] == T:
        pressure.append(df.iloc[index,2])
        volume.append(df.iloc[index,1])

print("Pressure array :",pressure[0:3],"...")
print("Volume array :",volume[0:3],"...")

fig , ax = plt.subplots()
ax.scatter(pressure , volume)
ax.plot(pressure , volume)

# Let's add the labels to the axes
ax.set(xlabel="Pressure (bar)")
ax.set(ylabel="Volume (L)")

plt.show()
```

If we want then to plot all isotherm in one graph we can wrap the code above in a loop over all the temperatures we have created.

* Note the indentation of the **for** loops and **if** conditional statement.
* Note how in this example we select portions of the DataFrame including a conditional statement in **[]** when we call the DataFrame
* Note that we used **.values** to transform the DataFrame into an array

```python
# First we have to create the figure
fig , ax = plt.subplots()

for T in listOfTemperatures:
    pressure = df[df["Temperature (K)"] == T]["Pressure (bar)"].values
    volume = df[df["Temperature (K)"] == T]["Volume (L)"].values

    # add one line to the plot for each temperature
    ax.plot(pressure , volume, label=T)

# Let's add the labels to the axes
ax.set(xlabel="Pressure (bar)")
ax.set(ylabel="Volume (L)")

# Let's also add the legend
ax.legend()

plt.show()
```


```python
import matplotlib.pyplot as plt
import plotly as py
import plotly.graph_objs as go
import numpy as np
import math
import ipywidgets as widgets
from IPython.display import display, Math, Latex, HTML, IFrame
from astropy.table import Table, Column
from ipywidgets import interact, interactive

py.offline.init_notebook_mode(connected=True)
%matplotlib inline

font = {'family' : 'sans-serif',
        'weight' : 'normal',
        'size'   : 14}

plt.rc('font', **font)

'''Above, we are importing all the necessary modules in order to run the notebook.
Numpy allows us to define arrays of values for our variables to plot them
matplotlib is what we use to create the figures
the display and widgets are to make the notebook look neat
'''

HTML('''
<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>''')

```

***

  **Light and Optics**

***

Gif taken from https://giphy.com/gifs/fandor-sun-eclipse-3o7OsM9vKFH2ESl0KA/links, August 1st, 2018.
 Figure 1: For hundreds of years, scientists have tried to understand the nature of light. With advances in technology, and inventions like telescopes, we have been able to see farther than ever before.

***

## Introduction

Throughout most of history, humans did not understand light as we do today. As science and technology have progressed over time, so too has our knowledge of the nature of light.

In this lesson, when we say the word "light", we will be referring to visible light (light that comes from the sun, lightbulbs etc.). We will go over how a few key experiments kickstarted a new way of thinking, and a few of the ways that we are able to manipulate light. We will also talk about how our eyes enable us to see.

## Background

If you had to describe to someone what light is, you may have a hard time. Some people think of light as the absence of darkness, but even that doesn't say much about light itself.

Our understanding of light truly began around the 17th century, when a few individuals started to realize that light was not a mystical substance. Scientists (or "natural philosophers", as they were called during that time) recognized that certain properties of light were measurable, and that some properties could be manipulated. Sir Isaac Newton and Ole Rømer were among the first scientists to take a step in this direction.

> ### Isaac Newton's Prism Experiment

Sir Isaac Newton has made contributions to many fields of science and mathematics. In 1666, while spending time at his childhood home in Lincolnshire, England, Newton began experimenting with light.

Using a small slit in his window shutters, Newton passed a narrow beam of sunlight through a glass prism. The light travelled through the prism, and projected a rainbow of color on the other side!

Picture taken from http://lightingmatters.com.au/wp/what-is-the-colour-of-white/white-light-prism-experiment/, July 30th, 2018.
 Figure 2: This picture shows how a prism can create a spectrum of color. This is what Newton would have seen in 1666.

Later on, scientists determined that the prism was actually splitting light into its component parts. This phenomenon is called **dispersion**.

Through this experiment, Newton demonstrated that white light was actually made up of all the individual colors of the rainbow!

> ### Ole Rømer and the Speed of Light

For many years, people thought that if somebody lit a match, the light from that match would be instantly visible to everyone, no matter how far away they were. However, in 1676 Ole Rømer proved that this is not the case.

Rømer spent a long time studying the orbit of Io, one of Jupiter's moons. As part of his study, he began predicting the times when Io should be hidden behind Jupiter's shadow (these periods are called eclipses). However, Rømer saw that his predictions for when these eclipses should occur were not always accurate.

Gif taken from https://giphy.com/gifs/timelapse-DXIa1beDspYRy, August 1st, 2018.
 Figure 3: Here we can see Jupiter as it looks through a telescope. You might be able to see a black spot move from the left to the right across Jupiter's surface. This is actually one of Jupiter's many moons!

Rømer then realized that these errors may be because the distance between Io and the Earth was always changing. Rømer thought that when the distance between Io and the Earth increased, it might take a longer time for light coming from Io to reach Earth. If this were the case, then the light must be travelling at a finite speed!

After taking many measurements and using some clever mathematics, Rømer calculated the speed of light to be roughly 220,000,000 m/s, or 792,000,000 km/h.

Today, we have measured the speed of light to be 299,792,458 m/s. Although he was not exactly right, Rømer provided one of the first mathematical calculations for the speed of light.

***
Since the time of Rømer and Newton, scientists have made many new discoveries about the nature of light. While not all of these discoveries agree with one another, here are two things we know for sure:
- Light is made up of a spectrum of color
- Light travels at a speed of 299,792,458 m/s

Now let's talk about some of the ways we can manipulate light.
***

## Reflection

We are all familiar with reflection; chances are, you look at your reflection more than once a day. But have you ever stopped to wonder what is really going on?

Reflection is the term used to describe how light can change direction when it comes into contact with certain surfaces.

When incoming light rays encounter a reflective surface, they bounce off the surface and continue moving in a new direction. The new direction in which it moves is determined by the **law of reflection**.

\begin{equation}
\rm Law\: of\: Reflection: Angle\: of\: Incidence = Angle\: of\: Reflection
\end{equation}

On the animation below, click on the flashlight to turn it on, and move your mouse to change the angle of incidence.

```python
IFrame('Animations/reflect.html',width=500,height=320)
```

As seen above, the **normal** is what we call the line that forms a 90$^{\circ}$ angle with  the surface. The **angle of incidence** is what we call the angle between the flash lights beam and the normal. Similarly, the **angle of reflection** is the angle that the newly reflected light beam makes with the normal. The law of reflection states that these two angles will always be equal.

## Refraction

Have you ever tried to reach down and grab an object sitting at the bottom of a pool of water? If you have, you may have noticed that the object isn't actually in the location that you thought it was.

 Image taken from http://legacy.sciencelearn.org.nz/Contexts/Light-and-Sight/Sci-Media/Video/Refraction/(quality)/hi on August 3rd, 2018.
 Figure 4: When you are looking into a body of water from above, the objects you see beneath the surface are not actually where they appear to be.

This phenomenon occurs because the light travelling to your eyes from the bottom of the pool **refracts**, or changes its direction of travel, when it transitions from water to air.

The **index of refraction** is a value that we use to show how much light will bend when travelling through a substance. For example, the index of refraction for air is approximately 1.00, and the index of refraction for water is about 1.33. Because these indexes are different, light will bend when passing from water to air, or vice versa.

Use the animation below to see how light refracts when passing from air to water. Click on the flashlight to turn it on.

```python
IFrame('Animations/refract.html',width=520,height=320)
```

Mathematically, reflection can be described using the following equation, known as Snell's Law:

\begin{equation}
\textrm{Snells Law:}\: n_1\sin(\theta_1) = n_2\sin(\theta_2)
\end{equation}

where $n_1$ is the index of refraction for the first medium, $\theta_1$ is the incident angle, $n_2$ is the index of refraction for the second medium, and $\theta_2$ is the angle of refraction.

Light will bend *towards* the normal when travelling from a medium with a *lower* index of refraction to one with a *higher* index of refraction, and vice versa.

***
Some of the most beautiful sights in nature are caused by reflection and refraction. Here are a couple of examples:

### Rainbows

Rainbows are a result of both reflection and refraction. As its raining, each water droplet acts like a tiny prism, just like the one we saw in Figure 2. The water droplets split visible light into colors, and these colors are then reflected back towards our eyes.

 Image taken from https://waterstories.nestle-waters.com/environment/how-does-a-rainbow-form/ on August 3rd, 2018.
 Figure 5: Water droplets use reflection and refraction to create the beautiful rainbows that we see while it is raining.

### Mirages

Have you ever been driving on a sunny day, and up ahead it looks as though a stream of water is running across the road? You are really seeing a mirage.
Mirages also occur because of refraction, but they do not result in a display of color like a rainbow. This type of refraction occurs due to a difference in temperature between separate layers of air.

As we were describing before, refraction occurs when light travels from one substance to another. Well, it turns out that hot air and cold air are actually different enough to act as different substances. Therefore, light will refract when passing through one to the other.

 Image taken from https://edexcellence.net/articles/what-the-mirage-gets-wrong-on-teacher-development on August 3rd, 2018.
 Figure 6: Although it may look like water running across the road, it is actually a mirage. These commonly occur in desert areas, where the road can become very hot.

When you are looking at a mirage, it can look as though the air is wavy and fluid, which is why it is common to think that you are looking at water. This appearance occurs when layers of hot and cold air are mixing together, and light passing through these layers is constantly being refracted in different directions.

You may see a mirage appear on top of a hot roadway, behind the exhaust pipe of a plane or car, or around any other source of heat.

## Applications of Reflection and Refraction

### Lenses

If you have glasses, or contact lenses, then you are constantly using refraction in order to help you see! Lenses use refraction to point light in specific directions.

Generally speaking, there are two types of lenses: **convex** and **concave**.

To see how each type of lense affects light, use the following animation.

```python
IFrame('Animations/convex.html',width=520,height=420)
```

As seen above, a convex lens focuses light towards a specific point, while a concave lens will spread light away from a point. These lenses can be combined in many ways in order to produce different effects. For example, a camera lens uses a series of both convex and concave lenses in order to direct incoming light towards the back of the camera.

 Image taken from https://www.reddit.com/r/pic/comments/3o3b7w/camera_lens_cut_in_half/ on August 3rd, 2018.
 Figure 5: This is what the inside of a camera lens looks like. The photographer can adjust how they want the picture to look by changing the distance between the individual lenses.

## Vision

Our eyes are very complex organs, but the process that enables us to see is actually pretty simple. The basic steps are as follows:

1. Light enters the eye through the **pupil**
2. The convex **lens** behind the pupil directs incoming light towards the **retina**, which is like a screen at the back of our eye.
3. The retina then sends this image to the brain.
4. The brain then interprets the image.

 Image taken from https://openclipart.org/detail/261647/eye-diagram on August 3rd, 2018.
 Figure 6: This diagram shows some of the key components of the eye that enable us to see.

However, the image that is projected onto the retina is actually upside down!

 Image taken from https://www.eetimes.com/author.asp?section_id=14&doc_id=1282795 on August 3rd, 2018.
 Figure 7: The convex lens at the front of our eye actually flips images upside down.

So the retina actually sends an upside down image to the brain, and the brain automatically flips the image rightside up.

Use the following link to see an animation showing how a convex lens flips images upside down:https://phet.colorado.edu/sims/geometric-optics/geometric-optics_en.html.

## Technology & Inventions

### The Telescope

The first telescope was made by Hans Lippershey in 1608, but it was Galileo Galilee who became famous by using it for astronomy. There are many different types of telescopes, but they all use reflection and refraction to make far away objects appear closer.

A telescope uses a large opening to collect incoming light, and then directs this light towards your eye by using mirrors and lenses.

 Image taken from https://www.skyandtelescope.com/press-releases/tips-for-first-time-telescope-buyers/ on August 3rd, 2018.
 Figure 8: Telescopes come in many different shapes and sizes.

The reason why things look bigger when looking through a telescope is because of the lenses.

### The Microscope

- talk about invention of microscope
- why it works

## Conclusion

Our understanding of light is the result of hundreds of years of research and innovation. Along the way, we have created incredible new technologies that have allowed us to look further than ever before.

 Image taken from https://xenlife.com.au/hubble-space-telescope-important/ on August 3rd, 2018.
 Figure 9: The Hubble Space Telescope has shown us pictures of galaxies that are billions of light years away.

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


```python
%matplotlib widget

import matplotlib
import numpy as np
import matplotlib.pyplot as plt
import mplcursors
```

```python
# Scale plot to fit page
plt.rcParams["figure.figsize"] = (8, 6)
plt.rcParams["font.size"] = 18
```

# Radiation Interatctions with Matter

### Learning Objectives

- Define uncollided flux
- Define linear interaction coefficient
- Apply linear interaction coefficients to a slab problem
- Identify the units of intensity, flux density, fluence, reaction rate
- Compare linear interaction coefficient and cross section
- Calculate uncollided flux in a medium
- Calculate mean free path of a particle in a medium
- Define the half thickness in a medium
- Apply the concept of buildup factor to attenuation in a slab
- Define microscopic cross section
- Calculate macroscopic cross sections, given a microscopic cross section
- Calculate the mass interaction coefficients of mixtures
- Calculate flux density
- Calculate Reaction Rate Density
- Recognize the dependence of flux on energy, position, and time
- Define radiation fluence
- Calculate uncollided flux density from isotropic point sources
- Apply the Kelin-Nishina formula to Compton Scattering
- Compare energy dependence of photon interaction cross sections
- Describe energy dependence of neutron interaction cross sections
- Recognize the comparative range of heavy vs. light particles
- Recognize the comparative range of charged particles

## Linear Interaction Coefficient

- The interaction of radiation with matter is always statistical in nature, and, therefore, must be described in probabilistic terms.

Consider a particle travelling through a homogeneous material.

\begin{align}
P_i(\Delta x) &= \mbox{probability the particle, causes a reaction of type i in distance }\Delta x\\
\end{align}

Empirically, we find that this probability becomes constant as $\Delta x \longrightarrow 0$. Thus:

\begin{align}
\mu_i &= \lim_{\Delta x \rightarrow 0}\frac{P_i(\Delta x)}{\Delta x}\\
\end{align}

Facts about $\mu_𝑖$:

- $\mu_i$ is an *intrinsic* property of the material for a given incident particle and interaction.
- $\mu_i$ is independent of the path length traveled prior to the interaction.
- $\mu_i$ may represent many types of interaction (scattering: $\mu_s$, absorption: $\mu_a$, ...)
- $\mu_i$ typically depends on particle energy

The probability, per unit path length, that a neutral particle undergoes some sort of reaction, is the sum of the probabilities, per unit path length of travel, for each type :

\begin{align}
\mu_t(E) = \sum_i \mu_i(E)
\end{align}

## Think Pair Share:

What are the units of the linear interaction coefficient?

### Attenuation of Uncollided Flux

Imagine a plane of neutral particles strike a slab of some material, normal to the surface.

We can describe this using $\mu_t$ or, equivalently, the macroscopic total cross section $\Sigma_t$.

\begin{align}
I(x) &= I_0e^{-\mu_t x}\\
I(x) &= I_0e^{-\Sigma_t x}\\
\end{align}

where

\begin{align}
        I(x) &= \mbox{uncollided intensity at distance x}\\
        I_0 &= \mbox{initial uncollided intensity}\\
        \mu_t &= \mbox{total linear interaction coefficient} \\
        \Sigma_t &= \mbox{macroscopic total cross section} \\
        x &= \mbox{distance into material [m]}\\
\end{align}

```python
import math
def attenuation(distance, initial=100, sig_t=1):
    """This function describes neutron attenuation into the slab"""
    return initial*math.exp(-sig_t*distance)

```

Rather than intensity, one can find the probability density:

We have a strong analogy between decay and attenuation, as above. In the case of decay the probability of decay in a time interval dt is:

\begin{align}
P(t)dt &= \lambda e^{-\lambda t}dt\\
 &= \mbox{probability of decay in interval dt}
\end{align}

From this, one can find the mean lifetime of a neutron before decay:

\begin{align}
\bar{t} &= \int_0^\infty t'P(t')dt'\\
        &= \int_0^\infty t'\lambda e^{-\lambda t'}dt'\\
        &= \frac{1}{\lambda}
\end{align}

In the case of attenuation:
\begin{align}
P(x)dx &= \Sigma_te^{-\Sigma_tx}dx
\end{align}

Such that:

\begin{align}
P(x)dx &= \mu_t e^{-\mu_t x}dx\\
 &= \Sigma_t e^{-\Sigma_t x}dx\\
 &= \mbox{probability of interaction in interval dx}
\end{align}

So, the mean free path is:

\begin{align}
\bar{l} &= \int_0^\infty x'P(x')dx'\\
        &= \int_0^\infty x'\Sigma_te^{-\Sigma_t x'}dx'\\
        &= \frac{1}{\Sigma_t}
\end{align}

Or, equivalently in $\mu_t$ notation:

\begin{align}
\bar{x} &= \int_0^\infty x'P(x')dx'\\
        &= \int_0^\infty x'\mu_te^{-\mu_t x'}dx'\\
        &= \frac{1}{\mu_t}
\end{align}

```python
def prob_dens(distance, initial=100, sig_t=1):
    return sig_t*attenuation(distance, initial=100, sig_t=1)

```

```python
sig_t = 0.2
i_0 = 100

# This code plots attenuation
import numpy as np
z = np.arange(24)
y = np.arange(24)
x = np.arange(24)
for h in range(0,24):
    x[h] = h
    y[h] = attenuation(h, initial=i_0, sig_t=sig_t)
    z[h] = prob_dens(h, initial=i_0, sig_t=sig_t)

# creates a figure and axes with matplotlib
fig, ax = plt.subplots()
scatter = plt.scatter(x, y, color='blue', s=y*20, alpha=0.4)
ax.plot(x, y, color='red')
ax.plot(x, z, color='green')

# adds labels to the plot
ax.set_ylabel('Percent of Neutrons')
ax.set_xlabel('Distance into slab')
ax.set_title('Attenuation')

# Add mpl widget for interactivity
labels = ['{0}% intensity'.format(i) for i in y]
mplcursors.cursor(ax).connect(
    "add", lambda sel: sel.annotation.set_text(labels[sel.index]))

plt.show()
```

## Half-thickness

In another analog to decay, the **half-thickness** of a material is the distance required for half of the incident radiation to interact with a medium:

\begin{align}
\frac{I(x_{1/2})}{I(0)} &= e^{-\mu_t x_{1/2}}\\
\implies x_{1/2} &= \frac{\ln{2}}{\mu_t}
\end{align}

## Think pair share:
What is the concept in the context of decay that is analogous to the half-thickness?

## Microscopic and Macroscopic Cross Sections

- The microscopic cross section $\sigma_i$ is  the likelihood of the event per unit area.
- The macroscopic cross section $\Sigma_i$ is  the likelihood of the event per unit area of a certain density of target isotopes.
- The macroscopic cross section $\Sigma_i$ is equivalent to the linear interaction coefficient $\mu_i$, but we tend to use $\Sigma_i$ in nuclear interactions, reserving $\mu_i$ for photon interactions.

\begin{align}
\mu_i &= \mbox{linear interaction coefficient}\\
\Sigma_i &= \mbox{macroscopic cross section}\\\\
 &= \sigma_i N\\
 &= \sigma_i \frac{\rho N_a}{A}\\
 \mbox{where }& \\
 N &= \mbox{atom density of medium}\\
 \rho &= \mbox{mass density of the medium}\\
 N_a &= \mbox{Avogadro's number}\\
 A &= \mbox{atomic weight of the medium}
\end{align}

```python
def macroscopic_xs(micro, N):
    """Returns the macroscopic cross section [cm^2] or [barns]

    Parameters
    ----------
    micro: double
        microscopic cross section [cm^2] or [barns]
    N: double
        atom density in the medium [atoms/cm^3]
    """
    return micro*N
```

```python
def NA():
    """Returns Avogadro's number
    6.022x10^23 atoms per mole
    """
    return 6.022E23

def num_dens_from_rho(rho, na, a):
    """The atomic number density.
    That is, the concentration of atoms or molecules per unit volume (V)

    Parameters
    -----------
    rho : double
        material density (in units like g/cm^3 or kg/m^3) of the sample
    na : double
        Avogadro's number
    a : double
        The atomic or molecular weight of the atom or molecule of interest
    """
    return rho*na/a
```

## Example:
Imagine a beam of neutrons striking a body of water, $H_2O$. Many will be absorbed by the hydrogen in the water, particularly $^1H$.

Find the macroscopic absorption cross section

```python
# Find the macroscpic absorption cross section
# of the 1H in H2O
sig_1h = 0.333 # barns

# First, molecular density of water
rho_h2o = 1 # g/cm^3
a_h2o = 18.0153 # g/mol
n_h2o = num_dens_from_rho(rho_h2o, NA(), a_h2o) # molecules water / cm^3
n_h2o_barn = n_h2o/10**(24) # 10^24 molecules water / cm^3
print('n_h2o [1/cm^3] = ', n_h2o)
print('n_h2o [10^(24)/cm^3] = ', n_h2o_barn)

# Now, there are two Hydrogens in each molecule of water, so:
macroscopic_h1 = macroscopic_xs(sig_1h, 2*n_h2o_barn)
print('absorption in water from 1H = ', macroscopic_h1)
```

    n_h2o [1/cm^3] =  3.342714248444378e+22
    n_h2o [10^(24)/cm^3] =  0.033427142484443784
    absorption in water from 1H =  0.02226247689463956

### Mixtures
In a medium that is a mixture of isotopes (e.g. $H_2O$), we can calculate the total macroscopic cross section based on individual microscopic cross sections and number densities for each component of the mixture. We may need to include information about relative isotopic abundances (f).

For the same problem as above (neutrons striking a body of water) we can calculate the absorption by *all* isotopes in the $H_2O$.

\begin{align}
\mu^{H_2O} \equiv \Sigma^{H_2O} &= N^1\sigma_a^1 + N^2\sigma_a^2 + N^{16}\sigma_a^{16}
+ N^{17}\sigma_a^{17}  + N^{18}\sigma_a^{18}\\
&= f^1N^H\sigma_a^1 + f^2N^H\sigma_a^2 + f^{16}N^O\sigma_a^{16} + f^{17}N^O\sigma_a^{17} + f^{18}N^O\sigma_a^{18}
\end{align}

Superscripts 1, 2, 16, 17, and 18 indicate isotopes $^1H$, $^2H$, $^{16}O$,$^{17}O$, and $^{18}O$.

\begin{align}
N^H = 2N^{H_2O}\\
N^{O} = N^{H_2O}\\
N^{H_2O} = \frac{\rho^{H_2O}N_a}{A^{H_2O}}
\end{align}

Thus:
\begin{align}
\mu^{H_2O} \equiv \Sigma^{H_2O} &= N^{H_2O}\left[2f^1\sigma_a^1 + 2f^2\sigma_a^2 + f^{16}\sigma_a^{16} + f^{17}\sigma_a^{17} + f^{18}\sigma_a^{18}\right]
\end{align}

```python
# We need a lot of data

# Abundances
f_1 = 0.99985
f_2 = 0.00015
f_16 = 0.99756
f_17 = 0.00039
f_18 = 0.00205

# Then, microscopic absorption cross sections
sig_1 = 0.333
sig_2 = 0.000506
sig_16 = 0.000190
sig_17 = 0.239
sig_18 = 0.000160

macroscopic_h2o = n_h2o_barn*(2*f_1*sig_1
                              + 2*f_2*sig_2
                              + f_16*sig_16
                              + f_17*sig_17
                              + f_18*sig_18)
print('absorption in water from all isos = ', macroscopic_h2o,"\n",
     'while absorption in water from 1H = ', macroscopic_h1,"\n",
     'Thus, absorption in water is mostly from 1H.')
```

    absorption in water from all isos =  0.02226860496564809
     while absorption in water from 1H =  0.02226247689463956
     Thus, absorption in water is mostly from 1H.

### Reaction Rates

- The microscopic cross section is just the likelihood of the event per unit area.
- The macroscopic cross section is just the likelihood of the event per unit area of a certain density of target isotopes.
- The reaction rate is the macroscopic cross section times the flux of incident neutrons.

\begin{align}
R_{i,j}(\vec{r}) &= N_j(\vec{r})\int dE \phi(\vec{r},E)\sigma_{i,j}(E)\\
R_{i,j}(\vec{r}) &= \mbox{reactions of type i involving isotope j } [reactions/cm^3s]\\
N_j(\vec{r}) &= \mbox{number of nuclei participating in the reactions } [\#/cm^3]\\
E &= \mbox{energy} [MeV]\\
\phi(\vec{r},E)&= \mbox{flux of neutrons with energy E at position i } [\#/cm^2s]\\
\sigma_{i,j}(E)&= \mbox{cross section } [cm^2]\\
\end{align}

This can be written more simply as $R_x = \Sigma_x \phi$.

Using flux notation, the density of ith type of neutron interaction with isotope j, per unit time is:

\begin{align}
R_{i,j}(\vec{r}) = \Sigma_{i,j}\phi(\vec{r})
\end{align}

## Flux density from Point Source
Finding $\phi(\vec{r}0$ generally requires *particle transport calculations.*

However, in some simple practical situations, the flux density can be approximated by the flux density of uncollided source particles.

### Point Source in Vacuum

Consider a source of particles:

- it emits $S_p$ particles per unit time
- all particles have energy E
- and they are emitted radially outward into an infinite vacuum
- isotropically (equally in all directions)
- from a single point in space

### Think-pair share:

- How many interactions occur?

### At a radius r:
Because the source is isotropic, each unit area on an imaginary spherical shell of radius $r$ has the same number of particles crossing it. Thus:

\begin{align}
\phi^o(r) &= \mbox{uncollided flux at radius r in any direction}\\
&= \frac{S_p}{4\pi r^2}
\end{align}

```python
def phi_o_r(r, s):
    """Returns the uncolided flux at radius r
    due to an isotropic point source in a vacuum

    Parameters
    -----------
    r : double
        radius away from the point [length]
    s : double
        point source strength [particles/time]
    """
    return s/(4*math.pi*pow(r,2))
```

```python
s=200
plt.clear()
plt.plot(range(1,10), [phi_o_r(r, s) for r in range(1,10)])
plt.show
```

The plot above, this $1/r^2$ reduction in flux and reaction rate, is called **geometric attenuation**

### Think-pair-share

What other phrase do you think we use for attenuation in a medium?

## Point Source in an Attenuating Medium
So, the unollided flux is
\begin{align}
\phi^o(r) &= \frac{S_p}{4\pi r^2}
\end{align}

### A small volume

At a distance r, we place a homogeneous mass with a volume $\Delta V_d$. The interaction rate $R_d$ in the mass is:

\begin{align}
&R^o(r)=\mu_d(E)\Delta V_d\frac{S_p}{4\pi r^2}\\
\mbox{where}&\\
&\mu_d(E)=\mbox{linear interaction coefficient in the volume}
\end{align}

### An inifinite volume

From this, we can imagine the point source embedded in an infinite medium of this material. A detector is at distance r in the volume:

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{-\mu r}\\
\mbox{where}&\\
&e^{-\mu r}=\mbox{material attenuation} \\
&\frac{S_p}{4\pi r^2}=\mbox{geometric attenuation}
\end{align}

### A slab shield

Imagine a slab shield, thickness t, at a distance r, between the point source and a detector.

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{-\mu t}\\
\mbox{where}&\\
&t=\mbox{thickness of the slab}
\end{align}

If it were made of a series of materials $i$, with coefficients $\mu_i$, and thicknesses $t_i$:

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{\sum_i -\mu_i t_i}\\
\mbox{where}&\\
&\mu_i=\mbox{linear interaction coefficient of ith slab}\\
&t_i=\mbox{thickness of ith slab}
\end{align}

### Heterogeneous Medium

An arbitrary heterogeneous medium can be described as having an interaction coefficient $\mu(\vec{r})$ at any point $\vec{r}$ in the medium, a funciton of position in the medium.

\begin{align}
&\phi^o(r) = \frac{S_p}{4\pi r^2}e^{\left[-\int_0^r \mu(s) ds\right]}\\
\end{align}

# Announcements
* If you had trouble tagging pages based on problems for HW06, this should be fixed by the end of class. If you already submitted, please edit your submission to tag problems appropriately.
* There will be a HW assigned next Friday. It will be entirely review material based on things that were difficult on previous assignments and the exam.
* The quiz and HW are posted. The quiz is still due on Monday.

## Polyenergetic Point Source

- Previous examples assume a **monoenergetic** point source (particles of a single energy, E).
- But, a single source can emit particles at several discrete energies, or even a continuum of energies.

Question: From last lecture, what do we expect of fission neutrons?

Let's define some variables:

\begin{align}
f_i &= \mbox{fraction of the source emitted with energy }E_i\\
E_i &= \mbox{discrete energy of }f_iS_p\mbox{ particles}\\
S_p &= \mbox{still the number of particles emitted from the point source}
\end{align}

The total interaction rate caused by uncollided particles streaming through a small volume mass at distance r from the source is the following, **for some set of i discrete energies**.

\begin{align}
R^o(r)=\sum_i\frac{S_p f_i\mu_d(E_i) \Delta V_d}{4\pi r^2}e^{\left[-\int_0^r \mu(s,E_i) ds\right]}\\
\end{align}

If the source emits a continuum of energies, it's best to define the fraction $f_i$ as a differential probability:

\begin{align}
N(E)dE\mbox{ the probability that a source particle is emitted with energy in dE about E}
\end{align}

With this definition, the sum over discrete energies becomes an integral.

\begin{align}
R^o(r)=\int_o^\infty \left[\frac{S_p N(E)\mu_d(E) \Delta V_d}{4\pi r^2}e^{\left[-\int_0^r \mu(s,E) ds\right]}\right]dE\\
\end{align}

Please note, you may see many nuclear texts list the dE first in the integral... don't be bamboozled. This is equivalent to the above:

\begin{align}
R^o(r)=\int_o^\infty dE\frac{S_p N(E)\mu_d(E) \Delta V_d}{4\pi r^2}e^{\left[-\int_0^r \mu(s,E) ds\right]}
\end{align}

### Example 7.4 from your book (Shultis & Faw)

A point source with an activity of 500 Ci emits 2-MeV photons with a frequency of 70% per decay.

\begin{align}
S_p = 500 Ci\\
f_2 = 0.7\\
\end{align}

What is the flux density of 2-MeV photons 1 meter from the source?

```python
s_p = 500 # Ci
f_2 = 0.7 # fraction emitted at 2MeV
mu = 1.0/187.0 # mean free path of 2MeV photon in air is 187m

# first, convert S_p is in number of particles per decay (Bq)
bq_to_ci = 3.7e10 # Bq/Ci
s_p = s_p*bq_to_ci

# Now, find uncollided flux of 2MeV photons at 1 m
r = 1.0 #m
s = s_p*f_2 # just want 2MeV photons
phi = phi_o_r(r, s)
print("Uncollided flux is : ", phi)

# Uh oh, we forgot the material attenuation!
phi = phi_o_r(r, s)*math.exp(-mu*r)
print("Flux after material attenuation is : ", phi)
```

    Uncollided flux is :  1030528256520.0223
    Flux after material attenuation is :  1025032118881.2917

### Think Pair Share

What are the units of $\phi^o$, above?

# Photon Interactions

**Recall:**

\begin{align}
c &= \mbox{speed of light}\\
  &=2.9979\times10^8\left[\frac{m}{s}\right]\\
E &= \mbox{photon energy}\\
  &=h\nu\\
  &=\frac{hc}{\lambda}\\
h &= \mbox{Planck's constant}\\
  &= 6.62608\times10^{−34} [J\cdot s] \\
\nu &=\mbox{photon frequency}\\
\lambda &= \mbox{photon wavelength}
\end{align}

**Nota bene:**
- **10eV - 20MeV** photons are important in radiation sheilding
- At **10eV - 20MeV**, only photoelectric effect, pair production, and Compton Scattering are significant

Figure from: "Radiation Interactions with Tissue." Radiology Key. Jan 8 2016.

Figure from: Cullen, D. E. 1994. "Photon and Electron Interaction Databases and Their Use in Medical Applications." UCRL-JC--117419. Lawrence Livermore National Lab. http://inis.iaea.org/Search/search.aspx?orig_q=RN:26035330.

## Klein Nishina

The total Compton cross section, per atom with Z electrons, based on the free-electron approximation, is given by the well-known Klein-Nishina formula [Evans 1955]:

\begin{align}
\sigma_c(E) =\pi Zr_e^2\lambda\left[(-2\lambda - 2\lambda^2)\ln{\left(1+\frac{2}{\lambda}\right)} + \frac{2(1+9\lambda + 8\lambda^2 + 2\lambda^3)}{(\lambda + 2)^2}\right]
\end{align}

Here $\lambda \equiv \frac{m_ec^2}{E}$, a dimensionless quantity, and $r_e$ is the classical electron radius. The value of $r_e$ is given by:

\begin{align}
r_e &\equiv \frac{e^2}{4\pi\epsilon_om_ec^2}\\
&= 2.8179\times10^{-13}cm
\end{align}

### Think pair share:
Conceptually, in the above equation:

- what is $r_e$?
- what is $e$?
- what is $\epsilon_o$?
- what is $m_ec^2$?

### Total Photon Cross Section
Various types of incoherent scattering, including Compton, are actually present in that intermediate energy range. It is occaisionally important to correct for all types of incoherent scattering, but it can typically be assumed to be primarily Compton scattering.

For photons, then $\mu$ becomes:

\begin{align}
\mu(E)&\equiv N\left[\sigma_{ph}(E) + \sigma_{inc}(E) + \sigma_{pp}(E)\right]\\
      &\simeq N\left[\sigma_{ph}(E) + \sigma_{c}(E) + \sigma_{pp}(E)\right]\\
    N &= \mbox{atom density}\\
      &= \frac{\rho N_a}{A}
\end{align}

It is common to denote this as the total mass interaction coefficient:

\begin{align}
\frac{\mu}{\rho} &= \frac{N_a}{A}\left[\sigma_{ph}(E) + \sigma_{c}(E) + \sigma_{pp}(E)\right]\\
&= \frac{N_a}{A}\left[\frac{\mu_{ph}(E)}{\rho} + \frac{\mu_{c}(E)}{\rho} + \frac{\mu_{pp}(E)}{\rho}\right]
\end{align}

## Neutron Interactions

Photons tend to interact with electrons in a target atom. **Neutrons tend to interact with the nucleus.**

Neutron cross sections:

- Vary rapidly with the incident neutron energy,
- Vary erratically from one element to another
- Even vary dramatically between isotopes of the same element.

There are lots of sources of neutron cross sections. The best place to start is the Brookhaven National Laboratory National Nuclear Data Center [https://www.nndc.bnl.gov/](https://www.nndc.bnl.gov/).

Your book has a clever table (7.1) listing some of the data needed for high and low energy interaction calculations. These include:

- Elastic scattering cross sections
- Angular distribution of elastically scattered neutrons
- Inelastic scattering cross sections
- Angular distribution of inelastically scattered neutrons
- Gamma-photon yields from inelastic neutron scattering
- Resonance absorption cross sections
- Thermal-averaged absorption cross sections
- Yield of neutron-capture gamma photons
- Fission cross sections and associated gamma-photon and neutron yields

# Total cross sections

**For light nuclei** ($A<25$) and $E<1keV$, the cross section typically varies as:

\begin{align}
\sigma_t = \sigma_1 + \frac{\sigma_2}{\sqrt{E}}
\mbox{where}&\\
\sigma_1 \mbox{ and }\sigma_2 \mbox{ are constants}& \\
\sigma_1&=\mbox{elastic scattering}\\
\frac{\sigma_2}{\sqrt{E}}&=\mbox{radiative capture}\\
\end{align}

**For solids** at energies less than about 0.01 eV, Bragg cutoffs apply. These are energies below which no coherent scattering is possible from the material's crystalline planes.

**For heavy nuclei**, the total cross section has a $\frac{1}{\sqrt{E}}$ behavior with low energy, narrow resonances and high energy broad resonances:

\begin{align}
\sigma_t \propto \frac{1}{\sqrt{E}}
\end{align}

### Recall fission cross sections :

```python

```