```python
import numpy as np
%load_ext autoreload
```

Notebook by **Maxime Dion** <maxime.dion@usherbrooke.ca>

For the QSciTech-QuantumBC virtual workshop on gate-based quantum computing

## Tutorial for  Activity 3.2

For this activity, make sure you can easily import your versions of `hamiltonian.py`, `pauli_string.py` and `mapping.py` that you have completed in the Activity 3.1 tutorial. You will also need your verions of `evaluator.py` and `solver.py`. Placing this notebook in the same `path` as these files is the easiest way to acheive this. At the end of this notebook, you should be in good position to complete these 2 additionnal files.

The solution we suggest here is NOT mandatory. If you find ways to make it better and more efficient, go on and impress us! On the other hand, by completing all sections of this notebook you'll be able to :
- Prepare a Quantum State based on a varitional form (circuit);
- Measure qubits in the X, Y and Z basis;
- Estimate expectation value of Pauli String on a quantum state;
- Evaluate the expectation value of an Hamiltonian in the form of a Linear Combinaison of Pauli Strings;
- Run a minimization algorithm on the energy expectation function to find the ground state of a Hamiltonian;
- Dance to express your overwhelming sense of accomplishment

**Important**

When you modify and save a `*.py` file you need to re-import it so that your modifications can be taken into account when you re-execute a call. By adding the magic command `%autoreload` at the beginning of a cell, you make sure that the modifications you did to the `*.py` files are taken into account when you re-run a celll and that you can see the effect.

If you encounter unusual results, restart the kernel and try again.

**Note on numbering**

When you ask a question in the Slack channel you can refer to the section name or the section number.

To enable the section numbering, please make sure you install [nbextensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/install.html). It is available in the conda distribution. After installation it you need to enable the option 'Table of contents (2)'.

# Variational Quantum States

Every quantum circuit starts with all qubits in the the state $|0\rangle$. In order to prepare a quantum state $|\psi\rangle$ we need to prepare a `QuantumCircuit` that will modify the states of the qubits in order to get this specific state. The action of a circuit can always be represented as a unitiary operator.

\begin{align}
    |\psi\rangle &= \hat{U} |0 \ldots 0\rangle
\end{align}

For a parametric state the `QuantumCircuit` and therefore the unitary $U$ will depend on some parameters that we wirte as $\boldsymbol{\theta}$.

\begin{align}
    |\psi(\boldsymbol{\theta})\rangle &= \hat{U}(\boldsymbol{\theta}) |0 \ldots 0\rangle
\end{align}

We will see 2 ways to define Parametrized Quantum Circuits that represent Variational Quantum States. For the first method we only need the `QuantumCircuit` class from `qiskit.circuit`.

```python
from qiskit.circuit import QuantumCircuit
```

## Generating function
The easiest way to generate a parametrized `QuantumCircuit` is to implement a function that takes parameters as arguments and returns a `QuantumCircuit`. Here is such a function that generates a 2 qubits QuantumCircuit.

```python
def example_2qubits_2params_quantum_circuit(theta,phi):
    qc = QuantumCircuit(2)
    qc.ry(theta,0)
    qc.rz(phi,0)
    qc.cx(0,1)
    return qc
```

To visualize this circuit we first need to call the generating function with dummy argument values for it to return a circuit. We can draw the circuit. The `'mpl'` option draws the circuit in a fancy way using `matplotlib`. If you are experiencing problems, you can remove this option.

```python
varform_qc = example_2qubits_2params_quantum_circuit
qc = varform_qc(1,2)
qc.draw('mpl')
```

## Using qiskit parameter

The other way to generate a parametrized `QuantumCircuit` is to use the `Parameter` class in `qiskit`.

```python
from qiskit.circuit import Parameter
```

Here is the same circuit as before done with this method.

```python
a = Parameter('a')
b = Parameter('b')
varform_qc = QuantumCircuit(2)
varform_qc.ry(a,0)
varform_qc.rz(b,0)
varform_qc.cx(0,1)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x13778e610>

Done this way the parametrized circuit can be drawn right away.

```python
varform_qc.draw('mpl')
```

To see what are the parameters of a parametrized `QuantumCircuit` you can use

```python
varform_qc.parameters
```

    {Parameter(a), Parameter(b)}

**Important** Beware that sometimes the parameters will not appear in the same order as you declared them!

To assign values to the different parameters we need to use the `QuantumCircuit.assign_paremeters()` method. This methods takes a `dict` as an argument containing the `Parameter`s and their `value`s.

```python
param_dict = {a : 1, b : 2}
qc = varform_qc.assign_parameters(param_dict)
qc.draw('mpl')
```

```python
param_dict = {a : 3, b : 4}
qc = varform_qc.assign_parameters(param_dict)
qc.draw('mpl')
```

If you want to provide the parameter values as a `list` or a `np.array` you can build the `dict` directly. Just make sure that the order you use in `param_values` corresponds to the other of `varform_qc.parameters`.

```python
param_values = [1, 2]
param_dict = dict(zip(varform_qc.parameters,param_values))
print(param_dict)
```

    {Parameter(a): 1, Parameter(b): 2}

## Varforms circuits for H2
Using the method of your choice, prepare 2 different 4-qubit `QuantumCircuit`s.
- The first should take 1 parameter to cover the real coefficients state sub space spanned by $|0101\rangle$ and $|1010\rangle$.
- The second should take 3 parameters to cover the real coefficients state sub space spanned by $|0101\rangle$, $|0110\rangle$, $|1001\rangle$ and $|1010\rangle$.

Revisit the presentation to find such circuits.

```python
varform_4qubits_1param = QuantumCircuit(4)
a = Parameter('a')
"""
Your code here
"""
varform_4qubits_1param.ry(a,1)
varform_4qubits_1param.x(0)
varform_4qubits_1param.cx(1,0)
varform_4qubits_1param.cx(0,2)
varform_4qubits_1param.cx(1,3)

varform_4qubits_1param.draw('mpl')
```

```python
varform_4qubits_3params = QuantumCircuit(4)
a = Parameter('a')
b = Parameter('b')
c = Parameter('c')
"""
Your code here
"""
varform_4qubits_3params.x(0)
varform_4qubits_3params.x(2)
varform_4qubits_3params.barrier(range(4))
varform_4qubits_3params.ry(a,1)
varform_4qubits_3params.cx(1,3)
varform_4qubits_3params.ry(b,1)
varform_4qubits_3params.ry(c,3)
varform_4qubits_3params.cx(3,2)
varform_4qubits_3params.cx(1,0)

varform_4qubits_3params.draw('mpl')
```

# Evaluator
The `Evaluator` is an object that will help us to evaluate the expectation value of a quantum operator (`LCPS`) on a specific variational form and backend. To initialize and `Evaluator` you should provide :

**Mandatory**
- A **variational form** that can create a `QuantumCircuit` given a set of `params`;
- A **backend** `qiskit.Backend` (a simulator or an actual device handle) on which to run the `QuantumCircuit`

**Optional**
- `execute_opt` is a `dict` containing the optionnal argument to pass to the `qiskit.execute` method (ex : `{'shots' : 1024}`.
- `measure_filter` a `qiskit.ignis...MeasurementFilter` that can be applied to the result of a circuit executation to mitigate readout errors.

The creation/usage of an `Evaluator` such as `BasicEvaluator` goes like this :

evaluator = BasicEvaluator(varform_qc,backend)
evaluator.set_linear_combinaison_pauli_string(operator_lcps)
expected_value = evaluator.eval(params)

First you initialize the evaluator.

Next, you provide the operator you want to evaluate using the `set_linear_combinaison_pauli_string(LCPS)` method.

Finally, you call the `eval(params)` method that will return the estimation of the operator's expected value. Mathematicaly, the use of this method corresponds to

\begin{align}
E(\boldsymbol{\theta}).
\end{align}

We will now go through the different pieces neccessary to complete the `Evaluator` class.

## Static methods
Being static, these method do not need an instance of a class to be used. They can be called directly from the class.

These methods are called before the first call to `eval(params)`. Most of these methods are implemented inside the abstract class `Evaluator` (except for `prepare_measurement_circuits_and_interpreters(LCPS)`)

### Pauli Based Measurements
We have seen that even if a quantum computer can only measure qubits in the Z-basis, the X and Y-basis are accessible if we *rotate* the quantum state before measuring.

Implement the `@staticmethod` : `pauli_string_based_measurement(PauliString)` in the `Evaluator` class in file `Evaluator.py` that returns a `QuantumCircuit` that measures each qubit in the basis given by the `PauliString`.

First we import the abstract class `Evaluator` and the `PauliString` class.

```python
from evaluator import Evaluator
from pauli_string import PauliString
```

Test your code with the next cell.

```python
%autoreload
pauli_string = PauliString.from_str('ZIXY')
measure_qc = Evaluator.pauli_string_based_measurement(pauli_string)
measure_qc.draw('mpl')
```

### Measurable eigenvalues

Implement the `@staticmethod` : `measurable_eigenvalues(PauliString)` in the `Evaluator` class in file `Evaluator.py` that returns a `np.array` that contains the eigenvalues of the measurable `PauliString` for each basis state. We noted this vector

\begin{align}
    \Lambda_q^{\hat{(\mathcal{P})}}.
\end{align}

Be mindful of the order of the basis state.

\begin{align}
    0000, 0001, 0010, \ldots, 1110, 1111
\end{align}

You can test your implementation on the `ZIXY` Pauli string.

```python
%autoreload
pauli_string = PauliString.from_str('ZIXY')
measurable_eigenvalues = Evaluator.measurable_eigenvalues(pauli_string)
print(measurable_eigenvalues)
```

    [ 1 -1 -1  1  1 -1 -1  1 -1  1  1 -1 -1  1  1 -1]

For the `PauliString` `'ZIXY'` (measurable `'ZIZZ'`) you should get the following eigenvalues :

[ 1 -1 -1  1  1 -1 -1  1 -1  1  1 -1 -1  1  1 -1]

### Measurement Circuits and Interpreters
The `prepare_measurement_circuits_and_interpreters(LCPS)` is specific to the sub-type of `Evaluator`. The two different types of `Evaluator`s considered in this workshop are :
- The `BasicEvaluator` will run a single `QuantumCircuit` for each `PauliString` present in the provided `LCPS`.
- The `BitwiseCommutingCliqueEvaluator` will exploit Bitwise Commuting Clique to combine the evaluation of Commuting `PauliStrin`s and reduce the number of different `QuantumCircuit` run for each evaluation.

Implement the `prepare_measurement_circuits_and_interpreters(LCPS)` method in the `BasicEvaluator` class in file `Evaluator.py`. This method should return 2 `list`. The first should contain one measurement `QuantumCircuit` for each `PauliString` in the `LCPS`. The second list should contain one `np.array` of the eigenvalues of the measurable `PauliString` for each basis state.

**Note** You can try to implement similar methods for the `BitwiseCommutingCliqueEvaluator`.

You can test your method on `2 ZIXY + 1 IXYZ`.

```python
from evaluator import BasicEvaluator
```

```python
%autoreload
lcps = 2*PauliString.from_str('ZIXY') + 1*PauliString.from_str('IXYZ')
measurement_circuits, interpreters = BasicEvaluator.prepare_measurement_circuits_and_interpreters(lcps)
```

You can visualize the interpreter and the measurement circuit for each term in the `LCPS` by using `i = 0` and `i = 1`.

```python
i = 1
print(interpreters[i])
measurement_circuits[i].draw('mpl')
```

The interpreters should be respectively :

[ 2 -2 -2  2  2 -2 -2  2 -2  2  2 -2 -2  2  2 -2]
[ 1 -1 -1  1 -1  1  1 -1  1 -1 -1  1 -1  1  1 -1]

### Set the LCPS
The method `set_linear_combinaison_pauli_string(LCPS)` is already implemented inside the abstract class `Evaluator`. Please take a look at it to notice that this method makes an immediate call to the `prepare_measurement_circuits_and_interpreters(LCPS)` method you have just implemented. The `measurement_circuits` and `interpreters` are also stored in attributes of the same name.

## Methods called inside `eval(params)`
Since we are entering the action of the `eval(params)` method we will need to instantiate an `Evaluator`. This will require a `backend`. We will use a local `qasm_simulator` for now, which is part of the `Aer` module. In the future, you can use a different `backend`. We will also soon need the `execute` method.

```python
from qiskit import Aer, execute
qasm_simulator = Aer.get_backend('qasm_simulator')
```

### Circuit preparation
The `prepare_eval_circuits(params)` will combine the variational form with these measurement `QuantumCircuit`s to form the complete circuit to be run. This method has 2 tasks :
- Assign the `params` to the variationnal form to get a `QuantumCircuit` that prepares the quantum state
- Combine this circuit with all the measurement circuits to return as many `QuantumCircuit` inside a `list`.

Implement this method inside the `Evaluator` class and test it here.

```python
%autoreload
lcps = 2*PauliString.from_str('ZXZX') + 1*PauliString.from_str('IIZZ')
varform = varform_4qubits_1param
backend = qasm_simulator
evaluator = BasicEvaluator(varform,backend)
evaluator.set_linear_combinaison_pauli_string(lcps)
params = [0,]
eval_circuits = evaluator.prepare_eval_circuits(params)
```

You can take a look at the `QuantumCircuit` for the first (`i=0`) and second (`i=1`) PauliString. What you should get is a circuit that begins with the state preparation circuit with the `params` applied to it followed by the measurement circuit.

```python
i = 0
eval_circuits[i].draw('mpl')
```

### Execution
The ultimate goal to the execution of a circuit is to get the number of times each basis state is measured. Let's execute our `eval_circuits`. We can run many `QuantumCircuit`s at the same time by placing them into a `list`, which they already are!

```python
execute_opts = {'shots' : 1024}
job = execute(eval_circuits, backend=qasm_simulator, **execute_opts)
result = job.result()
```

We can get the number of counts of each state for the execution of a given circuit with the follow lines. The counts are returned as a `dict`.

```python
i = 0
#i = 1
counts = result.get_counts(eval_circuits[i])
print(counts)
```

    {'0000': 266, '0001': 251, '0100': 262, '0101': 245}

If you `eval_circuits` are correct, you should get for, `i = 0` and `i = 1` respectively, something like this (exact value may vary since there is some randomness in the executation of a quantum circuit)

{'0000': 266, '0001': 262, '0100': 240, '0101': 256}
{'0101': 1024}

### counts2array

We will transform this `dict` into an array with the `counts2array` method.  Implement this method that will return the vector $N_q$. Be mindful of the order of the basis state.

\begin{align}
    0000, 0001, 0010, \ldots, 1110, 1111
\end{align}

**optional remark** While doing this will allow us to interpret the counts with a simple inner product, this implies creating an array of size $2^n$ where $n$ is the numbers of qubits. This might not be such a good idea for larger systems and the use of a `dict` might be more appropriate. Can you interpret the counts efficiently while keeping them in a `dict`?

```python
%autoreload
i = 0
counts = result.get_counts(eval_circuits[i])
evaluator.counts2array(counts)
```

    array([266, 251,   0,   0, 262, 245,   0,   0,   0,   0,   0,   0,   0,
             0,   0,   0])

For `i=0` in particular you should get something similar to:

array([228., 276.,   0.,   0., 269., 251.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.])

### Interpret counts
The action of interpreting the counts is actually the task of estimating the expectation value of a `PauliString` and multiplying by the coefficient associated with this `PauliString` in the LCPS. Implemented the `interpret_count_array` method that should return an array with the values for this expression.

\begin{align}
    h_i \langle \hat{\mathcal{P}}_i \rangle = \frac{h_i}{N_\text{tot}}\sum_{q} N_q \Lambda_q^{(\hat{\mathcal{P}}_i)}
\end{align}

```python
%autoreload
i = 1
counts_array = evaluator.counts2array(result.get_counts(eval_circuits[i]))
interpreter = evaluator.interpreters[i]
expected_value = evaluator.interpret_count_array(interpreter,counts_array)
print(expected_value)
```

    (-1+0j)

You should get something close to `0` for the first one and `-1` for the second.

### Evaluation
You have now all the pieces to complete the `eval(params)` method. This method should use all the methods you've implemented since the section *Methods called inside `eval(params)`* and then sum all the interpreted values. Mathematicaly, it should return the value of the expression

\begin{align}
    E(\boldsymbol{\theta}) = \sum_i h_i \langle\psi(\boldsymbol{\theta}) | \hat{\mathcal{P}}_i | \psi(\boldsymbol{\theta}) \rangle.
\end{align}

```python
%autoreload
lcps = 2*PauliString.from_str('ZXZX') + 1*PauliString.from_str('IIZZ')
varform = varform_4qubits_1param
backend = qasm_simulator
execute_opts = {'shots' : 1024}
evaluator = BasicEvaluator(varform, backend, execute_opts=execute_opts)
evaluator.set_linear_combinaison_pauli_string(lcps)
params = [0,]
expected_value = evaluator.eval(params)
print(expected_value)
```

    -1.0703125

Yes that's right, your code now returns an estimate of the expression

\begin{align}
    E(\theta) = \langle \psi(\theta) | \hat{\mathcal{H}} | \psi(\theta) \rangle.
\end{align}

for

\begin{align}
    \hat{\mathcal{H}} = 2\times \hat{Z}\hat{X}\hat{Z}\hat{X} + 1\times \hat{I}\hat{I}\hat{Z}\hat{Z}
\end{align}

and the varform `varform_4qubits_1param` for $\theta = 0$. The `evaluator.eval(params)` is now a method you can call like a function and it will return the energy $E(\theta)$.

Now comes the time to test this on the $\text{H}_2$ molecule Hamiltonian!

## The Hamiltonian evaluation test

We will now import the classes from the previous activity.

```python
from hamiltonian import MolecularFermionicHamiltonian
from mapping import JordanWigner
```

For ease of use we will import the integral values instead of using `pyscf`. We also import the Coulomb repulsion energy for later use. By now we are experts in building the Hamiltonian.

```python
with open('Integrals_sto-3g_H2_d_0.7350_no_spin.npz','rb') as f:
    out = np.load(f)
    h1_load_no_spin = out['h1']
    h2_load_no_spin = out['h2']
    energy_nuc =  out['energy_nuc']

molecular_hamiltonian = MolecularFermionicHamiltonian.from_integrals(h1_load_no_spin,h2_load_no_spin).include_spin()
```

We use the Jordan-Wigner mapping to the get the `LCPS` for the H2 molecule with `d=0.735`.

```python
%autoreload
mapping = JordanWigner()

lcps_h2 = mapping.fermionic_hamiltonian_to_linear_combinaison_pauli_string(molecular_hamiltonian).combine().apply_threshold().sort()
print(lcps_h2)
```

    15 pauli strings for 4 qubits (Real, Imaginary)
    IIII (-0.81055,+0.00000)
    IIIZ (+0.17218,+0.00000)
    IIZI (-0.22575,+0.00000)
    IIZZ (+0.12091,+0.00000)
    IZII (+0.17218,+0.00000)
    IZIZ (+0.16893,+0.00000)
    IZZI (+0.16615,+0.00000)
    ZIII (-0.22575,+0.00000)
    ZIIZ (+0.16615,+0.00000)
    ZIZI (+0.17464,+0.00000)
    ZZII (+0.12091,+0.00000)
    XXXX (+0.04523,+0.00000)
    XXYY (+0.04523,+0.00000)
    YYXX (+0.04523,+0.00000)
    YYYY (+0.04523,+0.00000)

We build an evaluator and feed it the `LCPS` of H2. And them we evaluate the energy. Use `params` in order that your `varform` prepares the state $|0101\rangle$.

```python
%autoreload
varform = varform_4qubits_1param
backend = qasm_simulator
execute_opts = {'shots' : 1024}
evaluator = BasicEvaluator(varform,backend,execute_opts = execute_opts)
evaluator.set_linear_combinaison_pauli_string(lcps_h2)
params = [0,]
expected_value = evaluator.eval(params)
print(expected_value)
```

    -1.8370563365153778

If your `varform` prepares the state $|0101\rangle$, you should get something around `-1.83`. This energy is already close to the ground state energy because the ground state is close to $|0101\rangle$, but still it's not the ground state. We need to find the `params` that will minimise the energy.

\begin{align}
    E_0 = \min_{\boldsymbol{\theta}} E(\boldsymbol{\theta})
\end{align}

# Solver

In a final step we need to implement a solver that will try to find the minimal energy. We will implement 2 solvers. The second is optional.
- First the one using the VQE algo in conjunction with a minimizer to try to minimize `evaluator.eval(params)`.
- Next we will make use of the `to_matrix()` method you implemented in the previous activity to find the exact value/solution.

## VQE Solver

Like any minimzation process this solver will need a couple of ingredients :
- A function to minimize, we will provide this with the evaluator
- A minimizer, an algorithm that generaly takes in a function and a set of starting parameters and returns the best guess for the optimal parameters that correspond to the minimal value of the function to minimize.
- A set of starting parameters.

### Minimizer

A minimizer that works OK for the VQE algorithme is the  Sequential Least SQuares Programming (SLSQP) algorithm. It's available in the `minimize` sub-module of [scipy](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html).

```python
from scipy.optimize import minimize
```

We will make a Lambda function with the minimizer so we can set all sorts of parameter before feeding it to the solver.

```python
minimizer = lambda fct, start_param_values : minimize(
    fct,
    start_param_values,
    method = 'SLSQP',
    options = {'maxiter' : 5,'eps' : 1e-1, 'ftol' : 1e-4, 'disp' : True, 'iprint' : 2})
```

The `minimizer` now takes only 2 arguments : the function and the starting parameters values. We also specify some options :
- A small value for the maximum number of iteration. You will find that running the VQE algorithm is expensive because of the `evaluator.eval(params)` method. Either it's long to simulate on `qasm_simulator` or because it's running on an actual quantum computer.
- A `eps` of `0.1`. This is the size of the step the algorithm is going to change the values of the parameters to try to estimate the slope of the function. By the way, a lot of minimizing algorithms use the slope of the function to know in which direction is the minimum. Since our parameters are all angles in radians a value of 0.1 seems reasonnable. Play with this value if you like.
- A `ftol` value of `1e-4`. This is the goal for the precision of the value of the minimum value. The chemical accuracy is around 1 milli-Hartree.
- We set `iprint` to `2` so see what is going on. For your final implementation you can set this to `0`.

Before implementing the `VQESolver` let's try this minimizer! The function is `evaluator.eval` and we start with a parameter of `0`. This will take a while.

```python
minimization_result = minimizer(evaluator.eval,[0,])
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.834406E+00     2.276211E-01
        2     6    -1.857755E+00     4.369813E-02
        3    11    -1.860118E+00     1.790857E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.8601177724457674
                Iterations: 3
                Function evaluations: 11
                Gradient evaluations: 3

In the end you should get an minimal energy around `1.86` Hartree. Which is a bit smaller then what we had before minimizing. You can explore the `minimization_result` to retreive this value but also the set of optimal parameters.

```python
opt_params = minimization_result.x
opt_value = minimization_result.fun
print(opt_params)
print(opt_value)
```

    [-0.22882531]
    -1.8601177724457674

### VQE Solver

Now you should be in good position to implement the `lowest_eig_value(lcps)` of the `VQESolver` class inside the `Solve.py` file. Test your method here.

```python
from solver import VQESolver
```

```python
%autoreload
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver')
opt_value, opt_params = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.839000E+00     2.553151E-01
        2     6    -1.865535E+00     6.395731E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.865591974724343
                Iterations: 2
                Function evaluations: 10
                Gradient evaluations: 2

The is only one thing missing to have the complete molecular energy : the Coulomb repulsion energy from the nucleus. This value was loaded when we imported the integrals. Let's add it to the electronic energy.

```python
print('Ground state position estimate (vqe) : ', opt_params)
print('Ground state energy estimate (electronic, vqe) : ', opt_value)
print('Ground state energy estimate (molecular, vqe) : ', opt_value + energy_nuc)
```

    Ground state position estimate (vqe) :  [-0.25526396]
    Ground state energy estimate (electronic, vqe) :  -1.865591974724343
    Ground state energy estimate (molecular, vqe) :  -1.1456229802753635

### The Eigenstate

What is the eigenstate? We can partially find out by using the `varform` with the parameters we have found and measure everything in the Z basis.

```python
eigenstate_qc = varform.copy()
eigenstate_qc.measure_all()

param_dict = dict(zip(eigenstate_qc.parameters,opt_params))
eigenstate_qc = eigenstate_qc.assign_parameters(param_dict)

eigenstate_qc.draw('mpl')
```

We now execute this circuit.

```python
execute_opts = {'shots' : 1024}
job = execute(eigenstate_qc,backend=qasm_simulator,**execute_opts)
result = job.result()
counts = result.get_counts(eigenstate_qc)
```

We will use the `plot_histogram` method from `qiskit.visualization` that takes the counts `dict` as an input.

```python
from qiskit.visualization import plot_histogram
```

```python
plot_histogram(counts)
```

```python
print(f"|a_0101| ~ {np.sqrt(counts['0101']/1024)}")
print(f"|a_1010| ~ {np.sqrt(counts['1010']/1024)}")
```

    |a_0101| ~ 0.9936320684740404
    |a_1010| ~ 0.11267347735824966

We see that the found solution is mostly the state $|0101\rangle$ which is the Hartree-Fock solution when the 2-body Hamiltonian is not present. Adding this 2-body physics, shifts the energy down a bit by introducing a small contribution of $|1010\rangle$. The actual statevector has a `-` sign between these two states.

\begin{align}
\alpha_{0101}|0101\rangle - \alpha_{1010}|1010\rangle
\end{align}

But this is not something we can know from this. Fortunatly, H2 is a small system which can be solved exactly and we can find out this phase.

## Exact Solver (optional)

If you want to compare the value you get with the VQE algorithm it would be nice to have the exact value. If you were able to implement the `to_matrix()` method for `PauliString` and `LinearCombinaisonPauliString` then you can find the exact value of the ground state. All you need is to diagonalise the matrix reprensenting the whole Hamiltonian and find the lowest eigenvalue! Obviously this will not be possible to do for very large systems.

```python
hamiltonian_matrix_h2 = lcps_h2.to_matrix()
eig_values, eig_vectors = np.linalg.eigh(hamiltonian_matrix_h2)
eig_order = np.argsort(eig_values)
eig_values = eig_values[eig_order]
eig_vectors = eig_vectors[:,eig_order]
ground_state_value, ground_state_vector = eig_values[0], eig_vectors[:,0]
print('Ground state vector (exact) : \n', ground_state_vector)
print('Ground state energy (electronic, exact) : ', ground_state_value)
print('Ground state energy (molecular, exact) : ', ground_state_value + energy_nuc)
```

    Ground state vector (exact) :
     [-0.        -0.j -0.        -0.j -0.        -0.j -0.        -0.j
     -0.        -0.j -0.9937604 -0.j  0.        +0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.11153594+0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.        +0.j  0.        +0.j]
    Ground state energy (electronic, exact) :  -1.8572750302023788
    Ground state energy (molecular, exact) :  -1.137306035753399

Now you can complete the `ExactSolver` in the `Solver.py` file.

```python
from solver import ExactSolver
```

```python
%autoreload
exact_solver = ExactSolver()
ground_state_value, ground_state_vector = exact_solver.lowest_eig_value(lcps_h2)
print('Ground state vector (exact) : ', ground_state_vector)
print('Ground state energy (electronic, exact) : ', ground_state_value)
print('Ground state energy (molecular, exact) : ', ground_state_value + energy_nuc)
```

    Ground state vector (exact) :  [-0.        -0.j -0.        -0.j -0.        -0.j -0.        -0.j
     -0.        -0.j -0.9937604 -0.j  0.        +0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.11153594+0.j  0.        +0.j
      0.        +0.j  0.        +0.j  0.        +0.j  0.        +0.j]
    Ground state energy (electronic, exact) :  -1.8572750302023788
    Ground state energy (molecular, exact) :  -1.137306035753399

What are the two basis state involved in the ground state? Let's plot the state vector using `matplotlib`.

```python
import matplotlib.pyplot as plt
```

```python
fig,ax = plt.subplots(1,1)
i_max = np.argmax(np.abs(ground_state_vector))
state = ground_state_vector * np.sign(ground_state_vector[i_max])
ax.bar(range(len(state)), np.abs(state), color=(np.real(state) > 0).choose(['r','b']))
plt.xticks(range(len(state)),[f"{i:04b}" for i in range(len(state))], size='small',rotation=60);
```

# What's next?

Now that you can find the ground state for a specific H2 molecule configuration (`d = 0.735`), you should be able to do that for many configurations, say `d = 0.2` to `2.5`. Doing that will enable you to plot the so-called dissociation curve : energy vs distance. Do not forget to include the Coulomb repulsion energy of the nucleus!

You could also run your algorithm on a noisy backend, either a noisy simulator or a real quantum computer. You've already seen on day 1 how to set/get a noisy backend. You'll see that noise messes things up pretty bad.

Running on real machine will introduce the problem of the qubit layout. You might want to change the `initial_layout` in the `execute_opts` so that your `varform` is not applying CNOT gates between qubits that are not connected. You know this needs to insert SWAP gate and this introduce more noise. Also covered in day 1.

To limit the effect of readout noise, you could add a `measure_filter` to your `evaluator`, so that each time you execute the `eval_circuits` you apply the filter to the results. Also covered in day 1.

Implement the simulatneous evaluation for bitwise commuting cliques or even for general commuting cliques.

Notebook by **Maxime Dion** <maxime.dion@usherbrooke.ca>

For the QSciTech-QuantumBC virtual workshop on gate-based quantum computing

### Plot the H2 Dissociation Curve

```python
from pyscf import gto
```

```python
%autoreload
n = 50
distances = np.linspace(0.3, 2.5, n)
gs_energies_exact = np.zeros(n)
gs_energies_vqe = np.zeros(n)
energy_nuc = np.zeros(n)

# define mapping
mapping = JordanWigner()

# define minimizer
minimizer = lambda fct, start_param_values : minimize(
    fct,
    start_param_values,
    method = 'SLSQP',
    options = {'maxiter' : 5,'eps' : 1e-1, 'ftol' : 1e-4})

# instantiate an exact solver for comparison
exact_solver = ExactSolver()

# VQE setup
vqe_evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator, execute_opts={'shots' : 1024})
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver')

# try a range of internuclear distances
for i, distance in enumerate(distances): #units in AA
    print('Trying Distance '+str(i+1), end="\r")

    # build the molecule and basis functions
    mol = gto.M(
        atom = [['H', (0,0,-distance/2)], ['H', (0,0,distance/2)]],
        basis = 'sto-3g'
        )

    # build the molecular Hamiltonian
    molecular_hamiltonian = MolecularFermionicHamiltonian.from_pyscf_mol(mol).include_spin()

    # map the Hamiltonian to a LCPS
    lcps_h2 = mapping.fermionic_hamiltonian_to_linear_combinaison_pauli_string(molecular_hamiltonian).combine().apply_threshold().sort()

    # store the nuclear energy
    energy_nuc[i] = mol.energy_nuc()

    # diagonalize the Hamiltonian to get energies
    Eh2_exact, _ = exact_solver.lowest_eig_value(lcps_h2)
    gs_energies_exact[i] = Eh2_exact + energy_nuc[i]

    # get the vqe energy
    Eh2_vqe, _ = vqe_solver.lowest_eig_value(lcps_h2)
    gs_energies_vqe[i] = Eh2_vqe + energy_nuc[i]

print("Done!                   ", end="\r")
```

    Done!

```python
# plot dissociation curve of H2
fig, ax = plt.subplots(1, 1, figsize=(10,8))
ax.plot(distances, gs_energies_exact, c='tab:red', label='Exact', linewidth=5)
ax.plot(distances, gs_energies_vqe, '.', c='tab:blue', label='VQE', ms=20)
ax.set_xlabel(r'Internuclear Distance / $\AA$', fontsize=20)
ax.set_ylabel('Energy / $E_h$', fontsize=20)
ax.set_title('Dissociation Curve of H2', fontsize=28)
ax.legend()
fig.savefig('H2_dissociation.png')
plt.show()
```

```python
# save these results
with open('h2_dissociation.npz','wb') as f:
    np.savez(f, atom='H2', basis=mol.basis, distances=distances, energy_nuc=energy_nuc, gs_exact=gs_energies_exact,
             gs_vqe=gs_energies_vqe, varform='varform_4qubits_1param', backend='qasm_simulator', execute_opts=execute_opts,
             mapping='Jordan Wigner', initial_params=[0,], minimizer='SLSQP', minimizer_options={'maxiter' : 5,'eps' : 1e-1, 'ftol' : 1e-4})
```

```python
# to reload any data...
with open('h2_dissociation.npz','rb') as f:
    out = np.load(f, allow_pickle=True)
    varform_load = out['varform']
    basis_load = out['basis']
    energy_nuc_load = out['energy_nuc']
    gs_exact_load = out['gs_exact']
    gs_vqe_load = out['gs_vqe']
    execute_opts_load = out['execute_opts']
```

# Now Let's Add A Realistic Noise Model to our Simulator

```python
from qiskit import IBMQ
from qiskit.providers.aer.noise import NoiseModel

# IBMQ.save_account(TOKEN)
IBMQ.load_account()
IBMQ.providers()

#provider = IBMQ.get_provider(hub='ibm-q-education')

provider = IBMQ.get_provider(hub='ibm-q-education', group='qscitech-quantum', project='qc-bc-workshop')
provider2 = IBMQ.get_provider(hub='ibm-q', group='open', project='main')
```

    /Users/bhenders/opt/miniconda3/envs/qiskit/lib/python3.8/site-packages/qiskit/providers/ibmq/ibmqfactory.py:192: UserWarning: Timestamps in IBMQ backend properties, jobs, and job results are all now in local time instead of UTC.
      warnings.warn('Timestamps in IBMQ backend properties, jobs, and job results '
    ibmqfactory.load_account:WARNING:2021-02-01 14:15:45,480: Credentials are already in use. The existing account in the session will be replaced.

## Let's Play Around With a Few Backends with different Topologies

```python
bogota = provider.get_backend('ibmq_bogota')
# santiago = provider.get_backend('ibmq_santiago')
casablanca = provider.get_backend('ibmq_casablanca')
rome = provider.get_backend('ibmq_rome')
qasm_simulator = Aer.get_backend('qasm_simulator')
valencia = provider2.get_backend('ibmq_valencia')
melbourne = provider2.get_backend('ibmq_16_melbourne')
```

```python
# Bogota
bogota_prop = bogota.properties()
bogota_conf = bogota.configuration()
bogota_nm = NoiseModel.from_backend(bogota_prop)

# Casablanca
casablanca_conf = casablanca.configuration()
casablanca_prop = casablanca.properties()
casablanca_nm = NoiseModel.from_backend(bogota_prop)

# Valencia
valencia_conf = valencia.configuration()
valencia_prop = valencia.properties()
valencia_nm = NoiseModel.from_backend(bogota_prop)

# Melbourne
melbourne_conf = melbourne.configuration()
melbourne_prop = melbourne.properties()
melbourne_nm = NoiseModel.from_backend(bogota_prop)
```

```python
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates}
evaluator = BasicEvaluator(varform,backend,execute_opts = execute_opts)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver')
opt_energy, opt_params = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.644299E+00     4.420509E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6534026525155174
                Iterations: 1
                Function evaluations: 9
                Gradient evaluations: 1

```python
execute_opts = {'shots' : 1024}
evaluator = BasicEvaluator(varform, bogota, execute_opts=execute_opts)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name = 'vqe_solver_bogota')
opt_energy, opt_params = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.615201E+00     3.814424E-01
        2    10    -1.619136E+00     8.165022E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6191355224178043
                Iterations: 2
                Function evaluations: 10
                Gradient evaluations: 2

Notice that the ground state energy of an $H_2$ molecule is found to be -1.653 $E_h$ when our circuit runs on a noisy backend, with the same qubit coupling map as the Bogota device. When we run on the actual Bogota backend, we obtain an even worse result of -1.619 $E_h$. We compare this to -1.866 $E_h$ when running on an ideal simulator with only statistical noise contributing to potential error. We can try to mitigate some of this discrepancy by applying a `MeasurementFilter` to our circuit measurements.

**Plan of Attack:**
1. Decide on an optimal qubit layout (or 2 good ones) to minimize extra CX gates and error-prone U2 gates.
2. Generate **measurement calibration circuits** using these layouts.
3. Add a measurement filter to the VQE evaluator
4. Compare to un-filtered results

Note that our circuit uses 4 qubits. Perhaps one of the biggest optimizations we could do would be using parity mapping to reduce qubit requirements.

### Qubit Mapping

Keep in mind our variational circuit:

```python
qc = varform_4qubits_1param.assign_parameters({a: 1})
qc.draw()
```

Take a look at the coupling on Bogota, the actual machine we hope to use. Then examine the error rates for the CX and single qubit unitaries on the Bogota machine to get a sense of how we might better map our problem to this device.

```python
bogota_conf.coupling_map
```

    [[0, 1], [1, 0], [1, 2], [2, 1], [2, 3], [3, 2], [3, 4], [4, 3]]

```python
# Print CNOT error from Bogota calibration data
cx_errors = list(map(lambda cm: bogota_prop.gate_error("cx", cm), bogota_conf.coupling_map))
for i in range(len(bogota_conf.coupling_map)):
    print(f'      -> qubits {bogota_conf.coupling_map[i]} CNOT error: {cx_errors[i]}')
```

          -> qubits [0, 1] CNOT error: 0.02502260182900265
          -> qubits [1, 0] CNOT error: 0.02502260182900265
          -> qubits [1, 2] CNOT error: 0.010158268780223023
          -> qubits [2, 1] CNOT error: 0.010158268780223023
          -> qubits [2, 3] CNOT error: 0.014415524414420677
          -> qubits [3, 2] CNOT error: 0.014415524414420677
          -> qubits [3, 4] CNOT error: 0.010503141811223582
          -> qubits [4, 3] CNOT error: 0.010503141811223582

* CNOT gates between [0,1] and between [2,3] seem to have the largest error. Can we avoid these?

```python
# Print U2 error from Bogota calibration data
u2_errors = list(map(lambda q: bogota_prop.gate_error("sx", q), range(bogota_conf.num_qubits)))
for i in range(bogota_conf.num_qubits):
    print(f'      -> qubits {i} U2 error: {u2_errors[i]}')
```

          -> qubits 0 U2 error: 0.00031209152498965555
          -> qubits 1 U2 error: 0.00029958716199301446
          -> qubits 2 U2 error: 0.00017693377775244873
          -> qubits 3 U2 error: 0.0004023787341145875
          -> qubits 4 U2 error: 0.00016725621608793646

* Qubit 3 seems to have the largest error. Can we avoid using it?

Let's experiment with several different qubit layouts to see if we can reduce the number of CX gates on problematic pairs and U2 gates on problematic qubits.

```python
# all of these have optimal number of CNOTS (3)
layout = [2,3,1,4]  # 1
layout = [1,2,0,3]  # 2
layout = [3,2,4,1]  # 3 Looks the most promising
layout = [2,1,3,0]  # 4

# These are the equivalent topologies on Valencia
layout_valencia = [1,3,2,4]
layout_valencia = [1,3,4,2]
layout_valencia = [3,1,4,2]
layout_valencia = [3,1,2,4]
layout_valencia = [3,1,4,0]
layout_valencia = [1,3,4,0]

qc_l1 = transpile(qc,
                  coupling_map=bogota_conf.coupling_map,
                  basis_gates=bogota_conf.basis_gates,
                  initial_layout=layout,
                  optimization_level=1)
qc_l1.draw()
```

```python
print(f'Original circuit depth: {qc.depth()} - Transpiled circuit depth: {qc_l1.depth()}')
```

    Original circuit depth: 3 - Transpiled circuit depth: 6

**Summary of Findings**:

* **Layout 1**: depth 6 with opt level 1 (no improvement for higher)
    - downside: uses cx between 0, 1, which has highest error rate, plus 4 U2s on q3, which has highest error rate. one cx on 2,3
* **Layout 2**: depth 6 with opt level 1 (no improvement for higher)
    - downside: uses cx between 0, 1, which has highest error rate, one cx on 2,3
* **Layout 3**: depth 6 with opt level 1 (no improvement for higher)
    - downside: One cx between 2,3
* **Layout 4**: depth 6 with opt level 1 (no improvement for higher)
    - downside: uses cx between 0, 1, which has highest error rate, plus one on 2,3.

**Conclusion**: Layout 3 is probably optimal.

## Now Create a Measurement Filter

```python
from qiskit.circuit import QuantumRegister
from qiskit.ignis.mitigation.measurement import complete_meas_cal

# Generate the calibration circuits for the 3 qubits we measure
qr = QuantumRegister(4)

# we need our measurement filter to handle 4 qubits
qubit_list = [0,1,2,3]

# meas_calibs is a list containing 2^n circuits, one for each state.
meas_calibs, state_labels = complete_meas_cal(qubit_list=qubit_list, qr=qr, circlabel='mcal')

print(f'Number of circuits: {len(meas_calibs)}')
meas_calibs[1].draw()
```

```python
# We need the filter to correspond to the layout we are using
calibration_layout = [3,2,4,1]
result = execute(meas_calibs,
                 qasm_simulator,
                 shots=8192,
                 noise_model=bogota_nm,
                 coupling_map=bogota_conf.coupling_map,
                 basis_gates=bogota_conf.basis_gates,
                 initial_layout=calibration_layout).result()
```

```python
from qiskit.visualization import plot_histogram

# For example, plot histogram for circuit corresponding to state '001' (index 1)
plot_histogram(result.get_counts(meas_calibs[5]))
```

```python
from qiskit.ignis.mitigation.measurement import CompleteMeasFitter

# Initialize the measurement correction fitter for a full calibration
meas_fitter = CompleteMeasFitter(result, state_labels)

# Get the filter object
meas_filter = meas_fitter.filter

fig, ax = plt.subplots(1,1, figsize=(10,8))
meas_fitter.plot_calibration(ax=ax)
fig.savefig('images/4_qubit_measurement_filter.svg')
```

### Bogota Noise Model But With all Default Qubit Layout and No Measurement Filtersavefig

```python
%autoreload
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates,
                }
evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator,execute_opts=execute_opts, measure_filter=None)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name='vqe_solver')
energy_default, opt_params_default = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.660741E+00     1.105373E-01
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6647674403625325
                Iterations: 1
                Function evaluations: 10
                Gradient evaluations: 1

### Bogota Noise Model With An Improved Qubit Layout but no Measurement Filter

```python
%autoreload
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates,
                'initial_layout': [3,2,4,1]
                }
evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator,execute_opts=execute_opts, measure_filter=None)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name='vqe_solver')
energy_layout, opt_params_layout = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.684631E+00     2.120353E-01
        2     8    -1.689694E+00     2.218513E-01
        3    14    -1.697248E+00     3.986125E-02
        4    18    -1.704745E+00     1.969790E-01
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.6806716594964146
                Iterations: 4
                Function evaluations: 29
                Gradient evaluations: 4

### Bogota Noise Model With An Improved Qubit Layout and Measurement Filter

```python
%autoreload
execute_opts = {'shots' : 1024,
                'noise_model': bogota_nm,
                'coupling_map':bogota_conf.coupling_map,
                'basis_gates':bogota_conf.basis_gates,
                'initial_layout': [3,2,4,1]
                }
evaluator = BasicEvaluator(varform_4qubits_1param, qasm_simulator,execute_opts=execute_opts, measure_filter=meas_filter)
vqe_solver = VQESolver(evaluator, minimizer, [0,], name='vqe_solver')
energy_layout_meas, opt_params_layout_meas = vqe_solver.lowest_eig_value(lcps_h2)
```

      NIT    FC           OBJFUN            GNORM
        1     3    -1.789338E+00     7.476305E-02
        2     6    -1.821437E+00     1.549079E-01
        3    10    -1.825480E+00     8.058220E-02
    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.8349246159398094
                Iterations: 3
                Function evaluations: 15
                Gradient evaluations: 3

### Analysis

**Running with a Simulated Noise Model and 1024 Shots**

Calculated electronic energies at $0.735 Angstrom$

| Method | Shots | Backend | Noise Model | Measurement Filter | Energy ($E_h$) | % Error |
|------|-------|------|-----|-----|-----|-------|
| Exact | N/A | N/A | N/A | N/A | -1.857275 | 0 |
| VQE | 1024 | Simulator | None | No | -1.86559 | -0.45 |
| VQE | 1024 | Simulator | Bogota | No | -1.66074 | 10.58 |
| VQE (layout optimized) | 1024 | Simulator | Bogota | No | -1.68067 | 9.51 |
| VQE (layout optimized) | 1024 | Simulator | Bogota | Yes | -1.83492 | 1.20 |
| VQE | 1024 | Bogota | N/A | No | -1.61914 | 12.82 |

With no optimizations, the calculated ground state energy is off by approximately 10.6% relative to the "exact" solution. Adding an improved qubit layout, we reduce this error to between 9 and 10%. Applying a measurement filter on top of this, we obtain a final error of about 1.2%.

Measurement errors therefore seem to be the most significant source of error for such a short circuit, especially once the number of required CX gates is reduced to 3. This circuit is short enough that decoherence does not play a very large role.

**Running on Bogota and 1024 Shots**

???

**Next Steps:**
We should examine how this error contributes to the dissociation curve, and whether it significantly alters the predicted equilibrium distance for $H_2$

```python

```

```python

```


```python
from google.colab import drive
drive.mount('/content/drive')
```

    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

    Enter your authorization code:
    ··········
    Mounted at /content/drive

# Neuromatch Academy: Week 2, Day 5, Tutorial 1
# Learning to Predict

__Content creators:__ Marcelo Mattar and Eric DeWitt with help from Matt Krause

__Content reviewers:__ Byron Galbraith and Michael Waskom

---

# Tutorial objectives

In this tutorial, we will learn how to estimate state-value functions in a classical conditioning paradigm using Temporal Difference (TD) learning and examine TD-errors at the presentation of the conditioned and unconditioned stimulus (CS and US) under different CS-US contingencies. These exercises will provide you with an understanding of both how reward prediction errors (RPEs) behave in classical conditioning and what we should expect to see if Dopamine represents a "canonical" model-free RPE.

At the end of this tutorial:
* You will learn to use the standard tapped delay line conditioning model
* You will understand how RPEs move to CS
* You will understand how variability in reward size effects RPEs
* You will understand how differences in US-CS timing effect RPEs

```python
# Imports
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
# @title Helper functions
from matplotlib import ticker

def plot_value_function(V, ax=None, show=True):
  """Plot V(s), the value function"""
  if not ax:
    fig, ax = plt.subplots()

  ax.stem(V, use_line_collection=True)
  ax.set_ylabel('Value')
  ax.set_xlabel('State')
  ax.set_title("Value function: $V(s)$")

  if show:
    plt.show()

def plot_tde_trace(TDE, ax=None, show=True, skip=400):
  """Plot the TD Error across trials"""
  if not ax:
    fig, ax = plt.subplots()

  indx = np.arange(0, TDE.shape[1], skip)
  im = ax.imshow(TDE[:,indx])
  positions = ax.get_xticks()
  # Avoid warning when setting string tick labels
  ax.xaxis.set_major_locator(ticker.FixedLocator(positions))
  ax.set_xticklabels([f"{int(skip * x)}" for x in positions])
  ax.set_title('TD-error over learning')
  ax.set_ylabel('State')
  ax.set_xlabel('Iterations')
  ax.figure.colorbar(im)
  if show:
    plt.show()

def learning_summary_plot(V, TDE):
  """Summary plot for Ex1"""
  fig, (ax1, ax2) = plt.subplots(nrows = 2, gridspec_kw={'height_ratios': [1, 2]})

  plot_value_function(V, ax=ax1, show=False)
  plot_tde_trace(TDE, ax=ax2, show=False)
  plt.tight_layout()

def reward_guesser_title_hint(r1, r2):
  """"Provide a mildly obfuscated hint for a demo."""
  if (r1==14 and r2==6) or (r1==6 and r2==14):
    return "Technically correct...(the best kind of correct)"

  if  ~(~(r1+r2) ^ 11) - 1 == (6 | 24): # Don't spoil the fun :-)
    return "Congratulations! You solved it!"

  return "Keep trying...."

#@title Default title text
class ClassicalConditioning:

    def __init__(self, n_steps, reward_magnitude, reward_time):

        # Task variables
        self.n_steps = n_steps
        self.n_actions = 0
        self.cs_time = int(n_steps/4) - 1

        # Reward variables
        self.reward_state = [0,0]
        self.reward_magnitude = None
        self.reward_probability = None
        self.reward_time = None

        self.set_reward(reward_magnitude, reward_time)

        # Time step at which the conditioned stimulus is presented

        # Create a state dictionary
        self._create_state_dictionary()

    def set_reward(self, reward_magnitude, reward_time):

        """
        Determine reward state and magnitude of reward
        """
        if reward_time >= self.n_steps - self.cs_time:
            self.reward_magnitude = 0

        else:
            self.reward_magnitude = reward_magnitude
            self.reward_state = [1, reward_time]

    def get_outcome(self, current_state):

        """
        Determine next state and reward
        """
        # Update state
        if current_state < self.n_steps - 1:
            next_state = current_state + 1
        else:
            next_state = 0

        # Check for reward
        if self.reward_state == self.state_dict[current_state]:
            reward = self.reward_magnitude
        else:
            reward = 0

        return next_state, reward

    def _create_state_dictionary(self):

        """
        This dictionary maps number of time steps/ state identities
        in each episode to some useful state attributes:

        state      - 0 1 2 3 4 5 (cs) 6 7 8 9 10 11 12 ...
        is_delay   - 0 0 0 0 0 0 (cs) 1 1 1 1  1  1  1 ...
        t_in_delay - 0 0 0 0 0 0 (cs) 1 2 3 4  5  6  7 ...
        """
        d = 0

        self.state_dict = {}
        for s in range(self.n_steps):
            if s <= self.cs_time:
                self.state_dict[s] = [0,0]
            else:
                d += 1 # Time in delay
                self.state_dict[s] = [1,d]

class MultiRewardCC(ClassicalConditioning):
  """Classical conditioning paradigm, except that one randomly selected reward,
    magnitude, from a list, is delivered of a single fixed reward."""
  def __init__(self, n_steps, reward_magnitudes, reward_time=None):
    """"Build a multi-reward classical conditioning environment
      Args:
        - nsteps: Maximum number of steps
        - reward_magnitudes: LIST of possible reward magnitudes.
        - reward_time: Single fixed reward time
      Uses numpy global random state.
      """
    super().__init__(n_steps, 1, reward_time)
    self.reward_magnitudes = reward_magnitudes

  def get_outcome(self, current_state):
    next_state, reward = super().get_outcome(current_state)
    if reward:
      reward=np.random.choice(self.reward_magnitudes)
    return next_state, reward

class ProbabilisticCC(ClassicalConditioning):
  """Classical conditioning paradigm, except that rewards are stochastically omitted."""
  def __init__(self, n_steps, reward_magnitude, reward_time=None, p_reward=0.75):
    """"Build a multi-reward classical conditioning environment
      Args:
        - nsteps: Maximum number of steps
        - reward_magnitudes: Reward magnitudes.
        - reward_time: Single fixed reward time.
        - p_reward: probability that reward is actually delivered in rewarding state
      Uses numpy global random state.
      """
    super().__init__(n_steps, reward_magnitude, reward_time)
    self.p_reward = p_reward

  def get_outcome(self, current_state):
    next_state, reward = super().get_outcome(current_state)
    if reward:
      reward*= int(np.random.uniform(size=1)[0] < self.p_reward)
    return next_state, reward

```

---
# Section 1: TD-learning

```python
#@title Video 1: Introduction
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="YoNbc9M92YY", width=854, height=480, fs=1)
print("Video available at https://youtu.be/" + video.id)
video
```

    Video available at https://youtu.be/YoNbc9M92YY

__Environment:__

- The agent experiences the environment in episodes or trials.
- Episodes terminate by transitioning to the inter-trial-interval (ITI) state and they are initiated from the ITI state as well. We clamp the value of the terminal/ITI states to zero.
- The classical conditioning environment is composed of a sequence of states that the agent deterministically transitions through. Starting at State 0, the agent moves to State 1 in the first step, from State 1 to State 2 in the second, and so on.  These states represent time in the tapped delay line representation
- Within each episode, the agent is presented a CS and US (reward).
- The CS is always presented at 1/4 of the total duration of the trial. The US (reward) is then delivered after the CS. The interval between the CS and US is specified by `reward_time`.
- The agent's goal is to learn to predict expected rewards from each state in the trial.

**General concepts**

* Return $G_{t}$: future cumulative reward, which can be written in arecursive form
\begin{align}
G_{t} &= \sum \limits_{k = 0}^{\infty} \gamma^{k} r_{t+k+1} \\
&= r_{t+1} + \gamma G_{t+1}
\end{align}
where $\gamma$ is discount factor that controls the importance of future rewards, and $\gamma \in [0, 1]$. $\gamma$ may also be interpreted as probability of continuing the trajectory.
* Value funtion $V_{\pi}(s_t=s)$: expecation of the return
\begin{align}
V_{\pi}(s_t=s) &= \mathbb{E} [ G_{t}\; | \; s_t=s, a_{t:\infty}\sim\pi] \\
& = \mathbb{E} [ r_{t+1} + \gamma G_{t+1}\; | \; s_t=s, a_{t:\infty}\sim\pi]
\end{align}
With an assumption of **Markov process**, we thus have:
\begin{align}
V_{\pi}(s_t=s) &= \mathbb{E} [ r_{t+1} + \gamma V_{\pi}(s_{t+1})\; | \; s_t=s, a_{t:\infty}\sim\pi] \\
&= \sum_a \pi(a|s) \sum_{r, s'}p(s', r)(r + V_{\pi}(s_{t+1}=s'))
\end{align}

**Temporal difference (TD) learning**

* With a Markovian assumption, we can use $V(s_{t+1})$ as an imperfect proxy for the true value $G_{t+1}$ (Monte Carlo bootstrapping), and thus obtain the generalised equation to calculate TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

* Value updated by using the learning rate constant $\alpha$:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

  (Reference: https://web.stanford.edu/group/pdplab/pdphandbook/handbookch10.html)

__Definitions:__

* TD-error:
\begin{align}
\delta_{t} = r_{t+1} + \gamma V(s_{t+1}) - V(s_{t})
\end{align}

* Value updates:
\begin{align}
V(s_{t}) \leftarrow V(s_{t}) + \alpha \delta_{t}
\end{align}

## Exercise 1: TD-learning with guaranteed rewards

Implement TD-learning to estimate the state-value function in the classical-conditioning world with guaranteed rewards, with a fixed magnitude, at a fixed delay after the conditioned stimulus, CS. Save TD-errors over learning (i.e., over trials) so we can visualize them afterwards.

In order to simulate the effect of the CS, you should only update $V(s_{t})$ during the delay period after CS. This period is indicated by the boolean variable `is_delay`. This can be implemented by multiplying the expression for updating the value function by `is_delay`.

Use the provided code to estimate the value function.

```python
def td_learner(env, n_trials, gamma=0.98, alpha=0.001):
  """ Temporal Difference learning

  Args:
    env (object): the environment to be learned
    n_trials (int): the number of trials to run
    gamma (float): temporal discount factor
    alpha (float): learning rate

  Returns:
    ndarray, ndarray: the value function and temporal difference error arrays
  """
  V = np.zeros(env.n_steps) # Array to store values over states (time)
  TDE = np.zeros((env.n_steps, n_trials)) # Array to store TD errors

  for n in range(n_trials):
    state = 0 # Initial state
    for t in range(env.n_steps):
      # Get next state and next reward
      next_state, reward = env.get_outcome(state)
      # Is the current state in the delay period (after CS)?
      is_delay = env.state_dict[state][0]

      ########################################################################
      ## TODO for students: implement TD error and value function update
      # Fill out function and remove
      #raise NotImplementedError("Student excercise: implement TD error and value function update")
      #################################################################################
      # Write an expression to compute the TD-error
      TDE[state, n] = (reward + gamma * V[next_state] - V[state])

      # Write an expression to update the value function
      V[state] += alpha * TDE[state, n] * is_delay

      # Update state
      state = next_state

  return V, TDE

# Uncomment once the td_learner function is complete
env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
V, TDE = td_learner(env, n_trials=20000)
learning_summary_plot(V, TDE)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_0d9c50de.py)

*Example output:*

## Interactive Demo 1: US to CS Transfer

During classical conditioning, the subject's behavioral response (e.g., salivating) transfers from the unconditioned stimulus (US; like the smell of tasty food) to the conditioned stimulus (CS; like Pavlov ringing his bell) that predicts it. Reward prediction errors play an important role in this process by adjusting the value of states according to their expected, discounted return.

Use the widget below to examine how reward prediction errors change over time. Before training (orange line), only the reward state has high reward prediction error. As training progresses (blue line, slider), the reward prediction errors shift to the conditioned stimulus, where they end up when the trial is complete (green line).

Dopamine neurons, which are thought to carry reward prediction errors _in vivo_, show exactly the same behavior!

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

n_trials = 20000

@widgets.interact
def plot_tde_by_trial(trial = widgets.IntSlider(value=5000, min=0, max=n_trials-1 , step=1, description="Trial #")):
  if 'TDE' not in globals():
    print("Complete Exercise 1 to enable this interactive demo!")
  else:

    fig, ax = plt.subplots()
    ax.axhline(0, color='k') # Use this + basefmt=' ' to keep the legend clean.
    ax.stem(TDE[:, 0], linefmt='C1-', markerfmt='C1d', basefmt=' ',
            label="Before Learning (Trial 0)",
            use_line_collection=True)
    ax.stem(TDE[:, -1], linefmt='C2-', markerfmt='C2s', basefmt=' ',
            label="After Learning (Trial $\infty$)",
            use_line_collection=True)
    ax.stem(TDE[:, trial], linefmt='C0-', markerfmt='C0o', basefmt=' ',
            label=f"Trial {trial}",
            use_line_collection=True)

    ax.set_xlabel("State in trial")
    ax.set_ylabel("TD Error")
    ax.set_title("Temporal Difference Error by Trial")
    ax.legend()
```

    interactive(children=(IntSlider(value=5000, description='Trial #', max=19999), Output()), _dom_classes=('widge…

## Interactive Demo 2: Learning Rates and Discount Factors

Our TD-learning agent has two parameters that control how it learns: $\alpha$, the learning rate, and $\gamma$, the discount factor. In Exercise 1, we set these parameters to $\alpha=0.001$ and $\gamma=0.98$ for you. Here, you'll investigate how changing these parameters alters the model that TD-learning learns.

Before enabling the interactive demo below, take a moment to think about the functions of these two parameters. $\alpha$ controls the size of the Value function updates produced by each TD-error. In our simple, deterministic world, will this affect the final model we learn? Is a larger $\alpha$ necessarily better in more complex, realistic environments?

The discount rate $\gamma$ applies an exponentially-decaying weight to returns occuring in the future, rather than the present timestep. How does this affect the model we learn? What happens when $\gamma=0$ or $\gamma \geq 1$?

Use the widget to test your hypotheses.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact
def plot_summary_alpha_gamma(alpha = widgets.FloatSlider(value=0.0001, min=0.001, max=0.1, step=0.0001, description="alpha"),
                             gamma = widgets.FloatSlider(value=0.980, min=0, max=1.1, step=0.010, description="gamma")):
  env = ClassicalConditioning(n_steps=40, reward_magnitude=10, reward_time=10)
  try:
    V_params, TDE_params = td_learner(env, n_trials=20000, gamma=gamma, alpha=alpha)
  except NotImplementedError:
    print("Finish Exercise 1 to enable this interactive demo")

  learning_summary_plot(V_params,TDE_params)

```

    interactive(children=(FloatSlider(value=0.001, description='alpha', max=0.1, min=0.001, step=0.0001), FloatSli…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_12ce49be.py)

---
# Section 2: TD-learning with varying reward magnitudes

In the previous exercise, the environment was as simple as possible. On every trial, the CS predicted the same reward, at the same time, with 100% certainty. In the next few exercises, we will make the environment more progressively more complicated and examine the TD-learner's behavior.

## Interactive Demo 3: Match the Value Functions

First, will replace the environment with one that dispenses one of several rewards, chosen at random. Shown below is the final value function $V$ for a TD learner that was trained in an enviroment where the CS predicted a reward of 6 or 14 units; both rewards were equally likely).

Can you find another pair of rewards that cause the agent to learn the same value function? Assume each reward will be dispensed 50% of the time.

Hints:
* Carefully consider the definition of the value function $V$. This can be solved analytically.
* There is no need to change $\alpha$ or $\gamma$.
* Due to the randomness, there may be a small amount of variation.

The average is 10

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

n_trials = 20000
np.random.seed(2020)
rng_state = np.random.get_state()
env = MultiRewardCC(40, [6, 14], reward_time=10)
V_multi, TDE_multi = td_learner(env, n_trials, gamma=0.98, alpha=0.001)

@widgets.interact
def reward_guesser_interaction(r1 = widgets.IntText(value=0, min=0, max=50, description="Reward 1"),
                               r2 = widgets.IntText(value=0, min=0, max=50, description="Reward 2")):
  try:
    env2 = MultiRewardCC(40, [r1, r2], reward_time=10)
    V_guess, _ = td_learner(env2, n_trials, gamma=0.98, alpha=0.001)
    fig, ax = plt.subplots()
    m, l, _ = ax.stem(V_multi, linefmt='y-', markerfmt='yo', basefmt=' ', label="Target",
            use_line_collection=True)
    m.set_markersize(15)
    m.set_markerfacecolor('none')
    l.set_linewidth(4)
    m, _, _ = ax.stem(V_guess, linefmt='r', markerfmt='rx', basefmt=' ', label="Guess",
                      use_line_collection=True)
    m.set_markersize(15)

    ax.set_xlabel("State")
    ax.set_ylabel("Value")
    ax.set_title("Guess V(s)\n" + reward_guesser_title_hint(r1, r2))
    ax.legend()
  except NotImplementedError:
    print("Please finish Exercise 1 first!")
```

    interactive(children=(IntText(value=0, description='Reward 1'), IntText(value=0, description='Reward 2'), Outp…

## Section 2.1 Examining the TD Error

Run the cell below to plot the TD errors from our multi-reward environment. A new feature appears in this plot? What is it? Why does it happen?

```python
plot_tde_trace(TDE_multi)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_dea47c05.py)

---
# Section 3: TD-learning with probabilistic rewards

In this environment, we'll return to delivering a single reward of ten units. However, it will be delivered intermittently: on 20 percent of trials, the CS will be shown but the agent will not receive the usual reward; the remaining 80% will proceed as usual.

 Run the cell below to simulate. How does this compare with the previous experiment?

Earlier in the notebook, we saw that changing $\alpha$ had little effect on learning in a deterministic environment. What happens if you set it to an large value, like 1, in this noisier scenario? Does it seem like it will _ever_ converge?

```python
np.random.set_state(rng_state) # Resynchronize everyone's notebooks
n_trials = 20000
try:
  env = ProbabilisticCC(n_steps=40, reward_magnitude=10, reward_time=10,
                        p_reward=0.8)
  V_stochastic, TDE_stochastic = td_learner(env, n_trials*2, alpha=.001)
  learning_summary_plot(V_stochastic, TDE_stochastic)
except NotImplementedError:
  print("Please finish Exercise 1 first")
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_cbbb9c00.py)

---
# Summary

In this notebook, we have developed a simple TD Learner and examined how its state representations and reward prediction errors evolve during training. By manipualting its environment and parameters ($\alpha$, $\gamma$), you developed an intuition for how it behaves.

This simple model closely resembles the behavior of subjects undergoing classical conditioning tasks and the dopamine neurons that may underlie that behavior. You may have implemented TD-reset or used the model to recreate a common experimental error. The update rule used here has been extensively studied for [more than 70 years](https://www.pnas.org/content/108/Supplement_3/15647) as a possible explanation for artificial and biological learning.

However, you may have noticed that something is missing from this notebook. We carefully calculated the value of each state, but did not use it to actually do anything. Using values to plan _**Actions**_ is coming up next!

# Bonus

## Exercise 2: Removing the CS

In Exercise 1, you (should have) included a term that depends on the conditioned stimulus. Remove it and see what happens. Do you understand why?
This phenomena often fools people attempting to train animals--beware!

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W2D5_ReinforcementLearning/solutions/W2D5_Tutorial1_Solution_a35b23f3.py)


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)

# Hide the code completely

# from IPython.display import HTML
# tag = HTML('''<style>
# div.input {
#     display:none;
# }
# </style>''')
# display(tag)
```

Toggle cell visibility here.

##  Regulator Design: observer and state feedback

This example demonstrates the design of a regulator: a type of controller composed by an observer and a full-state feedback controller. It can be proven that, thanks to the separation principle, the controller and the observer can be designed independently - closed-loop eigenvalues and observer dynamics can be set separately without affecting each other.

Nonetheless, closed-loop transient performance depends on how fast the observer is with respect to the desired closed-loop dynamics.

This example shows the design of a regulator for the controllable and observable system:

\begin{cases}
\dot{\textbf{x}}=\begin{bmatrix}1&0&3\\0&-4&-1\\0&1&-4\end{bmatrix}\textbf{x}+\begin{bmatrix}0\\0\\1\end{bmatrix}\textbf{u} \\ \\
\textbf{y}=\begin{bmatrix}1&0&0\end{bmatrix}\textbf{x}
\end{cases}

that has the transfer function:

$$
G(s) = C(sI-A)^{-1}B.
$$

### Development of the state feedback
The goal is to place 3 eigenvalues in $-1$ rad/s or faster in order to have a good transient response. A possible solution is: $K = \begin{bmatrix}\frac{8}{15}&-4.4&-4\end{bmatrix}$.

### Development of the observer
Given the eigenvalues of the controlled system, a better (faster and stable) choice for the observer is $\lambda_{1,2,3} = -10$ rad/s. This can be achieved with $L=\begin{bmatrix}23&66&\frac{107}{3}\end{bmatrix}^T$.

### Composition of the regulator

The regulator can be implemented in two ways, as a transfer function:

where:

$$
K(s) = -K(sI-A+LC+BK)^{-1}L\,.
$$

or as an observer with static feedback:

Recall that although the obtained closed dynamics is the same, the transfer function implementation may lead to an unstable controller - the eigenvalues of matrix (matrix $A-BK-LC$) may be unstable even if closed-loop dynamics is stable.

### How to use this notebook?
- Try to change the initial conditions of the estimator (default is $\begin{bmatrix}0.2&0.2&0.2\end{bmatrix}^T$) and the observer's eigenvalues and see how the controlled system behaviour changes.
- Try to change the values in order to achieve settling time for 5% tolerance band of less than 2 s.

**Note:**

- The ideal values refer to the case in which all the states of the system can be measured.
- The **Inverse reference gain** slider denotes the value for which the reference is divided (as you change the static gain of the closed-loop transfer function); in order to reach zero error for the step response its value is equal to the static gain.

```python
%matplotlib inline
import control as control
import numpy
import sympy as sym
from IPython.display import display, Markdown
import ipywidgets as widgets
import matplotlib.pyplot as plt

#print a matrix latex-like
def bmatrix(a):
     """Returns a LaTeX bmatrix - by Damir Arbula (ICCT project)

     :a: numpy array
     :returns: LaTeX bmatrix as a string
     """
     if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
     lines = str(a).replace('[', '').replace(']', '').splitlines()
     rv = [r'\begin{bmatrix}']
     rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
     rv +=  [r'\end{bmatrix}']
     return '\n'.join(rv)

# Display formatted matrix:
def vmatrix(a):
    if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{vmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{vmatrix}']
    return '\n'.join(rv)

#matrixWidget is a matrix looking widget built with a VBox of HBox(es) that returns a numPy array as value !
class matrixWidget(widgets.VBox):
    def updateM(self,change):
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.M_[irow,icol] = self.children[irow].children[icol].value
                #print(self.M_[irow,icol])
        self.value = self.M_

    def dummychangecallback(self,change):
        pass

    def __init__(self,n,m):
        self.n = n
        self.m = m
        self.M_ = numpy.matrix(numpy.zeros((self.n,self.m)))
        self.value = self.M_
        widgets.VBox.__init__(self,
                             children = [
                                 widgets.HBox(children =
                                              [widgets.FloatText(value=0.0, layout=widgets.Layout(width='90px')) for i in range(m)]
                                             )
                                 for j in range(n)
                             ])

        #fill in widgets and tell interact to call updateM each time a children changes value
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
                self.children[irow].children[icol].observe(self.updateM, names='value')
        #value = Unicode('example@example.com', help="The email value.").tag(sync=True)
        self.observe(self.updateM, names='value', type= 'All')

    def setM(self, newM):
        #disable callbacks, change values, and reenable
        self.unobserve(self.updateM, names='value', type= 'All')
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].unobserve(self.updateM, names='value')
        self.M_ = newM
        self.value = self.M_
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].observe(self.updateM, names='value')
        self.observe(self.updateM, names='value', type= 'All')

                #self.children[irow].children[icol].observe(self.updateM, names='value')

#overlaod class for state space systems that DO NOT remove "useless" states (what "professor" of automatic control would do this?)
class sss(control.StateSpace):
    def __init__(self,*args):
        #call base class init constructor
        control.StateSpace.__init__(self,*args)
    #disable function below in base class
    def _remove_useless_states(self):
        pass
```

```python
# Preparatory cell

A = numpy.matrix('1 0 3; 0 -4 -1; 0 1 -4')
B = numpy.matrix('0; 0; 1')
C = numpy.matrix('1 0 0')
X0 = numpy.matrix('0.2; 0.2; 0.2')
K = numpy.matrix([8/15,-4.4,-4])
L = numpy.matrix([[23],[66],[107/3]])

Aw = matrixWidget(3,3)
Aw.setM(A)
Bw = matrixWidget(3,1)
Bw.setM(B)
Cw = matrixWidget(1,3)
Cw.setM(C)
X0w = matrixWidget(3,1)
X0w.setM(X0)
Kw = matrixWidget(1,3)
Kw.setM(K)
Lw = matrixWidget(3,1)
Lw.setM(L)

eig1c = matrixWidget(1,1)
eig2c = matrixWidget(2,1)
eig3c = matrixWidget(1,1)
eig1c.setM(numpy.matrix([-1.]))
eig2c.setM(numpy.matrix([[-1.],[0.]]))
eig3c.setM(numpy.matrix([-1.]))

eig1o = matrixWidget(1,1)
eig2o = matrixWidget(2,1)
eig3o = matrixWidget(1,1)
eig1o.setM(numpy.matrix([-10.]))
eig2o.setM(numpy.matrix([[-10.],[0.]]))
eig3o.setM(numpy.matrix([-10.]))
```

```python
# Misc

#create dummy widget
DW = widgets.FloatText(layout=widgets.Layout(width='0px', height='0px'))

#create button widget
START = widgets.Button(
    description='Test',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Test',
    icon='check'
)

def on_start_button_clicked(b):
    #This is a workaround to have intreactive_output call the callback:
    #   force the value of the dummy widget to change
    if DW.value> 0 :
        DW.value = -1
    else:
        DW.value = 1
    pass
START.on_click(on_start_button_clicked)

# Define type of method
selm = widgets.Dropdown(
    options= ['Set K and L', 'Set the eigenvalues'],
    value= 'Set the eigenvalues',
    description='',
    disabled=False
)

# Define the number of complex eigenvalues
sele = widgets.Dropdown(
    options= ['0 complex eigenvalues', '2 complex eigenvalues'],
    value= '0 complex eigenvalues',
    description='Complex eigenvalues:',
    style = {'description_width': 'initial'},
    disabled=False
)

#define type of ipout
selu = widgets.Dropdown(
    options=['impulse', 'step', 'sinusoid', 'square wave'],
    value='step',
    description='Type of reference:',
    style = {'description_width': 'initial'},
    disabled=False
)
# Define the values of the input
u = widgets.FloatSlider(
    value=1,
    min=0,
    max=20.0,
    step=0.1,
    description='Reference:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.1f',
)
period = widgets.FloatSlider(
    value=0.5,
    min=0.01,
    max=4,
    step=0.01,
    description='Period: ',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.2f',
)

gain_w = widgets.FloatText(
    value=1.,
    description='',
    disabled=True
)

gain_id_w = widgets.FloatText(
    value=1.,
    description='',
    disabled=True
)

gain_w2 = widgets.FloatText(
    value=1.,
    description='',
    disabled=True
)

gain_id_w2 = widgets.FloatText(
    value=1.,
    description='',
    disabled=True
)
```

```python
# Support functions

def eigen_choice(sele):
    if sele == '0 complex eigenvalues':
        eig1c.children[0].children[0].disabled = False
        eig2c.children[1].children[0].disabled = True
        eig1o.children[0].children[0].disabled = False
        eig2o.children[1].children[0].disabled = True
        eig = 0
    if sele == '2 complex eigenvalues':
        eig1c.children[0].children[0].disabled = True
        eig2c.children[1].children[0].disabled = False
        eig1o.children[0].children[0].disabled = True
        eig2o.children[1].children[0].disabled = False
        eig = 2
    return eig

def method_choice(selm):
    if selm == 'Set K and L':
        method = 1
        sele.disabled = True
    if selm == 'Set the eigenvalues':
        method = 2
        sele.disabled = False
    return method
```

## Implementation as transfer function controller

```python
sols = numpy.linalg.eig(A)

def main_callback(Aw, Bw, X0w, K, L, eig1c, eig2c, eig3c, eig1o, eig2o, eig3o, u, period, selm, sele, selu, DW):
    eige = eigen_choice(sele)
    method = method_choice(selm)

    if method == 1:
        solc = numpy.linalg.eig(A-B*K)
        solo = numpy.linalg.eig(A-L*C)
    if method == 2:
        if eige == 0:
            K = control.acker(A, B, [eig1c[0,0], eig2c[0,0], eig3c[0,0]])
            Kw.setM(K)
            L = control.acker(A.T, C.T, [eig1o[0,0], eig2o[0,0], eig3o[0,0]]).T
            Lw.setM(L)
        if eige == 2:
            K = control.acker(A, B, [eig3c[0,0],
                                     numpy.complex(eig2c[0,0],eig2c[1,0]),
                                     numpy.complex(eig2c[0,0],-eig2c[1,0])])
            Kw.setM(K)
            L = control.acker(A.T, C.T, [eig3o[0,0],
                                         numpy.complex(eig2o[0,0],eig2o[1,0]),
                                         numpy.complex(eig2o[0,0],-eig2o[1,0])]).T
            Lw.setM(L)

    Gs = sss(A,B,C,0)
    Ks = sss(A-B*K-L*C,L,-K,0)
    Fs = control.series(-Ks,Gs)
    sys = control.feedback(Fs)

    Gs_id = sss(A,B,sym.eye(3),sym.zeros(3,1))
    Fs_id = control.series(K,Gs_id)
    A1 = numpy.matrix(Fs_id.A-Fs_id.B*Fs_id.C)
    B1 = numpy.matrix(Fs_id.B*sym.Matrix([[1],[0],[0]]))
    C1 = numpy.matrix(sym.Matrix([1,0,0]).T*Fs_id.C)
    sys_id = sss(A1,B1,C1,0)

    sys_o = sss(A-L*C,numpy.hstack((L,B)),sym.eye(3),sym.zeros(3,2))

    dcgain = control.dcgain(sys)
    t = numpy.linspace(0, 1000, 2)
    t, y = control.step_response(sys_id,t)
    dcgain_id = y[-1]
    gain_w.value = dcgain
    gain_id_w.value = dcgain_id
    if dcgain != 0 and dcgain_id != 0:
        u1 = u/gain_w.value
        u2 = u/gain_id_w.value
    else:
        print('The inverse gain set is 0 and it is changed to 1')
        u1 = u/1
        u2 = u/1

    solc = numpy.linalg.eig(sys.A)
    solo = numpy.linalg.eig(A-L*C-B*K)
    print('The system\'s eigenvalues are:', round(sols[0][0],2),',', round(sols[0][1],2),'and', round(sols[0][2],2))
    print('The controlled closed loop system\'s eigenvalues are:',
          round(solc[0][0],2),',',
          round(solc[0][1],2),',',
          round(solc[0][2],2),',',
          round(solc[0][3],2),',',
          round(solc[0][4],2),'and',
          round(solc[0][5],2))
    print('The controller\'s eigenvalues are:', round(solo[0][0],2),',', round(solo[0][1],2),'and', round(solo[0][2],2))
    print('')
    print('The static gain of the closed loop system (from the reference to the output) is: %.5f' %dcgain)
    print('The static gain of the closed loop ideal system (from the reference to the output) is: %.5f' %dcgain_id)

    X0w1 = numpy.matrix([[X0w[0,0]],[X0w[1,0]],[X0w[2,0]],[0],[0],[0]])
    T = numpy.linspace(0, 12, 1000)

    if selu == 'impulse': #selu
        U = [0 for t in range(0,len(T))]
        U[0] = u
        U1 = [0 for t in range(0,len(T))]
        U1[0] = u1
        U2 = [0 for t in range(0,len(T))]
        U2[0] = u2
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
        T, yout_k, xout_k = control.forced_response(Ks,T,yout-U1,X0w)
        T, yout_o, xout_o = control.forced_response(sys_o,T,[yout,yout_k],X0w)
    if selu == 'step':
        U = [u for t in range(0,len(T))]
        U1 = [u1 for t in range(0,len(T))]
        U2 = [u2 for t in range(0,len(T))]
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
        T, yout_k, xout_k = control.forced_response(Ks,T,yout-U1,X0w)
        T, yout_o, xout_o = control.forced_response(sys_o,T,[yout,yout_k],X0w)
    if selu == 'sinusoid':
        U = u*numpy.sin(2*numpy.pi/period*T)
        U1 = u1*numpy.sin(2*numpy.pi/period*T)
        U2 = u2*numpy.sin(2*numpy.pi/period*T)
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
        T, yout_k, xout_k = control.forced_response(Ks,T,yout-U1,X0w)
        T, yout_o, xout_o = control.forced_response(sys_o,T,[yout,yout_k],X0w)
    if selu == 'square wave':
        U = u*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        U1 = u1*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        U2 = u2*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
        T, yout_k, xout_k = control.forced_response(Ks,T,yout-U1,X0w)
        T, yout_o, xout_o = control.forced_response(sys_o,T,[yout,yout_k],X0w)
    # N.B. i primi 3 stati di xout sono quelli dello stimatore, mentre gli ultimi 3 sono quelli del sistema "reale"

    fig = plt.figure(num='Simulation1', figsize=(16,17))
    mag, phase, omega = control.bode_plot(sys,Plot = False)
    mag = control.mag2db(mag)
    phase = phase*180/numpy.pi
    fig.add_subplot(321)
    plt.title('Bode plot: magnitude')
    plt.semilogx(omega,mag)
    plt.xlabel('$\omega$ [rad/s]')
    plt.ylabel('Mag. [dB]')
    plt.grid(True,which="both")

    fig.add_subplot(323)
    plt.title('Bode plot: phase')
    plt.semilogx(omega,phase)
    plt.xlabel('$\omega$ [rad/s]')
    plt.ylabel('Phase [deg]')
    plt.grid(True,which="both")

    fig.add_subplot(325)
    plt.title('Output response')
    plt.ylabel('Output')
    plt.plot(T,yout,T,yout_id,'g',T,U,'r--')
    plt.xlabel('$t$ [s]')
    plt.legend(['$y$','$y_{ideal}$','Reference'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(322)
    plt.title('First state response')
    plt.ylabel('$x_1$')
    plt.plot(T,xout_o[0],T,xout[3],T,xout_id[0],'g')
    plt.xlabel('$t$ [s]')
    plt.legend(['$x_{1est}$','$x_{1real}$','$x_{1ideal}$'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(324)
    plt.title('Second state response')
    plt.ylabel('$x_2$')
    plt.plot(T,xout_o[1],T,xout[4],T,xout_id[1],'g')
    plt.xlabel('$t$ [s]')
    plt.legend(['$x_{2est}$','$x_{2real}$','$x_{2ideal}$'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(326)
    plt.title('Third state response')
    plt.ylabel('$x_3$')
    plt.plot(T,xout_o[2],T,xout[5],T,xout_id[2],'g')
    plt.xlabel('$t$ [s]')
    plt.legend(['$x_{3est}$','$x_{3real}$','$x_{3ideal}$'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

alltogether = widgets.VBox([widgets.HBox([selm,
                                          sele,
                                          selu]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.Label('K:',border=3), Kw,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Eigenvalues:',border=3),
                                          eig1c,
                                          eig2c,
                                          eig3c,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('X0 est.:',border=3), X0w]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.Label('L:',border=3), Lw,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Eigenvalues:',border=3),
                                          eig1o,
                                          eig2o,
                                          eig3o,
                                          widgets.Label(' ',border=3),
                                          widgets.VBox([widgets.Label('Inverse reference gain:',border=3),
                                                        widgets.Label('Inverse ideal reference gain:',border=3)]),
                                          widgets.VBox([gain_w,gain_id_w])]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([u,
                                          period,
                                          START])])
out = widgets.interactive_output(main_callback, {'Aw':Aw, 'Bw':Bw, 'X0w':X0w, 'K':Kw, 'L':Lw,
                                                 'eig1c':eig1c, 'eig2c':eig2c, 'eig3c':eig3c, 'eig1o':eig1o, 'eig2o':eig2o, 'eig3o':eig3o,
                                                 'u':u, 'period':period, 'selm':selm, 'sele':sele, 'selu':selu, 'DW':DW})
out.layout.height = '1120px'
display(out, alltogether)
```

    Output(layout=Layout(height='1120px'))

    VBox(children=(HBox(children=(Dropdown(index=1, options=('Set K and L', 'Set the eigenvalues'), value='Set the…

## Implementation as observer

```python
sols = numpy.linalg.eig(A)

def main_callback2(Aw, Bw, X0w, K, L, eig1c, eig2c, eig3c, eig1o, eig2o, eig3o, u, period, selm, sele, selu, DW):
    eige = eigen_choice(sele)
    method = method_choice(selm)

    if method == 1:
        solc = numpy.linalg.eig(A-B*K)
        solo = numpy.linalg.eig(A-L*C)
    if method == 2:
        if eige == 0:
            K = control.acker(A, B, [eig1c[0,0], eig2c[0,0], eig3c[0,0]])
            Kw.setM(K)
            L = control.acker(A.T, C.T, [eig1o[0,0], eig2o[0,0], eig3o[0,0]]).T
            Lw.setM(L)
        if eige == 2:
            K = control.acker(A, B, [eig3c[0,0],
                                     numpy.complex(eig2c[0,0],eig2c[1,0]),
                                     numpy.complex(eig2c[0,0],-eig2c[1,0])])
            Kw.setM(K)
            L = control.acker(A.T, C.T, [eig3o[0,0],
                                         numpy.complex(eig2o[0,0],eig2o[1,0]),
                                         numpy.complex(eig2o[0,0],-eig2o[1,0])]).T
            Lw.setM(L)

    Gs = sss(A,B,numpy.vstack((numpy.eye(3),[0,0,0])),[[0],[0],[0],[1]])
    Os = sss(A-L*C,numpy.hstack((L,B)),numpy.vstack((-K,numpy.eye(3))),[[0,0],[0,0],[0,0],[0,0]])
    Gas = control.append(Gs,Os)
    sys = control.connect(Gas,[[2,1],[3,4],[1,5]],[1],[1,2,3,6,7,8])

    Gs_id = sss(A,B,sym.eye(3),sym.zeros(3,1))
    Fs_id = control.series(K,Gs_id)
    A1 = numpy.matrix(Fs_id.A-Fs_id.B*Fs_id.C)
    B1 = numpy.matrix(Fs_id.B*sym.Matrix([[1],[0],[0]]))
    C1 = numpy.matrix(sym.Matrix([1,0,0]).T*Fs_id.C)
    sys_id = sss(A1,B1,C1,0)

    dcgain = control.dcgain(sys[0,0])
    t = numpy.linspace(0, 1000, 2)
    t, y = control.step_response(sys_id,t)
    dcgain_id = y[-1]
    gain_w2.value = dcgain
    gain_id_w2.value = dcgain_id
    if dcgain != 0 and dcgain_id != 0:
        u1 = u/gain_w2.value
        u2 = u/gain_id_w2.value
    else:
        print('The inverse gain setted is 0 and it is changed to 1')
        u1 = u/1
        u2 = u/1

    solc = numpy.linalg.eig(sys.A)
    print('The system\'s eigenvalues are:', round(sols[0][0],2),',', round(sols[0][1],2),'and', round(sols[0][2],2))
    print('The controlled closed loop system\'s eigenvalues are:',
          round(solc[0][0],2),',',
          round(solc[0][1],2),',',
          round(solc[0][2],2),',',
          round(solc[0][3],2),',',
          round(solc[0][4],2),'and',
          round(solc[0][5],2))
    print('')
    print('The static gain of the closed loop system (from the reference to the output) is: %.5f' %dcgain)
    print('The static gain of the closed loop ideal system (from the reference to the output) is: %.5f' %dcgain_id)

    X0w1 = numpy.matrix([[0],[0],[0],[X0w[0,0]],[X0w[1,0]],[X0w[2,0]]])
    T = numpy.linspace(0, 12, 1000)

    if selu == 'impulse': #selu
        U = [0 for t in range(0,len(T))]
        U[0] = u
        U1 = [0 for t in range(0,len(T))]
        U1[0] = u1
        U2 = [0 for t in range(0,len(T))]
        U2[0] = u2
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
    if selu == 'step':
        U = [u for t in range(0,len(T))]
        U1 = [u1 for t in range(0,len(T))]
        U2 = [u2 for t in range(0,len(T))]
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
    if selu == 'sinusoid':
        U = u*numpy.sin(2*numpy.pi/period*T)
        U1 = u1*numpy.sin(2*numpy.pi/period*T)
        U2 = u2*numpy.sin(2*numpy.pi/period*T)
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
    if selu == 'square wave':
        U = u*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        U1 = u1*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        U2 = u2*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        T, yout, xout = control.forced_response(sys,T,U1,X0w1)
        T, yout_id, xout_id = control.forced_response(sys_id,T,U2,[0, 0, 0])
    # N.B. i primi 3 stati di xout sono quelli del sistema, mentre gli ultimi 3 sono quelli dell'osservatore

    fig = plt.figure(num='Simulation1', figsize=(16,17))
    mag, phase, omega = control.bode_plot(sys[0,0],Plot = False)
    mag = control.mag2db(mag)
    phase = phase*180/numpy.pi
    fig.add_subplot(321)
    plt.title('Bode plot: magnitude')
    plt.semilogx(omega,mag)
    plt.xlabel('$\omega$ [rad/s]')
    plt.ylabel('Mag. [dB]')
    plt.grid(True,which="both")

    fig.add_subplot(323)
    plt.semilogx(omega,phase)
    plt.title('Bode plot: phase')
    plt.xlabel('$\omega$ [rad/s]')
    plt.ylabel('Phase [deg]')
    plt.grid(True,which="both")

    fig.add_subplot(325)
    plt.title('Output response')
    plt.ylabel('Output')
    plt.plot(T,yout[0],T,yout_id,'g',T,U,'r--')
    plt.xlabel('$t$ [s]')
    plt.legend(['$y$','$y_{ideal}$','Reference'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(322)
    plt.title('First state response')
    plt.ylabel('$x_1$')
    plt.plot(T,yout[3],T,yout[0],T,xout_id[0],'g')
    plt.xlabel('$t$ [s]')
    plt.legend(['$x_{1est}$','$x_{1real}$','$x_{1ideal}$'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(324)
    plt.title('Second state response')
    plt.ylabel('$x_2$')
    plt.plot(T,yout[4],T,yout[1],T,xout_id[1],'g')
    plt.xlabel('$t$ [s]')
    plt.legend(['$x_{2est}$','$x_{2real}$','$x_{2ideal}$'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(326)
    plt.title('Third state response')
    plt.ylabel('$x_3$')
    plt.plot(T,yout[5],T,yout[2],T,xout_id[2],'g')
    plt.xlabel('$t$ [s]')
    plt.legend(['$x_{3est}$','$x_{3real}$','$x_{3ideal}$'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

alltogether2 = widgets.VBox([widgets.HBox([selm,
                                          sele,
                                          selu]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.Label('K:',border=3), Kw,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Eigenvalues:',border=3),
                                          eig1c,
                                          eig2c,
                                          eig3c,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('X0 est.:',border=3), X0w]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.Label('L:',border=3), Lw,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Eigenvalues:',border=3),
                                          eig1o,
                                          eig2o,
                                          eig3o,
                                          widgets.Label(' ',border=3),
                                          widgets.VBox([widgets.Label('Inverse reference gain:',border=3),
                                                        widgets.Label('Inverse ideal reference gain:',border=3)]),
                                          widgets.VBox([gain_w2,gain_id_w2])]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([u,
                                          period,
                                          START])])
out2 = widgets.interactive_output(main_callback2, {'Aw':Aw, 'Bw':Bw, 'X0w':X0w, 'K':Kw, 'L':Lw,
                                                 'eig1c':eig1c, 'eig2c':eig2c, 'eig3c':eig3c, 'eig1o':eig1o, 'eig2o':eig2o, 'eig3o':eig3o,
                                                 'u':u, 'period':period, 'selm':selm, 'sele':sele, 'selu':selu, 'DW':DW})
out2.layout.height = '1120px'
display(out2, alltogether2)
```

    Output(layout=Layout(height='1120px'))

    VBox(children=(HBox(children=(Dropdown(index=1, options=('Set K and L', 'Set the eigenvalues'), value='Set the…

```python

```