# Example of DOV search methods for CPT measurements (sonderingen)

## Use cases explained below
* Get CPT measurements in a bounding box
* Get CPT measurements with specific properties
* Get CPT measurements in a bounding box based on specific properties
* Select CPT measurements in a municipality and return depth
* Get CPT measurements based on fields not available in the standard output dataframe
* Get CPT measurements data, returning fields not available in the standard output dataframe
* Get CPT measurements in a municipality and where groundwater related data are available

```python
%matplotlib inline
import inspect, sys
import warnings; warnings.simplefilter('ignore')
```

```python
# check pydov path
import pydov
```

## Get information about the datatype 'Sondering'

```python
from pydov.search.sondering import SonderingSearch
sondering = SonderingSearch()
```

A description is provided for the 'Sondering' datatype:

```python
sondering.get_description()
```

    'In DOV worden de resultaten van sonderingen ter beschikking gesteld. Bij het uitvoeren van de sondering wordt een sondeerpunt met conus bij middel van buizen statisch de grond ingedrukt. Continu of met bepaalde diepte-intervallen wordt de weerstand aan de conuspunt, de plaatselijke wrijvingsweerstand en/of de totale indringingsweerstand opgemeten. Eventueel kan aanvullend de waterspanning in de grond rond de conus tijdens de sondering worden opgemeten met een waterspanningsmeter. Het op diepte drukken van de sondeerbuizen gebeurt met een indrukapparaat. De nodige reactie voor het indrukken van de buizen wordt geleverd door een verankering en/of door het gewicht van de sondeerwagen. De totale indrukcapaciteit varieert van 25 kN tot 250 kN, afhankelijk van apparaat en opstellingswijze.'

The different fields that are available for objects of the 'Sondering' datatype can be requested with the get_fields() method:

```python
fields = sondering.get_fields()

# print available fields
for f in fields.values():
    print(f['name'])
```

    x
    formele_stratigrafie
    gemeente
    u
    diepte_sondering_tot
    pkey_sondering
    weerstandsdiagram
    informele_stratigrafie
    diepte_sondering_van
    id
    i
    diepte_gw_m
    opdrachten
    conus
    qc
    z
    y
    datum_gw_meting
    fs
    sondeermethode
    Qt
    generated_id
    start_sondering_mtaw
    hydrogeologische_stratigrafie
    uitvoerder
    sondeernummer
    datum_aanvang
    apparaat
    meetreeks

You can get more information of a field by requesting it from the fields dictionary:
* *name*: name of the field
* *definition*: definition of this field
* *cost*: currently this is either 1 or 10, depending on the datasource of the field. It is an indication of the expected time it will take to retrieve this field in the output dataframe.
* *notnull*: whether the field is mandatory or not
* *type*: datatype of the values of this field

```python
fields['diepte_sondering_tot']
```

    {'cost': 1,
     'definition': 'Maximumdiepte van de sondering ten opzichte van het aanvangspeil, in meter.',
     'name': 'diepte_sondering_tot',
     'notnull': True,
     'query': True,
     'type': 'float'}

Optionally, if the values of the field have a specific domain the possible values are listed as *values*:

```python
fields['conus']['values']
```

    ['E', 'M1', 'M2', 'M4', 'U', 'onbekend']

## Example use cases

### Get CPT measurements in a bounding box

Get data for all the CPT measurements that are geographically located within the bounds of the specified box.

The coordinates are in the Belgian Lambert72 (EPSG:31370) coordinate system and are given in the order of lower left x, lower left y, upper right x, upper right y.

```python
from pydov.util.location import Within, Box

df = sondering.search(location=Within(Box(152999, 206930, 153050, 207935)))
df.head()
```

    [000/001] c

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      z
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100KN
      NaN
      NaN
      0.2
      1.6
      2.0600
      NaN
      NaN
      NaN

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100KN
      NaN
      NaN
      0.4
      3.6
      4.2600
      NaN
      NaN
      NaN

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100KN
      NaN
      NaN
      0.6
      2.6
      3.4600
      NaN
      NaN
      NaN

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100KN
      NaN
      NaN
      0.8
      4.0
      5.6600
      NaN
      NaN
      NaN

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-72/555-SXVIII
      153008.0
      206985.0
      15.8
      0.0
      36.0
      1973-03-21
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      100KN
      NaN
      NaN
      1.0
      3.0
      6.5300
      NaN
      NaN
      NaN

The dataframe contains one CPT measurement where multiple measurement points. The available data are flattened to represent unique attributes per row of the dataframe.

Using the *pkey_sondering* field one can request the details of this borehole in a webbrowser:

```python
for pkey_sondering in set(df.pkey_sondering):
    print(pkey_sondering)
```

    https://www.dov.vlaanderen.be/data/sondering/1973-016812

### Get CPT measurements with specific properties

Next to querying CPT based on their geographic location within a bounding box, we can also search for CPT measurements matching a specific set of properties. For this we can build a query using a combination of the 'Sondering' fields and operators provided by the WFS protocol.

A list of possible operators can be found below:

```python
[i for i,j in inspect.getmembers(sys.modules['owslib.fes'], inspect.isclass) if 'Property' in i]
```

    ['PropertyIsBetween',
     'PropertyIsEqualTo',
     'PropertyIsGreaterThan',
     'PropertyIsGreaterThanOrEqualTo',
     'PropertyIsLessThan',
     'PropertyIsLessThanOrEqualTo',
     'PropertyIsLike',
     'PropertyIsNotEqualTo',
     'PropertyIsNull',
     'SortProperty']

In this example we build a query using the *PropertyIsEqualTo* operator to find all CPT measuremetns that are within the community (gemeente) of 'Herstappe':

```python
from owslib.fes import PropertyIsEqualTo

query = PropertyIsEqualTo(propertyname='gemeente',
                          literal='Elsene')
df = sondering.search(query=query)

df.head()
```

    [000/029] ccccccccccccccccccccccccccccc

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      z
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25KN
      NaN
      1.97
      1.0
      3.3
      NaN
      NaN
      NaN
      NaN

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25KN
      NaN
      1.97
      1.1
      2.9
      NaN
      NaN
      NaN
      NaN

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25KN
      NaN
      1.97
      1.2
      2.7
      NaN
      NaN
      NaN
      NaN

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25KN
      NaN
      1.97
      1.3
      2.4
      NaN
      NaN
      NaN
      NaN

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-75/194-S1
      150310.0
      169796.0
      56.3
      0.0
      4.5
      1975-05-20
      Rijksinstituut voor Grondmechanica
      discontinu mechanisch
      25KN
      NaN
      1.97
      1.4
      3.6
      NaN
      NaN
      NaN
      NaN

Once again we can use the *pkey_sondering* as a permanent link to the information of these CPT measurements:

```python
for pkey_sondering in set(df.pkey_sondering):
    print(pkey_sondering)
```

    https://www.dov.vlaanderen.be/data/sondering/1992-000337
    https://www.dov.vlaanderen.be/data/sondering/1992-000338
    https://www.dov.vlaanderen.be/data/sondering/1976-014638
    https://www.dov.vlaanderen.be/data/sondering/1992-000335
    https://www.dov.vlaanderen.be/data/sondering/1971-023091
    https://www.dov.vlaanderen.be/data/sondering/1975-014063
    https://www.dov.vlaanderen.be/data/sondering/1976-014640
    https://www.dov.vlaanderen.be/data/sondering/1971-022775
    https://www.dov.vlaanderen.be/data/sondering/1976-013899
    https://www.dov.vlaanderen.be/data/sondering/1971-022776
    https://www.dov.vlaanderen.be/data/sondering/1976-013900
    https://www.dov.vlaanderen.be/data/sondering/1971-023323
    https://www.dov.vlaanderen.be/data/sondering/1971-023321
    https://www.dov.vlaanderen.be/data/sondering/1980-024719
    https://www.dov.vlaanderen.be/data/sondering/1976-030140
    https://www.dov.vlaanderen.be/data/sondering/1971-023320
    https://www.dov.vlaanderen.be/data/sondering/1971-022777
    https://www.dov.vlaanderen.be/data/sondering/1975-014064
    https://www.dov.vlaanderen.be/data/sondering/1971-023322
    https://www.dov.vlaanderen.be/data/sondering/1971-023319
    https://www.dov.vlaanderen.be/data/sondering/1976-030150
    https://www.dov.vlaanderen.be/data/sondering/1976-030128
    https://www.dov.vlaanderen.be/data/sondering/1992-000336
    https://www.dov.vlaanderen.be/data/sondering/1974-016926
    https://www.dov.vlaanderen.be/data/sondering/1976-013898
    https://www.dov.vlaanderen.be/data/sondering/1974-016927
    https://www.dov.vlaanderen.be/data/sondering/1980-024720
    https://www.dov.vlaanderen.be/data/sondering/1992-000339
    https://www.dov.vlaanderen.be/data/sondering/1976-030148

### Get CPT measurements in a bounding box based on specific properties

We can combine a query on attributes with a query on geographic location to get the CPT measurements within a bounding box that have specific properties.

The following example requests the CPT measurements with a depth greater than or equal to 2000 meters within the given bounding box.

(Note that the datatype of the *literal* parameter should be a string, regardless of the datatype of this field in the output dataframe.)

```python
from owslib.fes import PropertyIsGreaterThanOrEqualTo

query = PropertyIsGreaterThanOrEqualTo(
            propertyname='diepte_sondering_tot',
            literal='20')

df = sondering.search(
    location=Within(Box(200000, 211000, 205000, 214000)),
    query=query
    )

df.head()
```

    [000/009] ccccccccc

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      z
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.30
      1.22
      NaN
      1.0
      NaN
      0.8

      1
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.35
      3.19
      NaN
      2.0
      NaN
      1.0

      2
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.40
      7.21
      NaN
      63.0
      NaN
      1.2

      3
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.45
      12.75
      NaN
      138.0
      NaN
      1.2

      4
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-10/095-S1
      200030.2
      212577.5
      26.58
      1.25
      20.65
      2010-08-30
      VO - Afdeling Geotechniek
      continu elektrisch
      200kN - TRACK-TRUCK
      2010-08-30 12:50:00
      1.45
      1.50
      15.26
      NaN
      143.0
      NaN
      1.4

We can look at one of the CPT measurements in a webbrowser using its *pkey_sondering*:

```python
for pkey_sondering in set(df.pkey_sondering):
    print(pkey_sondering)
```

    https://www.dov.vlaanderen.be/data/sondering/2009-000054
    https://www.dov.vlaanderen.be/data/sondering/2007-049200
    https://www.dov.vlaanderen.be/data/sondering/2015-054999
    https://www.dov.vlaanderen.be/data/sondering/2010-062407
    https://www.dov.vlaanderen.be/data/sondering/2015-054995
    https://www.dov.vlaanderen.be/data/sondering/2009-000053
    https://www.dov.vlaanderen.be/data/sondering/2015-055496
    https://www.dov.vlaanderen.be/data/sondering/2009-000052
    https://www.dov.vlaanderen.be/data/sondering/2007-049201

### Select CPT measurements in a municipality and return depth

We can limit the columns in the output dataframe by specifying the *return_fields* parameter in our search.

In this example we query all the CPT measurements in the city of Ghent and return their depth:

```python
query = PropertyIsEqualTo(propertyname='gemeente',
                          literal='Gent')
df = sondering.search(query=query,
                      return_fields=('diepte_sondering_tot',))
df.head()
```

      diepte_sondering_tot

      0
      2.7

      1
      1.4

      2
      7.6

      3
      11.5

      4
      18.6

```python
df.describe()
```

      diepte_sondering_tot

      count
      3560.000000

      mean
      18.495803

      std
      8.505536

      min
      1.000000

      25%
      11.400000

      50%
      18.600000

      75%
      24.600000

      max
      52.600000

```python
df.boxplot()
```

### Get CPT measurements based on fields not available in the standard output dataframe

To keep the output dataframe size acceptable, not all availabe WFS fields are included in the standard output. However, one can use this information to select CPT measurements as illustrated below.

For example, make a selection of the CPT measurements in municipality the of Antwerp, using a conustype 'U':

```python
from owslib.fes import And

query = And([PropertyIsEqualTo(propertyname='gemeente',
                               literal='Antwerpen'),
             PropertyIsEqualTo(propertyname='conus',
                               literal='U')]
            )
df = sondering.search(query=query,
                      return_fields=('pkey_sondering', 'sondeernummer', 'x', 'y', 'diepte_sondering_tot', 'datum_aanvang'))
df.head()
```

      pkey_sondering
      sondeernummer
      x
      y
      diepte_sondering_tot
      datum_aanvang

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      29.70
      1993-03-02

      1
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-02/111-S1
      150347.3
      214036.4
      29.95
      2002-12-17

      2
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-04/123-SKD4-E
      146437.7
      222317.5
      4.45
      2004-07-12

      3
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-04/123-SKD6-E
      146523.9
      222379.7
      7.40
      2004-07-14

      4
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-04/123-SKD5-E
      146493.4
      222298.8
      1.65
      2004-07-16

### Get CPT data, returning fields not available in the standard output dataframe

As denoted in the previous example, not all available fields are available in the default output frame to keep its size limited. However, you can request any available field by including it in the *return_fields* parameter of the search:

```python
query = And([PropertyIsEqualTo(propertyname='gemeente', literal='Gent'),
             PropertyIsEqualTo(propertyname='conus', literal='U')])

df = sondering.search(query=query,
                      return_fields=('pkey_sondering', 'sondeernummer', 'diepte_sondering_tot',
                                     'conus', 'x', 'y'))

df.head()
```

      pkey_sondering
      sondeernummer
      x
      y
      diepte_sondering_tot
      conus

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SV
      110241.6
      204692.2
      33.80
      U

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SI
      110062.5
      205051.4
      15.65
      U

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SII
      110107.0
      204965.3
      26.50
      U

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIII
      110152.4
      204876.1
      16.50
      U

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIV
      110197.8
      204787.0
      16.70
      U

```python
df
```

      pkey_sondering
      sondeernummer
      x
      y
      diepte_sondering_tot
      conus

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SV
      110241.6
      204692.2
      33.80
      U

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SI
      110062.5
      205051.4
      15.65
      U

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SII
      110107.0
      204965.3
      26.50
      U

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIII
      110152.4
      204876.1
      16.50
      U

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIV
      110197.8
      204787.0
      16.70
      U

      5
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SIX
      110479.5
      205240.7
      27.60
      U

      6
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SVI
      110288.5
      204608.8
      16.80
      U

      7
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SVII
      110334.3
      204519.8
      26.70
      U

      8
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SX
      110685.0
      204845.5
      27.50
      U

      9
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SXI
      109941.5
      204346.9
      25.60
      U

      10
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/020-SXII
      110412.2
      204398.1
      26.50
      U

      11
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/096-SIX(CPT9)
      105018.0
      190472.0
      17.60
      U

      12
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/096-SVII(CPT7)
      105046.0
      190550.0
      26.05
      U

      13
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-94/096-SVIII(CPT8)
      104997.0
      190521.0
      24.75
      U

      14
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-97/002-S2
      105376.6
      189104.3
      29.90
      U

      15
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-97/002-S3
      105391.3
      189083.7
      5.90
      U

      16
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-97/002-S1
      105399.3
      189065.2
      30.60
      U

      17
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S1
      106104.1
      188699.4
      18.05
      U

      18
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S2
      106045.3
      188708.4
      17.30
      U

      19
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S3
      106100.5
      188743.8
      18.70
      U

      20
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S5
      106130.0
      188712.0
      17.30
      U

      21
      https://www.dov.vlaanderen.be/data/sondering/2...
      GEO-01/162-S4
      106077.5
      188686.0
      17.00
      U

## Resistivity plot

The data for the reporting of resistivity plots with the online application, see for example [this report](https://www.dov.vlaanderen.be/zoeken-ocdov/proxy-sondering/sondering/1993-001275/rapport/identifygrafiek?outputformaat=PDF), is also accessible with the pydov package. Querying the data for this specific _sondering_:

```python
query = PropertyIsEqualTo(propertyname='pkey_sondering',
                          literal='https://www.dov.vlaanderen.be/data/sondering/1993-001275')
df_sond = sondering.search(query=query)

df_sond.head()
```

    [000/001] c

      pkey_sondering
      sondeernummer
      x
      y
      start_sondering_mtaw
      diepte_sondering_van
      diepte_sondering_tot
      datum_aanvang
      uitvoerder
      sondeermethode
      apparaat
      datum_gw_meting
      diepte_gw_m
      z
      qc
      Qt
      fs
      u
      i

      0
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200KN
      NaN
      NaN
      0.6
      11.60
      NaN
      130.0
      69.0
      NaN

      1
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200KN
      NaN
      NaN
      0.7
      6.30
      NaN
      100.0
      29.0
      NaN

      2
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200KN
      NaN
      NaN
      0.8
      6.22
      NaN
      120.0
      -4.0
      NaN

      3
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200KN
      NaN
      NaN
      0.9
      4.92
      NaN
      120.0
      -48.0
      NaN

      4
      https://www.dov.vlaanderen.be/data/sondering/1...
      GEO-93/023-SII-E
      152740.0
      215493.0
      6.25
      0.0
      29.7
      1993-03-02
      MVG - Afdeling Geotechniek
      continu elektrisch
      200KN
      NaN
      NaN
      1.0
      4.40
      NaN
      80.0
      -35.0
      NaN

We have the depth (`z`) available, together with the measured values for each depth of the variables (in dutch):

* `qc`: Opgemeten waarde van de conusweerstand, uitgedrukt in MPa.
* `Qt`: Opgemeten waarde van de totale weerstand, uitgedrukt in kN.
* `fs`: Opgemeten waarde van de plaatelijke kleefweerstand uitgedrukt in kPa.
* `u`: Opgemeten waarde van de porienwaterspanning, uitgedrukt in kPa.
* `i`: Opgemeten waarde van de inclinatie, uitgedrukt in graden.

To recreate the resistivity plot, we also need the `resistivity number` (wrijvingsgetal `rf`), see [DOV documentation](https://www.dov.vlaanderen.be/page/sonderingen).

\begin{equation}
R_f = \frac{f_s}{q_c}
\end{equation}

**Notice:** $f_s$ is provide in kPa and $q_c$ in MPa.

Adding `rf` to the dataframe:

```python
df_sond["rf"] = df_sond["fs"]/df_sond["qc"]/10
```

Recreate the resistivity plot:

```python
import matplotlib.pyplot as plt
```

```python
def make_patch_spines_invisible(ax):
    ax.set_frame_on(True)
    ax.patch.set_visible(False)
    for sp in ax.spines.values():
        sp.set_visible(False)
```

```python
fig, ax0 = plt.subplots(figsize=(8, 12))

# Prepare the individual axis
ax_qc = ax0.twiny()
ax_fs = ax0.twiny()
ax_u = ax0.twiny()
ax_rf = ax0.twiny()

for i, ax in enumerate([ax_qc, ax_fs, ax_u]):
    ax.spines["top"].set_position(("axes", 1+0.05*(i+1)))
    make_patch_spines_invisible(ax)
    ax.spines["top"].set_visible(True)

# Plot the data on the axis
df_sond.plot(x="rf", y="z", label="rf", ax=ax_rf, color='purple', legend=False)
df_sond.plot(x="qc", y="z", label="qc (MPa)", ax=ax_qc, color='black', legend=False)
df_sond.plot(x="fs", y="z", label="fs (kPa)", ax=ax_fs, color='green', legend=False)
df_sond.plot(x="u", y="z", label="u (kPa)", ax=ax_u, color='red',
        legend=False, xlim=(-100, 300)) # ! 300 is hardocded here for the example

# styling and configuration
ax_rf.xaxis.label.set_color('purple')
ax_fs.xaxis.label.set_color('green')
ax_u.xaxis.label.set_color('red')

ax0.axes.set_visible(False)
ax_qc.axes.yaxis.set_visible(False)
ax_fs.axes.yaxis.set_visible(False)
for i, ax in enumerate([ax_rf, ax_qc, ax_fs, ax_u, ax0]):
    ax.spines["right"].set_visible(False)
    ax.spines["bottom"].set_visible(False)
    ax.xaxis.label.set_fontsize(15)
    ax.xaxis.set_label_coords(-0.05, 1+0.05*i)
    ax.spines['left'].set_position(('outward', 10))
    ax.spines['left'].set_bounds(0, 30)
ax_rf.set_xlim(0, 46)

ax0.invert_yaxis()
ax_rf.invert_xaxis()
ax_u.set_ylabel("Depth(m)", fontsize=12)
fig.legend(loc='lower center', ncol=4)
fig.tight_layout()
```

## Visualize locations

Using Folium, we can display the results of our search on a map.

```python
# import the necessary modules (not included in the requirements of pydov!)
import folium
from folium.plugins import MarkerCluster
from pyproj import Proj, transform
```

```python
# convert the coordinates to lat/lon for folium
def convert_latlon(x1, y1):
    inProj = Proj(init='epsg:31370')
    outProj = Proj(init='epsg:4326')
    x2,y2 = transform(inProj, outProj, x1, y1)
    return x2, y2
df['lon'], df['lat'] = zip(*map(convert_latlon, df['x'], df['y']))
# convert to list
loclist = df[['lat', 'lon']].values.tolist()
```

```python
# initialize the Folium map on the centre of the selected locations, play with the zoom until ok
fmap = folium.Map(location=[df['lat'].mean(), df['lon'].mean()], zoom_start=11)
marker_cluster = MarkerCluster().add_to(fmap)
for loc in range(0, len(loclist)):
    folium.Marker(loclist[loc], popup=df['sondeernummer'][loc]).add_to(marker_cluster)
fmap

```

```python

```


# 1. Chapter 1 - Plausible Reasoning
As we tread further into the twenty first century, almost everyone is expected to memorize the mantra "we must make data driven decisions" (well, at least most people in the technology space, and certainly data scientists). However, I want us to pause for a moment and think about what that really means?

In an idealized, rigid, platonic world this may simply mean to cast our first intuitions aside, instead using metrics related to the problem at hand, past decision's outcomes, and so on. Now, in a simplistic system this is a satisfactory approach. Consider the following (overly simple) scenario; you want to figure out the fastest way to drive from your house to the grocery store at rush hour (note how _constrained_ this situation already is). You intuitively feel as though route $A$ will be faster, so in general you take that route. You know that on average it takes about 9 minutes. Then, one day your roommate decides to join you and recommends route $B$, stating that they always drive that way and that on average it takes 6 minutes. You give route $B$ a shot an sure enough it takes 6 and half minutes.

This is an example of a very simplistic scenario that is conducive to basic data driven decision making. You have (essentially) all the data/variables you need in order to represent the scenario at hand. In other words, the decision is based on a univariate function:

$$\text{Time spent driving to grocery story} = f(\text{route})$$

We have data regarding driving time of both routes, and in this toy example there is really nothing else we need to consider in order to make an optimal decision that reduces driving time.

It should be no surprise that this is _not_ how things work in the real world. The real world is messy, contains an abundance of variables, and these variables manifest into **uncertainty**. The question that I have become obsessed with is as follows:

> How can we reason optimally in complex and uncertain situations?

For instance, let's now say that your company sells widgets. You, as a person in marketing, are in charge of coming up with sales offerings around the holiday season. Your initial intuition is that if you give out 20 dollar coupons to anyone who makes a purchase, you will get more purchases. However, there is a competing hypothesis from your colleague that suggests offering a discount to customers who make over 1000 dollar in purchases would actually be more effective at generating revenue. You have some historical time series data, but _neither_ have specifically been conditioned upon the exact hypotheses you are both proposing (i.e., you have no data that was collected during a 20 dollar coupon period, or during a 1000 dollar purchase discount period). The data that you have is necessarily incomplete, and even if it wasn't we still must confront the following logical problem:

> How do we use data (frequencies of events) to estimate plausibilities of beliefs?

In other words, how can we use the data present to estimate how plausible on hypothesis is compared to the other? This question is the central focus of _Probability Theory: The Logic of Science_, by E.T. Jaynes. Often viewed as the first text to make probability theory a "hard" branch of mathematics (compared to a group of ad hoc methods), it is an incredibly ambitious and thought provoking book that should be on any data scientist's or statistician's bookshelf. With that said, at times it is rather dense, and I wanted to take the time to create a set of articles that serve as chapter summaries. Note, these do not mean to replace the original text; rather, they can be read in tandem to clear up any sources of confusion and ensure clear understanding.

With that said, let's begin digging into the book, starting with the preface and chapter 1.

## 0.1 Preface
Jaynes starts the book by stating who the intended audience is; while this is generally not very informative, here it actually has a good deal of merit! He states that the material is meant to help those who are in fields such as physics, chemistry, biology, geology, medicine, economics, sociology, engineering, operations research, etc.; any field where **inference** is needed. He adds a footnote stating that by inference he means:

> **Inference:** **Deductive** reasoning whenever enough information is at hand to permit it; **inductive** or _plausible_ reasoning when the necessary information is not available (as it almost never is in real problems). If a problem can be solved with deductive reasoning, probability theory is not needed for it. Thus, the topic of this book is **the optimal processing of incomplete information**.

As a data scientist this type of footnote should send tingles down your spine. In nearly every situation you will encounter you are often treading the line of making inferences based off of a combination of deductive and inductive reasoning; who wouldn't want to be making those inferences in an optimal way?

Now, for those unfamiliar with the concepts of deduction and induction I recommend checking out this section of my article on Bayesian Inference. But, for a quick recap we can think of **deduction** as forming some hypothesis about the state of the world or it's workings, gathering data about that state, and then seeing if our data confirms or denies our hypothesis. It can be visualized below:

That is deduction. We perform deduction every day with relative ease. On the other hand we have **induction**, which works in the opposite direction:

Here, we gather data about the world around us, and after picking up on a pattern we induce a hypothesis, and then work towards concluding it's validity. Now, it should be noted that this idea of reasoning from evidence to hypothesis can be thought of as a parallel of reasoning from effect to cause.

As a small example (that I came across when reading _The Book of Why_ by Judea Pearl, consider Sherlock Holmes. Imagine that Sherlock Holmes was called to investigate a broken window. Sure he could go about it deductively and, while driving to the location of the window (i.e. before gathering any data), generate the hypothesis that the it was a group of children who broke the window. This is what humans do with ease, daily. However, what truly made Sherlock Holmes so powerful was his ability to perform induction. In this case, he arrives at the scene of the broken window, sees glass scattered about, and surely a host of other interesting things; he takes all of this _data_ and then generates a hypothesis about how the window was broken.

If that example leads you to feel slightly intimidated by the process of induction, do not worry! Without digging into it too deeply, there is actually an _asymmetry_ going on here that can lead you into _chaos theory_. For this I will borrow another example, this time from Nassim Taleb. Imagine that you want to understand what happens when you leave a block of ice out at room temperature. You hypothesize that it will melt (form your hypothesis). You then take 10 blocks of ice, leave them all out at room temperature, and see that they do indeed all melt (collect data). You just performed a very straight forward deduction, making use of what is referred to as the **forward process**.

Consider the inverse case now. Imagine that you walk into your kitchen and see that there is a puddle of water on the ground. You try and gather data, seeing if there is a leak in a pipe anywhere, if a cup spilled, etc. There is an incredibly long list of ways that this water could have gotten there. Trying to determine (from what is most likely incomplete data) that the water is actually there due to a block of ice that melted, is incredibly challenging. This is known as the **backward process**. If you are interested in this type of problem I recommend looking at the [Butterfly Effect](https://en.wikipedia.org/wiki/Butterfly_effect); for now I will leave it here to prevent going down a rabbit hole.

### 0.2 The Theme of the Book
To give us a north star to focus on, I want to take a moment to highlight what is essentially the theme of the book:

> **Probability theory as extended logic**.

We will dig into this much further in subsequent posts, but what this book does is create a framework in which the rules of probability theory can be viewed as uniquely valid principles of _logic_ in general, leaving out reference to _chance_ or _random variables_. This allows for the imaginary distinction between probability theory and statistical inference to disappear, allowing logical unity as well as greater technical power.

This theme amounts to recognition that the mathematical rules of probability theory are not merely rules for calculating frequencies of "random variables"; they are also the unique and consistent rules for conducting inference (i.e. plausible reasoning) of any kind.

This set of rules will automatically include all **Bayesian** calculations, as well as all **frequentist** calculations. Never the less, our basic rules are broader than either of these, and in many applications the calculations will not fit into either category. As explained by Jaynes:

> The traditional frequentist methods which only use sampling distributions are usable and useful in particularly simple, idealized problems; however, the represent the most proscribed cases of probability theory, because they presuppose conditions (independent repetitions of a 'random experiment' but no relevant prior information) that are hardly ever met in real problems. This approach is quite inadequate for the current needs of science.

Jaynes proceeds to dig into the idea of **prior** information/knowledge, and how it is essential that it is included in data analysis and inference. He writes that a false premise built into a model which is never questioned cannot be removed by any amount of new data. The use of models which correctly represent the prior information that scientists have about the mechanism at work can prevent such folly in the future. By ignoring prior information we not only set ourselves up to fail in the inference we are trying to make, but we also risk stalling out scientific progression itself. _No amount of analyzing coin tossing data by a stochastic model could have led us to the discovery of Newtonian Mechanics, which alone determines those data_.

With our stage set we are ready to dive into what is one of the most thought provoking and eye opening books in the realm of mathematics and science. Enter chapter 1.

# 1. Chapter 1: Plausible Reasoning
Suppose that on some dark night a policeman walks down a street, apparently deserted. Sudddenly he hears a burglar alarm, looks across the street, and sees a jewelry store with a broken window. Then a gentleman wearing a mask comes crawling through the broken window, carrying a bag which turns out to be full of expensive jewelry. The policeman doesn't hesitate at all in deciding that this gentleman is dishonest. But the question is: **By what reasoning process does he arrive at this conclusion?**

#### 1.1 Deductive and Plausible Reasoning
Keep in mind the key theme of this book, these articles, and more importantly data science in general: _How can we reason optimally in complex and uncertain situations?_ It may seem as though this preface and introduction get slightly into the weeds; this is a book on probability after all, right? Jaynes goes to great length to work through the _philosophy_ behind this vantage point of probability (seen as extended logic instead of based off of the standard Kolmogorov axioms); it is not unreasonable to ask why. The key lies in the fact that Jaynes is not simply trying to replace one axiomatic framework with another, but rather he is trying to build up a way to _reason_ consistently and logically, and apply that reasoning to any area of inference. I recommend taking a moment to appreciate that and let it sink in. Many of the problems that Jaynes works this in the forthcoming chapters could have been solved with standard orthodox or bayesian statistics; what makes this framework so powerful is that it provides a way to _reason consistently_. You are not required to hold in your head a bag of ad hoc methods and techniques. Rather, you have a methodology that allows you to approach and reason about any problem logically and consistently. Surely this is what any scientifically minded problem solver would hope for!

With that said let's return to our problem. A bit of thought makes it clear that the policeman's conclusion was not a logical deduction from the evidence; there may have been a perfectly reasonable explanation for everything! For instance, it is certainly _possible_ that the man was the owner of the jewelery store and was coming home from a masquerade party, and as he passed his store some teenagers threw a rock at the window, and he was simply trying to protect his property.

So, clearly the policemans reasoning process was not strictly logical deduction. However, we can grant that it did possess a high degree of validity. The evidence did not make the gentleman's dishonesty _certain_, but it did make it extremely _plausible_. This is the type of reasoning that humans have become proficient with long before studying mathematical theories. We encounter an abundance of these decisions daily (will it rain or won't it?) where we do not have enough information to permit deductive reasoning; but still we must decide immediately what to do.

Now, inspite of how familiar this process is to all of us, the formation of plausible conclusions is very subtle. This book allows us to replace intuitive judgements with definite theorems, and ad hoc procedures are replaced by rules that are determined uniquely by some elementary and inescapable criteria of rationality.

Now, let's take a moment to try and place this within the context of aristotelian logic; that is the most appropriate starting point to dissect the difference between **[deductive reasoning](https://en.wikipedia.org/wiki/Deductive_reasoning)** and **plausible reasoning**.

#### Deductive Reasoning
Deductive reasoning is defined as the process of reasoning from one or more **statements** (premises) in order to reach a logically certain conclusion. We can think of deductive reasoning as follows:

1. All men are mortal. (First premise)
2. Socrates is a man. (Second premise)
3. Therefore, Socrates is mortal. (Conclusion)

And this is defined mathematically below (known as **[modus ponens](https://en.wikipedia.org/wiki/Deductive_reasoning#Modus_ponens)**, the law of detachment):

$$
\begin{equation}
\text{If A is true, then B is true} \\
\frac{\text{A is true}}
{\text{therefore, B is true}}
\end{equation}
$$

It is the primary deductive rule of inference. An example would be:
1. If it is raining, then there are clouds in the sky.
2. It is raining.
3. Therefore, there are clouds in the sky.

There is also the inverse (known as **[modus tollens](https://en.wikipedia.org/wiki/Deductive_reasoning#Modus_tollens)**, the law of contrapositive), another deductive rule of inference:

$$
\begin{equation}
\text{If A is true, then B is true} \\
\frac{\text{B is false}}
{\text{therefore, A is false}}
\end{equation}
$$

And we have the counter simple example:
1. If it is raining, then there are clouds in the sky.
2. There are no clouds in the sky.
3. Thus, it is not raining.

Both of the above are known as **strong syllogisms**. Now in general, Deductive reasoning ("top-down logic") contrasts with inductive reasoning ("bottom-up logic") in the following way; in deductive reasoning, a conclusion is reached reductively by applying general rules which hold over the entirety of a closed domain of discourse, narrowing the range under consideration until only the conclusion(s) is left. In inductive reasoning, the conclusion is reached by generalizing or extrapolating from specific cases to general rules, i.e., there is epistemic uncertainty (for example the black swan). However, the inductive reasoning mentioned here is not the same as induction used in mathematical proofs – mathematical induction is actually a form of deductive reasoning.

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

```python

```


# Final assignment

## Deadline: March 31th 2022 at 16:00

## Instructions

* Write a report of all the problems that you have solved explaining clearly all the steps that you have taken in solving them. Include also the code that you used. **Remember to add comments to your answers and include also some testing part!**

* You can use any packages or tools that you see most fit for the purpose.

* Your work will be evaluated based on your report and the tools used. Therefore, pay also attention to how you prepare the report having a clear and logical structure.

* **Submit your answers by using Moodle.** The link is https://moodle.jyu.fi/mod/assign/view.php?id=705286&forceview=1.

* *Reminder about course grading: weekly exercises **65%**, assignment **35%*** **(modified after mixing exercise 8 and the final assignment)**.

* Max 25 points (5 for the first three problems and 10 points for problem 4)

## Problem 1

A window is being built and the bottom is a rectangle and the top is a semicircle. If there is 12 m of framing materials what must the dimensions of the window be to make the window area as big as possible?

Model the decision problem as an optimization problem and solve it with a method of your choosing. **Analyze the result!**

## Problem 2

The 10-dimensional Robsenbrock function (one of the variants) is defined as
$$
f(\mathbf{x}) = \sum_{i=1}^{9} 100 (x_{i+1} - x_i^2 )^2 + (1-x_i)^2
$$
for $x\in\mathbb R^{10}$.

Compare at least two different optimization method's performance in minimizing this function over $\mathbb R^{10}$. You can decide the method of comparison as the one that makes most sense to you. **Analyze the results!**

## Problem 3

Study biobjective optimization problem
$$
\begin{align}
\min \ &(\|x-(1,0)\|,\|x-(0,1)\|)\\
\text{s.t. }&x\in \mathbb R^2.
\end{align}
$$
Try to generate an evenly spread representation of the Pareto front. Plot the results in both the decision and objective spaces. **Analyze the results!**

## Problem 4

Find an application of optimization in a scientific paper, and replicate the results.

If you do not have the data or the model, then you need to come up with a mock-up data or model yourself. If you do not have the data and you cannot come up with any model or data that would somehow resemble the model or the data in the paper, then you need to write a description of why this could not be done.

If you want, you can apply any other optimization method than what was applied in the paper. This would actually be interesting, since then we can compare the results.


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Promijeni vidljivost <a href="javascript:code_toggle()">ovdje</a>.''')
display(tag)
```

Promijeni vidljivost ovdje.

```python
%matplotlib notebook
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
import ipywidgets as widgets
```

## Linearizacija - Jednostavno njihalo

Pri radu s modelima sustava, linearizacija je postupak modeliranja nelinearnog sustava s linearnom diferencijalnom jednadžbom u susjedstvu neke radne točke (obično točke ravnoteže). U ovom primjeru taj je postupak demonstriran pomoću jednostavnog njihala. Sila koja uzrokuje oscilacijsko gibanje njihala (prikazano na donjoj slici) definirana je kao $-mgsin\theta$. Jednadžba gibanja njihala definirana je kao:

\begin{equation}
    mL^2\frac{d^2\theta}{dt^2}=-mgsin\theta L.
\end{equation}

Nakon sređivanja dobivamo sljedeću nelinearnu diferencijalnu jednadžbu drugog reda:

\begin{equation}
    \frac{d^2\theta}{dt^2}+\frac{g}{L}sin\theta=0.
\end{equation}

U slučaju malih kutnih pomaka, vrijedi aproksimacija malog kuta (i.e. $sin\theta\approx\theta$) i dobiva se sljedeća linearna diferencijalna jednadžba drugog reda:

\begin{equation}
    \frac{d^2\theta}{dt^2}+\frac{g}{L}\theta=0.
\end{equation}

---

        Jednostavno njihalo

### Kako koristiti ovaj interaktivni primjer?

Pomičite klizače za promjenu duljine njihala $L$ i vrijednosti početnih uvjeta $\theta_0$ i $\dot{\theta_0}$.

```python
# create figure
fig = plt.figure(figsize=(9.8, 3),num='Diskretizacija - jednostavno njihalo')

# add sublot
ax = fig.add_subplot(111)
ax.set_title('Vremenski odziv')
ax.set_ylabel('izlaz')
ax.set_xlabel('$t$ [s]')
ax.axhline(y=0, xmin=-1, xmax=6, color='k', linewidth=1)

ax.grid(which='both', axis='both', color='lightgray')

nonlinear, = ax.plot([], [])
linear, = ax.plot([], [])

style = {'description_width': 'initial'}

g=9.81 # m/s^2

def model_nonlinear(ic,t,L):
    fi, fidot = ic
    return [fidot,-g/L*np.sin(fi)]

def model_linear(ic,t,L):
    fi, fidot = ic
    return [fidot,-g/L*fi]

def build_model(y0,ypika0,L):
    ic=[y0,ypika0]
    t=np.linspace(0,5,num=500)
    fi=odeint(model_nonlinear,ic,t,args=(L,))
    ys=fi[:,0]

    fi_linear=odeint(model_linear,ic,t,args=(L,))
    ys_linear=fi_linear[:,0]

    global nonlinear, linear

    ax.lines.remove(nonlinear)
    ax.lines.remove(linear)

    nonlinear, = ax.plot(t,ys,label='original',color='C0', linewidth=5)
    linear, = ax.plot(t,ys_linear,label='linearizacija',color='C3', linewidth=2)

    ax.legend()

    ax.relim()
    ax.autoscale_view()

L_slider=widgets.FloatSlider(value=0.3, min=.01, max=2., step=.01,
                                        description='$L$ [m]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

ypika0_slider=widgets.FloatSlider(value=1, min=-3, max=3, step=0.1,
                                        description='$\dot \\theta_0$ [rad/s]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

y0_slider=widgets.FloatSlider(value=1, min=-3, max=3, step=0.1,
                                        description='$\\theta_0$ [rad]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

input_data=widgets.interactive_output(build_model, {'y0':y0_slider,'ypika0':ypika0_slider,'L':L_slider})

display(L_slider,y0_slider,ypika0_slider,input_data)
```

    <IPython.core.display.Javascript object>

    FloatSlider(value=0.3, continuous_update=False, description='$L$ [m]:', max=2.0, min=0.01, step=0.01)

    FloatSlider(value=1.0, continuous_update=False, description='$\\theta_0$ [rad]:', max=3.0, min=-3.0)

    FloatSlider(value=1.0, continuous_update=False, description='$\\dot \\theta_0$ [rad/s]:', max=3.0, min=-3.0)

    Output()


## Data-driven Design and Analyses of Structures and Materials (3dasm)

## Lecture 6

### Miguel A. Bessa | M.A.Bessa@tudelft.nl  | Associate Professor

**What:** A lecture of the "3dasm" course

**Where:** This notebook comes from this [repository](https://github.com/bessagroup/3dasm_course)

**Reference for entire course:** Murphy, Kevin P. *Probabilistic machine learning: an introduction*. MIT press, 2022. Available online [here](https://probml.github.io/pml-book/book1.html)

**How:** We try to follow Murphy's book closely, but the sequence of Chapters and Sections is different. The intention is to use notebooks as an introduction to the topic and Murphy's book as a resource.
* If working offline: Go through this notebook and read the book.
* If attending class in person: listen to me (!) but also go through the notebook in your laptop at the same time. Read the book.
* If attending lectures remotely: listen to me (!) via Zoom and (ideally) use two screens where you have the notebook open in 1 screen and you see the lectures on the other. Read the book.

**Optional reference (the "bible" by the "bishop"... pun intended 😆) :** Bishop, Christopher M. *Pattern recognition and machine learning*. Springer Verlag, 2006.

**References/resources to create this notebook:**
* [Car figure](https://korkortonline.se/en/theory/reaction-braking-stopping/)

Apologies in advance if I missed some reference used in this notebook. Please contact me if that is the case, and I will gladly include it here.

## **OPTION 1**. Run this notebook **locally in your computer**:
1. Confirm that you have the 3dasm conda environment (see Lecture 1).

2. Go to the 3dasm_course folder in your computer and pull the last updates of the [repository](https://github.com/bessagroup/3dasm_course):
```
git pull
```
3. Open command window and load jupyter notebook (it will open in your internet browser):
```
conda activate 3dasm
jupyter notebook
```
4. Open notebook of this Lecture.

## **OPTION 2**. Use **Google's Colab** (no installation required, but times out if idle):

1. go to https://colab.research.google.com
2. login
3. File > Open notebook
4. click on Github (no need to login or authorize anything)
5. paste the git link: https://github.com/bessagroup/3dasm_course
6. click search and then click on the notebook for this Lecture.

```python
# Basic plotting tools needed in Python.

import matplotlib.pyplot as plt # import plotting tools to create figures
import numpy as np # import numpy to handle a lot of things!
from IPython.display import display, Math # to print with Latex math

%config InlineBackend.figure_format = "retina" # render higher resolution images in the notebook
plt.style.use("seaborn") # style for plotting that comes from seaborn
plt.rcParams["figure.figsize"] = (8,4) # rescale figure size appropriately for slides
```

## Outline for today

* Continuation of previous lecture: Bayesian inference for one hidden rv
    - Prior
    - Likelihood
    - Marginal likelihood
    - Posterior
    - Gaussian pdf's product

**Reading material**: This notebook + Chapter 3

## Recap of Lecture 5: car stopping distance with known $x$ and $p(z_2)$

We focused on the car stopping distance problem with two rv's under the following conditions:
* We kept $x=75$ m/s.
* The "true" distribution of one of the rv's  was known: $p(z_2)=\mathcal{N}(\mu_{z_2}=0.1,\sigma_{z_2}^2=0.01^2)$
* But the distribution of the other rv ($z \equiv z_1$) is not known: $p(z)=?$

Under these conditions, recall the "true" model by observing the following plot, including some data observations.

```python
# This cell is hidden during presentation. It's just to define a function to plot the governing model of
# the car stopping distance problem. Defining a function that creates a plot allows to repeatedly run
# this function on cells used in this notebook.
def car_fig_2rvs(ax):
    x = np.linspace(3, 83, 1000)
    mu_z1 = 1.5; sigma_z1 = 0.5;  # parameters of the "true" p(z_1)
    mu_z2 = 0.1; sigma_z2 = 0.01; # parameters of the "true" p(z_2)
    mu_y = mu_z1*x + mu_z2*x**2 # From Homework of Lecture 4
    sigma_y = np.sqrt( (x*sigma_z1)**2 + (x**2*sigma_z2)**2 ) # From Homework of Lecture 4
    ax.set_xlabel("x (m/s)", fontsize=20) # create x-axis label with font size 20
    ax.set_ylabel("y (m)", fontsize=20) # create y-axis label with font size 20
    ax.set_title("Car stopping distance problem with two rv's", fontsize=20); # create title with font size 20
    ax.plot(x, mu_y, 'k:', label="Governing model $\mu_y$")
    ax.fill_between(x, mu_y - 1.9600 * sigma_y,
                    mu_y + 1.9600 * sigma_y,
                    color='k', alpha=0.2,
                    label='95% confidence interval ($\mu_y \pm 1.96\sigma_y$)') # plot 95% credence interval
    ax.legend(fontsize=15)
```

```python
# This cell is also hidden during presentation.
from scipy.stats import norm # import the normal dist, as we learned before!
def samples_y_with_2rvs(N_samples,x): # observations/measurements/samples for car stop. dist. prob. with 2 rv's
    mu_z1 = 1.5; sigma_z1 = 0.5;
    mu_z2 = 0.1; sigma_z2 = 0.01;
    samples_z1 = norm.rvs(mu_z1, sigma_z1, size=N_samples) # randomly draw samples from the normal dist.
    samples_z2 = norm.rvs(mu_z2, sigma_z2, size=N_samples) # randomly draw samples from the normal dist.
    samples_y = samples_z1*x + samples_z2*x**2 # compute the stopping distance for samples of z_1 and z_2
    return samples_y # return samples of y
```

```python
# vvvvvvvvvvv this is just a trick so that we can run this cell multiple times vvvvvvvvvvv
fig_car_new, ax_car_new = plt.subplots(1,2); plt.close() # create figure and close it
if fig_car_new.get_axes():
    del ax_car_new; del fig_car_new # delete figure and axes if they exist
    fig_car_new, ax_car_new = plt.subplots(1,2) # create them again
# ^^^^^^^^^^^ end of the trick ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
N_samples = 3 # CHANGE THIS NUMBER AND RE-RUN THE CELL
x = 75; empirical_y = samples_y_with_2rvs(N_samples, x); # Empirical measurements of N_samples at x=75
empirical_mu_y = np.mean(empirical_y); empirical_sigma_y = np.std(empirical_y); # empirical mean and std
car_fig_2rvs(ax_car_new[0]) # a function I created to include the background plot of the governing model
for i in range(2): # create two plots (one is zooming in on the error bar)
    ax_car_new[i].errorbar(x , empirical_mu_y,yerr=1.96*empirical_sigma_y, fmt='m*', markersize=15);
    ax_car_new[i].scatter(x*np.ones_like(empirical_y),empirical_y, s=40,
                          facecolors='none', edgecolors='k', linewidths=2.0)
print("Empirical mean[y] is",empirical_mu_y, "(real mean[y]=675)")
print("Empirical std[y] is",empirical_sigma_y,"(real std[y]=67.6)")
fig_car_new.set_size_inches(25, 5) # scale figure to be wider (since there are 2 subplots)
```

## Recap of Lecture 5: Summary of our model

1. The **observation distribution**:

$$
p(y|z) = \mathcal{N}\left(y | \mu_{y|z}=w z+b, \sigma_{y|z}^2\right) = \frac{1}{C_{y|z}} \exp\left[ -\frac{1}{2\sigma_{y|z}^2}(y-\mu_{y|z})^2\right]
$$

where $C_{y|z} = \sqrt{2\pi \sigma_{y|z}^2}$ is the **normalization constant** of the Gaussian pdf, and where $\mu_{y|z}=w z+b$, with $w$, $b$ and $\sigma_{y|z}^2$ being constants.

2. and the **prior distribution**: $p(z) = \frac{1}{C_z}$

where $C_z = z_{max}-z_{min}$ is the **normalization constant** of the Uniform pdf, i.e. the value that guarantees that $p(z)$ integrates to one.

### Recap of Lecture 5: Data

* Since we usually don't know the true process, we can only observe/collect data $y=\mathcal{D}_y$:

```python
print("Example of N=%1i data points for y at x=%1.1f m/s with :" % (N_samples,x), empirical_y)
```

    Example of N=3 data points for y at x=75.0 m/s with : [641.47373038 733.92797742 637.52728953]

## Recap of Lecture 5: Posterior from Bayes' rule applied to data

Use Bayes' rule applied to data to determine the posterior:

$\require{color}$
$$
{\color{green}p(z|y=\mathcal{D}_y)} = \frac{ {\color{blue}p(y=\mathcal{D}_y|z)}{\color{red}p(z)} } {p(y=\mathcal{D}_y)}
$$

That requires calculating the likelihood (here, it results from a product of Gaussian densities):

$$
{\color{blue}p(y=\mathcal{D}_y | z)} = \frac{1}{|w|^N} \cdot C \cdot \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left[ -\frac{1}{2\sigma^2}(z-\mu)^2\right]
$$

where $\mu = \frac{w^2\sigma^2}{\sigma_{y|z}^2} \sum_{i=1}^N \mu_i$

$\sigma^2 = \frac{\sigma_{y|z}^2}{w^2 N}$, and

$C = \frac{1}{2\pi^{(N-1)/2}} \sqrt{\frac{\sigma^2}{\left( \frac{\sigma_{y|z}^2}{w^2}\right)^N}}
$

After calculating the likelihood, we determined the marginal likelihood:

$$
p(y=\mathcal{D}_y) = \frac{C}{|w|^N C_z}
$$

From which we got the posterior:

$$\require{color}\begin{align}
{\color{green}p(z|y=\mathcal{D}_y)} &= \frac{ p(y=\mathcal{D}_y|z)p(z) } {p(y=\mathcal{D}_y)} \\
&= \frac{1}{p(y=\mathcal{D}_y)} \cdot \frac{1}{|w|^N} C \cdot \mathcal{N}(z|\mu,\sigma^2) \cdot \frac{1}{C_z} \\
&= \mathcal{N}(z|\mu, \sigma^2)
\end{align}
$$

which is a **normalized** Gaussian pdf in $z$ with mean and variance as shown in the previous cell.

## Determining the Posterior Predictive Distribution (PPD) from the posterior

However, as we mentioned, Bayes' rule is just a way to calculate the posterior:

$$
p(z|y=\mathcal{D}_y) = \frac{ p(y=\mathcal{D}_y|z)p(z) } {p(y=\mathcal{D}_y)}
$$

What we really want is the Posterior Predictive Distribution (PPD) . This comes after calculating the posterior given some data $\mathcal{D}_y$:

$$\require{color}
{\color{orange}p(y|y=\mathcal{D}_y)} = \int p(y|z) p(z|y=\mathcal{D}_y) dz
$$

which is often written in simpler notation: $p(y|\mathcal{D}_y) = \int p(y|z) p(z|\mathcal{D}_y) dz$

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \int \underbrace{p(y|z)}_{\text{observation}\\ \text{distribution}} \overbrace{p(z|y=\mathcal{D}_y)}^{\text{posterior}} dz
$$

Considering the terms we found before, we get:

$$\begin{align}
p(y|\mathcal{D}_y) &= \int \underbrace{\frac{1}{|w|}\frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\} }_{\text{observation}\\ \text{distribution}} \overbrace{\mathcal{N}(z|\mu, \sigma^2)}^{\text{posterior}} dz
\end{align}
$$

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \frac{1}{|w|} \int {\color{blue}\frac{1}{\sqrt{2\pi \left(\frac{\sigma_{y|z}}{w}\right)^2}} \exp\left\{ -\frac{1}{2\left(\frac{\sigma_{y|z}}{w}\right)^2}\left[z-\left(\frac{y-b}{w}\right)\right]^2\right\}} \mathcal{N}(z|\mu, \sigma^2) dz
$$

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \frac{1}{|w|} \int \mathcal{N}\left(z\left|\frac{y-b}{w}, \left(\frac{\sigma_{y|z}}{w}\right)^2\right.\right) \mathcal{N}(z|\mu, \sigma^2) dz
$$

This is (again!) the product of two Gaussians!

In Lecture 5 (and the Homework!) you saw (and demonstrated!) that the product of two or more univariate (and multivariate!) Gaussians is...

* Another Gaussian! Although it needs to be scaled by a constant...

So, we conclude that the PPD is an integral of a Gaussian:

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \frac{1}{|w|} \int  C^* \mathcal{N}\left(z|\mu^*, \left(\sigma^*\right)^2\right) dz
$$

where $\mu^* = \left(\sigma^* \right)^2 \left( \frac{\mu}{\sigma^2} + \frac{(y-b)/w}{\left(\frac{\sigma_{y|z}}{w}\right)^2} \right) = \left(\sigma^* \right)^2 \left( \frac{\mu}{\sigma^2} + \frac{(y-b)\cdot w}{\sigma_{y|z}^2} \right)$

$\left( \sigma^* \right)^2 = \frac{1}{\frac{1}{\sigma^2}+\frac{1}{\left( \frac{\sigma_{y|z}}{w}\right)^2}}= \frac{1}{\frac{1}{\sigma^2}+\frac{w^2}{\sigma_{y|z}^2}}$

$C^* = \frac{1}{\sqrt{2\pi \left( \sigma^2 + \frac{\sigma_{y|z}^2}{w^2} \right)}}\exp\left[ - \frac{\left(\mu - \frac{y-b}{w}\right)^2}{2\left( \sigma^2+\frac{\sigma_{y|z}^2}{w^2}\right)}\right]$

This integral is simple to solve!

$$\require{color}
\begin{align}
{\color{orange}p(y|\mathcal{D}_y)} &= \frac{1}{|w|} \int  C^* \mathcal{N}\left(z|\mu^*, \left(\sigma^*\right)^2\right) dz \\
&= \frac{C^*}{|w|} \int   {\color{blue}\mathcal{N}\left(z|\mu^*, \left(\sigma^*\right)^2\right)} dz
\end{align}
$$

What's the result of integrating the blue term?

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \frac{C^*}{|w|}
$$

## Exercise 1

Rewrite the PPD to show that it becomes:

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \mathcal{N}\left(y| b+\mu w, w^2\sigma^2+\sigma_{y|z}^2 \right)
$$

a normalized univariate Gaussian!

## A long way to show that the PPD is a simple Gaussian...

$$\require{color}
{\color{orange}p(y|\mathcal{D}_y)} = \mathcal{N}\left(y| b+\mu w, w^2\sigma^2+\sigma_{y|z}^2 \right)
$$

where we recall that each constant is:

$b = 0.1x^2 = 562.5$

$w = x = 75$

$\sigma_{y|z}^2 = (x^2 \sigma_{z_2})^2=(75^2\cdot0.01)^2=56.25^2$

$\sigma^2 = \frac{\sigma_{y|z}^2}{w^2 N} = \frac{(x^2\cdot\sigma_{z_2})^2}{x^2 N} = \frac{x^2\cdot\sigma_{z_2}^2}{N} $

$\mu = \frac{w^2 \sigma^2}{\sigma_{y|z}^2} \sum_{i=1}^{N} \mu_i = \cdots = \frac{\sum_{i=1}^N y_i}{w N}-\frac{b}{w}$

#### Note on algebra to determine $\mu$ parameter:

$$
\begin{align}
\mu &= \frac{w^2 \sigma^2}{\sigma_{y|z}^2} \sum_{i=1}^{N} \mu_i \\
&= \frac{\sum_{i=1}^{N} \mu_i}{N} = \frac{1}{N} \sum_{i=1}^{N} \left(\frac{y_i-b}{w}\right) \\
&= \frac{1}{w N} \sum_{i=1}^N\left( y_i-b\right) \\
&= \frac{1}{w N} \left( \sum_{i=1}^N y_i-N b\right) \\
& = \frac{\sum_{i=1}^N y_i}{w N}-\frac{b}{w}
\end{align}
$$

## A long way to show that the PPD is a simple Gaussian...

$$\require{color}
\begin{align}
{\color{orange}p(y|\mathcal{D}_y)} &= \mathcal{N}\left(y| b+\mu w, w^2\sigma^2+\sigma_{y|z}^2 \right) \\
&= \mathcal{N}\left(y \left| \left(\sum_{i=1}^N \frac{y_i}{N}\right), \sigma_{y|z}^2 \left(\frac{1}{N} + 1 \right) \right. \right)
\end{align}
$$

where $y_i$ are each of the $N$ data points of the observed data $\mathcal{D}_y$, and $\sigma_{y|z}^2 = (x^2 \sigma_{z_2})^2=(75^2\cdot0.01)^2=56.25^2$ is the variance arising from the contribution of $z_2$ on $y$.

* **Very Important Questions (VIQs)**: What does this result tell us? Did you expect this predicted distribution for $y$?

```python
fig_car_PPD, ax_car_PPD = plt.subplots(1,2); plt.close() # create figure and close it
if fig_car_new.get_axes():
    del ax_car_PPD; del fig_car_PPD; fig_car_PPD, ax_car_PPD = plt.subplots(1,2) # delete fig & axes & create them
N_samples = 30000 # CHANGE THIS NUMBER AND RE-RUN THE CELL
x = 75; empirical_y = samples_y_with_2rvs(N_samples, x); # Empirical measurements of N_samples at x=75
empirical_mu_y = np.mean(empirical_y); empirical_sigma_y = np.std(empirical_y); # empirical mean and std
# Calculate PPD mean and standard deviation:
PPD_mu_y = np.mean(empirical_y); sigma_z2 = 0.01; PPD_sigma_y = np.sqrt( (x**2*sigma_z2)**2*(1/N_samples + 1) )
car_fig_2rvs(ax_car_PPD[0]) # a function I created to include the background plot of the governing model
for i in range(2): # create two plots (one is zooming in on the error bar)
    ax_car_PPD[i].errorbar(x , empirical_mu_y,yerr=1.96*empirical_sigma_y, fmt='m*', markersize=15, elinewidth=6);
    ax_car_PPD[i].errorbar(x , PPD_mu_y,yerr=1.96*PPD_sigma_y, color='#F39C12', fmt='*', markersize=5, elinewidth=3);
    ax_car_PPD[i].scatter(x*np.ones_like(empirical_y),empirical_y, s=100,facecolors='none', edgecolors='k', linewidths=2.0)
print("PPD & empirical mean[y] are the same:",empirical_mu_y, "(real mean[y]=675)")
print("PPD std[y] is",PPD_sigma_y, "& empirical std[y] is",empirical_sigma_y,"(real std[y]=67.6)")
fig_car_PPD.set_size_inches(25, 5) # scale figure to be wider (since there are 2 subplots)
```

### Reflection on what we are observing

1. Generally speaking, our PPD is quite reasonable!
    * For few data points it is more reasonable than just calculating the standard deviation directly from the data.

2. However, as the number of data points increases it starts getting "overconfident" (see PPD as $N\rightarrow \infty$ or play with the figure above by increasing $N$).
    * This results from our choice of prior... Our belief was incorrect.
        * The hidden rv $z$ is actually a Gaussian distribution, instead of a noninformative Uniform distribution

Please keep this in your head:
* (Bayesian) ML is not magic. Every modeling choice you make affects the predictions you get.
* Of course, there are ways of getting "closer" to the truth! We'll take some steps in that direction in the remainder of the course.

# HOMEWORK

Consider the same problem, but now starting from a different model:

1. Same **observation distribution** as before:

$$
p(y|z) = \mathcal{N}\left(y | \mu_{y|z}=w z+b, \sigma_{y|z}^2\right) = \frac{1}{C_{y|z}} \exp\left[ -\frac{1}{2\sigma_{y|z}^2}(y-\mu_{y|z})^2\right]
$$

2. but now assuming a different **prior distribution**: $p(z) = \mathcal{N}\left(z| \overset{\scriptscriptstyle <}{\mu}_z=3, \overset{\scriptscriptstyle <}{\sigma}_z^2=2^2
\right)$

In my notation, the superscript $\overset{\scriptscriptstyle <}{(\cdot)}$ indicates a parameter of the prior distribution.

### Notes about the prior distribution

* We would have to be very lucky if our "belief" coincided with the "true" distribution of $z$.
    * Usually, we have beliefs but they are not really true (not talking about religion 😆).
        - Our hope is that our beliefs are at least reasonable!

* When defining a prior we are making a decision about two things:
    1. The distribution.
        * For example, in this exercise we are assuming that the prior is Gaussian (before we assumed a noninformative Uniform prior). In this case we hit the jackpot! But remember that we are cheating here... That's why we know the actual distribution of $z$ is a Gaussian!
    2. The parameters of the distribution.
        * For example, in this exercise we are assuming values that are not the true ones! This is normal! As I said, usually we don't know the truth about the "hidden" variable. Most times we don't even know how  many hidden variables we have...

### See you next class

Have fun!


```python
from resources.workspace import *
```

$
% START OF MACRO DEF
% DO NOT EDIT IN INDIVIDUAL NOTEBOOKS, BUT IN macros.py
%
\newcommand{\Reals}{\mathbb{R}}
\newcommand{\Expect}[0]{\mathbb{E}}
\newcommand{\NormDist}{\mathcal{N}}
%
\newcommand{\DynMod}[0]{\mathscr{M}}
\newcommand{\ObsMod}[0]{\mathscr{H}}
%
\newcommand{\mat}[1]{{\mathbf{{#1}}}}
%\newcommand{\mat}[1]{{\pmb{\mathsf{#1}}}}
\newcommand{\bvec}[1]{{\mathbf{#1}}}
%
\newcommand{\trsign}{{\mathsf{T}}}
\newcommand{\tr}{^{\trsign}}
\newcommand{\tn}[1]{#1}
\newcommand{\ceq}[0]{\mathrel{≔}}
%
\newcommand{\I}[0]{\mat{I}}
\newcommand{\K}[0]{\mat{K}}
\newcommand{\bP}[0]{\mat{P}}
\newcommand{\bH}[0]{\mat{H}}
\newcommand{\bF}[0]{\mat{F}}
\newcommand{\R}[0]{\mat{R}}
\newcommand{\Q}[0]{\mat{Q}}
\newcommand{\B}[0]{\mat{B}}
\newcommand{\C}[0]{\mat{C}}
\newcommand{\Ri}[0]{\R^{-1}}
\newcommand{\Bi}[0]{\B^{-1}}
\newcommand{\X}[0]{\mat{X}}
\newcommand{\A}[0]{\mat{A}}
\newcommand{\Y}[0]{\mat{Y}}
\newcommand{\E}[0]{\mat{E}}
\newcommand{\U}[0]{\mat{U}}
\newcommand{\V}[0]{\mat{V}}
%
\newcommand{\x}[0]{\bvec{x}}
\newcommand{\y}[0]{\bvec{y}}
\newcommand{\z}[0]{\bvec{z}}
\newcommand{\q}[0]{\bvec{q}}
\newcommand{\br}[0]{\bvec{r}}
\newcommand{\bb}[0]{\bvec{b}}
%
\newcommand{\bx}[0]{\bvec{\bar{x}}}
\newcommand{\by}[0]{\bvec{\bar{y}}}
\newcommand{\barB}[0]{\mat{\bar{B}}}
\newcommand{\barP}[0]{\mat{\bar{P}}}
\newcommand{\barC}[0]{\mat{\bar{C}}}
\newcommand{\barK}[0]{\mat{\bar{K}}}
%
\newcommand{\D}[0]{\mat{D}}
\newcommand{\Dobs}[0]{\mat{D}_{\text{obs}}}
\newcommand{\Dmod}[0]{\mat{D}_{\text{obs}}}
%
\newcommand{\ones}[0]{\bvec{1}}
\newcommand{\AN}[0]{\big( \I_N - \ones \ones\tr / N \big)}
%
% END OF MACRO DEF
$
In this tutorial we shall derive:

# the Kalman filter for multivariate systems.

The [forecast step](T3%20-%20Univariate%20Kalman%20filtering.ipynb#Exc-3.7:-The-forecast-step:)
remains essentially unchanged.
The only difference is that $\DynMod$ is now a matrix, as well as the use of the transpose ${}^T$ in the covariance equation:
$\begin{align}
\bb_k
&= \DynMod_{k-1} \hat{\x}_{k-1} \, , \tag{1a} \\\
\B_k
&= \DynMod_{k-1} \bP_{k-1} \DynMod_{k-1}^T + \Q_{k-1} \, . \tag{1b}
\end{align}$

However, the analysis step [[Exc 2.18](T2%20-%20Bayesian%20inference.ipynb#Exc--2.18-'Gaussian-Bayes':)] gets a little more complicated...

#### Exc 2 (The likelihood):

The analysis step is only concerned with a single time (index). We therefore drop the $k$ subscript in the following.

Suppose the observation, $\y$, is related to the true state, $\x$, via a (possibly rectangular) matrix, $\bH$:
\begin{align*}
\y &= \bH \x + \br \, , \;\; \qquad (2)
\end{align*}
where the noise follows the law $\br \sim \NormDist(\bvec{0}, \R)$ for some $\R>0$ (i.e. $\R$ is symmetric-positive-definite).

Derive the expression for the likelihood, $p(\y|\x)$.

```python
#show_answer('Likelihood derivation')
```

The following exercise derives the analysis step

#### Exc 4 (The 'precision' form of the KF):
Similarly to [Exc 2.18](T2%20-%20Bayesian%20inference.ipynb#Exc--2.18-'Gaussian-Bayes':),
it may be shown that the prior $p(\x) = \NormDist(\x \mid \bb,\B)$
and likelihood $p(\y|\x) = \NormDist(\y \mid \bH \x,\R)$,
yield the posterior:
\begin{align}
p(\x|\y)
&= \NormDist(\x \mid \hat{\x}, \bP) \tag{4}
\, ,
\end{align}
where the posterior/analysis mean (vector) and covariance (matrix) are given by:
\begin{align}
			\bP &= (\bH\tr \Ri \bH + \Bi)^{-1} \, , \tag{5} \\
			\hat{\x} &= \bP\left[\bH\tr \Ri \y + \Bi \bb\right] \tag{6} \, ,
\end{align}
Prove eqns (4-6).
Hint: as in [Exc 2.18](T2%20-%20Bayesian%20inference.ipynb#Exc--2.18-'Gaussian-Bayes':), the main part lies in "completing the square" in $\x$.

```python
#show_answer('KF precision')
```

We have now derived (one form of) the Kalman filter. In the multivariate case,
we know how to:

Propagate our estimate of $\x$ to the next time step using eqns (1a) and (1b).

Update our estimate of $\x$ by assimilating the latest observation $\y$, using eqns (5) and (6).

However, the computations can be pretty expensive...

#### Exc 5: Suppose $\x$ is $M$-dimensional and has a covariance matrix $\B$.
 * (a). What's the size of $\B$?
 * (b). How many "flops" (approximately, i.e. to leading order) are required
 to compute the "precision form" of the KF update equation, eqn (5) ?
 * (c). How much memory (bytes) is required to hold its covariance matrix $\B$ ?
 * (d). How many mega bytes's is this if $M$ is a million?

```python
#show_answer('Cov memory')
```

This is one of the principal reasons why basic extended KF is infeasible for DA.
The following derives another, often more practical, form of the KF analysis update.

#### Exc 6 (The "Woodbury" matrix inversion identity):
The following is known as the Sherman-Morrison-Woodbury lemma/identity,
$$\begin{align}
    \bP = \left( \B^{-1} + \V\tr \R^{-1} \U \right)^{-1}
    =
    \B - \B \V\tr \left( \R + \U \B \V\tr \right)^{-1} \U \B \, ,
    \tag{W}
\end{align}$$
which holds for any (suitably shaped matrices)
$\B$, $\R$, $\V,\U$ *such that the above exists*.

Prove the identity. Hint: don't derive it, just prove it!

```python
#show_answer('Woodbury')
```

#### Exc 7:
- Show that $\B$ and $\R$ must be square.
- Show that $\U$ and $\V$ are not necessarily square, but must have the same dimensions.
- Show that $\B$ and $\R$ are not necessarily of equal size.

Exc 7 makes it clear that the Woodbury identity may be used to compute $\bP$ by inverting matrices of the size of $\R$ rather than the size of $\B$.
Of course, if $\R$ is bigger than $\B$, then the identity is useful the other way around.

#### Exc 8 (Corollary 1):
Prove that, for any symmetric, positive-definite (SPD) matrices $\R$ and $\B$, and any matrix $\bH$,
$$\begin{align}
 	\left(\bH\tr \R^{-1} \bH + \B^{-1}\right)^{-1}
    &=
    \B - \B \bH\tr \left( \R + \bH \B \bH\tr \right)^{-1} \bH \B \tag{C1}
    \, .
\end{align}$$
Hint: consider the properties of [SPD](https://en.wikipedia.org/wiki/Definiteness_of_a_matrix#Properties) matrices.

```python
#show_answer('Woodbury C1')
```

#### Exc 10 (Corollary 2):
Prove that, for the same matrices as for Corollary C1,
$$\begin{align}
	\left(\bH\tr \R^{-1} \bH + \B^{-1}\right)^{-1}\bH\tr \R^{-1}
    &= \B \bH\tr \left( \R + \bH \B \bH\tr \right)^{-1}
    \tag{C2}
    \, .
\end{align}$$

```python
#show_answer('Woodbury C2')
```

#### Exc 12 (The "gain" form of the KF):
Now, let's go back to the KF, eqns (5) and (6). Since $\B$ and $\R$ are covariance matrices, they are symmetric-positive. In addition, we will assume that they are full-rank, making them SPD and invertible.

Define the Kalman gain by:
 $$\begin{align}
    \K &= \B \bH\tr \big(\bH \B \bH\tr + \R\big)^{-1} \, . \tag{K1}
\end{align}$$
 * (a) Apply (C1) to eqn (5) to obtain the Kalman gain form of analysis/posterior covariance matrix:
$$\begin{align}
    \bP &= [\I_M - \K \bH]\B \, . \tag{8}
\end{align}$$

* (b) Apply (C2)  to (5) to abtain the identity
$$\begin{align}
    \K &= \bP \bH\tr \R  \, . \tag{K2}
\end{align}$$

* (c) Show that $\bP \Bi = [\I_M - \K \bH]$.
* (d) Use (b) and (c) to obtain the Kalman gain form of analysis/posterior covariance
$$\begin{align}
     \hat{\x} &= \bb + \K\left[\y - \bH \bb\right] \, . \tag{9}
\end{align}$$

Together, eqns (8) and (9) define the Kalman gain form of the KF update.
The inversion (eqn 7) involved is of the size of $\R$, while in eqn (5) it is of the size of $\B$.

## In summary:
We have derived two forms of the multivariate KF analysis update step: the "precision matrix" form, and the "Kalman gain" form. The latter is especially practical when the number of observations is smaller than the length of the state vector.

### Next: [Time series analysis](T5%20-%20Time%20series%20analysis.ipynb)

notebooks/T4 - Multivariate Kalman.ipynb