 **Chapter 3:  [Imaging](Ch3-Imaging.ipynb)**

# Energy-Loss Near-Edge Structure

part of

 **[Analysis of Transmission Electron Microscope Data](_Analysis_of_Transmission_Electron_Microscope_Data.ipynb)**

by Gerd Duscher, 2019

Microscopy Facilities

Joint Institute of Advanced Materials

The University of Tennessee, Knoxville

Model based analysis and quantification of data acquired with transmission electron microscopes

## Content

- Retrieving and Plotting of reference EELS spectra from the [EELS database](https://eelsdb.eu/spectra/)
- Discussion of the energy-loss near-edge structure (ELNES) of specific edges.

## Load important packages

### Check Installed Packages

```python
import sys
from pkg_resources import get_distribution, DistributionNotFound

def test_package(package_name):
    """Test if package exists and returns version or -1"""
    try:
        version = get_distribution(package_name).version
    except (DistributionNotFound, ImportError) as err:
        version = '-1'
    return version

# Colab setup ------------------
if 'google.colab' in sys.modules:
    !pip install pyTEMlib -q
# pyTEMlib setup ------------------
else:
    if test_package('sidpy') < '0.0.5':
        print('installing sidpy')
        !{sys.executable} -m pip install  --upgrade pyTEMlib -q
    if test_package('pyTEMlib') < '0.2021.4.20':
        print('installing pyTEMlib')
        !{sys.executable} -m pip install  --upgrade pyTEMlib -q
# ------------------------------
print('done')
```

    done

### Import all relevant libraries

Please note that the EELS_tools package from pyTEMlib is essential.

```python
import sys
if 'google.colab' in sys.modules:
    %pylab --no-import-all inline
else:
    %pylab --no-import-all notebook
    %gui qt

import warnings
warnings.filterwarnings('ignore')

from scipy.ndimage.filters import gaussian_filter

## import the configuration files of pyTEMlib (we need access to the data folder)
import pyTEMlib
import pyTEMlib.file_tools as ft
import pyTEMlib.eels_tools as eels

# For archiving reasons it is a good idea to print the version numbers out at this point
print('pyTEM version: ',pyTEMlib.__version__)

```

    Populating the interactive namespace from numpy and matplotlib
    pyTEM version:  0.2021.04.20

## Chemical Shift

The chemical shift is the first feature that we discuss in respect of the shape or appearance of the ionization edges: the energy-loss near edge structure (ELNES).
This section and the following one explain how to do simple analysis of near--edge features.

The chemical shift refers to small changes (up to a few eV) of the edge onset, and how this shift depends on the bonding of an element in a solid.
Going back to figure in the [Introduction to Core-Loss Spectra](CH4_07-Introduction_Core_Loss.ipynb), we see that such a change can be caused by a change of the band-gap, in which case the final states are moving or by a movement of the core-levels (initial states).

Please note, that this explanation above is a simplification; what we measure is the energy difference of an excited atom to one in ground state. In the excited atom all states react to the new electronic configuration and not only the final and initial states. In fact, to calculate the energy-difference, one cannot use the difference between core-levels and bottom of the conduction band.

However, we want to know which of the above effects (band gap changes or core-level shift) is the major one, so that we can conclude back on bonding of the element in question.

As an example of chemical shift we look at reference data of the silicon L$_{2,3}$ edge.

### Load reference data

```python
Si_L_reference_spectra = eels.get_spectrum_eels_db(element='Si',edge='L2,3')
```

    a_SiO2_Si_L3_S_Schamm_63
    a_SiO2_Si_L3_S_Schamm_64
    Si_Si_L3_S_Schamm_58
    Si3N4_Si_L3_Lingyang_Li_166
    Si3N4_(alpha)_Si_L3_S_Schamm_61
    Si3N4_(alpha)_Si_L3_S_Schamm_62
    SiC(6H)_Si_L3_S_Schamm_65
    SiC(6H)_Si_L3_S_Schamm_66
    found 8 spectra in EELS database)

### Plot silicon spectra

```python
plt.figure()
for name, spectrum  in Si_L_reference_spectra.items():
    if 'Core' in  spectrum['TITLE'] or 'L3' in  spectrum['TITLE']:
        #plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = spectrum['TITLE'])
        pass

for name, spectrum  in Si_L_reference_spectra.items():
    if 'a_SiO2_Si_L3_S_Schamm_63' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: a-SiO$_2$')
    if 'Si_Si_L3_S_Schamm_58' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: Si')
    if 'Si3N4_(alpha)_Si_L3_S_Schamm_62' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: Si$_3$N$_4$')
    if 'SiC(6H)_Si_L3_S_Schamm_66'in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Si-L$_{2,3}$: SiC')

plt.legend();

```

    <IPython.core.display.Javascript object>

The shift of the edges as above can be caused by the intial and/or the final states.

### Band gap
The band gap changes are treated in the solid state theory of band structure and are, therefore, well covered in other textbooks. The trend is that with increased oxidation (or more general increased electronegativity of the reaction partner as in the series: B, C, N, O), the band gap opens and the edge shifts to higher energies.
This is seen in the figure above,  where the onset of the Si-L$_{2,3}$ edge shifts to higher energies with Pauli electron--negativity of the reaction partner.

In fact, one can monitor band gap changes with stoichiometry at interfaces by the shift of the edge.
Please be aware that we see only the shift of the conduction band bottom and not the whole band gap change. This effect of the band gap is obvious between Si and SiO$_2$, where the edge shifts by about 6.5 eV.

### Core-level shift
The initial state, the ``core-level``,  can also shift, for example after oxidation. Some electrons will transfer to an anion (for example oxygen) and less electrons are available to fill the band structure. This is shown below for the case of Cu and its two oxides Cu$_2$O and CuO.

The more electrons transfer to oxygen for the ionic bonding of these materials, the more the edges shift to lower energies, even though a band gap opens up. The  opening up of the band gap will cause a shift too higher energies and counteracts the effect of ionization. Due to lower electron densities, at the Cu atoms in the oxides, the core levels are assumed to shift to higher energies (see below) and compensate a little for the effect.

>
>
>

The core-level shift is generally a small effect. This core states react to increase of electron density at the atom site with a decrease and vice versa. Simplified , we can think of the core level electrons getting repulsed from an increased electron density (through Coulomb interaction) and pushed closer (lower in energy) to the core.

```python
Cu_L_reference_spectra = eels.get_spectrum_eels_db(element='Cu',edge='L2,3')
```

    Cu_Cu_L3_Y_Kihn_124
    Cu2O_Cu_L3_Y_Kihn_129
    Profile Of Cu4O3-Cu L23
    CuO_Cu_L3_Y_Kihn_127
    found 4 spectra in EELS database)

```python
plt.figure()

for name, spectrum  in Cu_L_reference_spectra.items():
    if 'Cu_Cu_L3_Y_Kihn_124' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Cu-L$_{2,3}$: Cu')
    if 'CuO_Cu_L3_Y_Kihn_127' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Cu-L$_{2,3}$: CuO')
    if 'Cu2O_Cu_L3_Y_Kihn_129' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'Cu-L$_{2,3}$: Cu$_2$O')

plt.legend();
plt.xlim(910, 980)

```

    <IPython.core.display.Javascript object>

    (910.0, 980.0)

The case of the oxidized Cu where the slightly oxidized Cu$_2$O does shift to slighly lower energies compared to pure Cu,
the shift to lower energies due to ionic bonding is a little larger than the opening of the band gap and the core level shift to higher energies (further away from the nucleus) because of less Coulomb repulsion from the valence electrons.

This effect is even more pronounced for CuO with an larger band gap than Cu$_2$O.

In the figure below we see that the Si-L$_{3,2}$ edge shifts to higher energies with increased Ge content, when the spectra are taken from different locations at the diffuse Si - Ge interface. Intuitively, we would expect a shift to lower energies, because the band gap of SiGe alloys and of Ge is smaller than Si.

*We see that as we acquire spectra at the diffuse interface between Si and Ge, that the Si-L${_3,2}$ edge shifts to higher energies. This is surprising as SiGe and Ge posses a smaller band gap than Si and one would expect the opposite.*

This shift can be explained by a shift of core levels. An all electron calculations can determine the 2p$_{3/2}$ core levels of an atom in a compound which is shown for Si-Ge and Si-GaAs alloys in the figure below. The calculations show that there is a core level shift to lower energies with increased Ge and GaAs content. Ge and GaAs will add additional electrons to the electronic structure and the Coulomb repulsion between core level electrons and valence electrons will increase, pushing the core levels to lower energies.

*All electron ab initio calculations of the core level states for Si-Ge and Si-GaAs alloys with different composition. The calculations show an 2p$_{3/2}$ core level shift to lower energies with deviation of composition from pure Si (on the left).*

The shift of the core--levels to lower energies will increase the distance between core--level and conduction band bottom, which results in a simple picture in a shift to higher energies. We see that for pure Si, the 2p$_{3/2}$ core level is at about 89 eV but the edge is at 99.8 eV. The difference in energy is caused by relaxation of valence and core electrons. Effectively, we measure with the EELS edge onset  the energy difference between an excited atom and an atom in its ground state.

All electrons will relax according to the overall electron density at the atom sites and the calculated core--level shifts can not be used for predicting the edge shifts.  However, these calculations can explain the physical origin of the edge shift.

### Conclusion
In summation, we can say that the following effects (strongest first, weakest last) cause a chemical shift:

- band gap opening
- ionic bonding
- core level shift

All of these effects can be present at once, but usually  only one dominates the chemical shift.

## White Line

In this section, we try to analyze a distinct feature of the transition metal elements. The d-states of transition metal elements form a very flat band in the band structure. This flat band, creates a strong peak in the density of states. This analysis is based on the following simplification:

In the figure below, we see a s or p free electron like band in the general shape of a parabola, This parabola gives rise to a saw tooth like feature in the density of states (DOS) because flat bands have a higher density of states than steep ones. The DOS of the conduction band (above the Fermi level) is closely related to our EELS spectra. A flat d-band will cause a very prominent peak, a so called white line (in the age of photographic recording, these peaks appeared as white lines).

 *A schematic of the relationship of density of states (DOS on the left) and band structure of a transition metal element on the right). The s and p free electron like bands (parabola) give rise to saw tooth like feature in the DOS and the flat d bands (red) cause a sharp peak in the DOS.*

Since these d-bands are so prominent we can easily separate them from the rest. In figure \ref{fig:Ti-whiteline} we use the calculated cross--section as a model of the s and p free electron like states. After a subtraction, we get the pure d-band contribution.

*We use the cross section of  the Ti-L$_{2,3}$ edge (green) as a model for the free electron gas and subtract it from the experimental Ti-L$_{2,3}$ edge (red). The residual peaks (blue) can be analyzed as pure d-states. The two double peaks of  the Ti-L$_{2,3}$ edge indicate that there is some structure to the d-bands (here crystal field splitting).*

A simple analysis of the white line ratios of Ti-L$_3$ to Ti-L$_2$ of SrTiO$_3$ yields a intensity ratio of 242 / 314 = 0.8. However, just considering the initial states (and assuming the transition probability ( or more accurately the transition matrix elements) are the same for both edges) with 4 electrons in p$_{3/2}$ and 2 electrons in p$_{1/2}$ would let us expect a ration of 2 to 1.

>Please, note that both the Ti-L$_3$ and Ti-L$_2$ edge are split in two. We will discuss this crystal field splitting in chapter \ref{sec:Titanium} as an ELNES feature. Here we just consider the sum over the the whole Ti-L$_3$ or/and Ti-L$_2$ and ignore this splitting.

The deviation from the 2 : 1 white line ratio is assumed to be caused by J-J coupling, and is, therefore, symmetry dependent. The anomalous white line ratios have been used to determine valency of transition elements in compounds. Physically this approach is on shaky ground, because we do not know all the reasons for the change in the ratios, it has, however, been shown to be reliable for binary metallic alloys.

Fortunately, there is a easier method (from the physical point of view). We compare the total amount of white line intensities (which corresponds to the empty d-states) and normalize them by the free-electron gas like intensity beyond the white lines.

We use the method of Okamoto et al. \cite{Okamoto-Disko1992}.

The energy-window for the free-electron like part of the edge can be chosen arbitrarily and consistently.
Following Okamoto et al., a  50 eV  integration window should be used 50 eV beyond the edge onset. These will allow to compare the results to values in the literature.
The edge has to be taken in very thin areas and if possible corrected for single scattering distribution, because otherwise the free electron like part contains plasmon contributions, which change the analysis.

For the above spectrum in the figure  above, we get for the white line / free electron gas ratio (50 eV - 50 eV beyond edge onset) =  556 / 974 = 0.57. Ti in SrTiO$_3$ can be considered as Ti$^{4+}$ with no  electrons in the d-bands, but using this ratio in the paper of Okamoto et al.  would yield a d band occupancy of 4 as in a metal. The difference may lay in the usage of a Hatree-Slater X-section for the analysis while Okamoto et al. use a hydrogenic one. Also, the SrTiO$_3$ spectrum was presumably taken under completely different acquisition conditions than Okamoto's spectra.
\
For example, the SrTiO$_3$ spectrum was not corrected for convergence angle, even though it was acquired in Z-contrast mode. Another source of error, is of course the background fit, which could change especially the free electron integration result. The fact that the SrTiO$_3$ spectrum was not corrected for single scattering distribution may also overestimate the free electron gas contribution, even though the spectrum was taken in a every thin area.

For TiO$_2$ spectrum of the core-loss atlas I get for the white line / free electron gas 256 / 494 = 0.52. TiO$_2$ contains also only Ti$^{4+}$. This is the level of agreement we can expect, if we use two spectra with completely different acquisition parameters.

In the plotof the Cu-L edges above, we can see that Cu has no empty d-states but with oxidation the d-bands get unoccupied and white lines appear. The more electrons get transferred to the oxygen neighbors, the more empty d-states and the more prominent the white lines will appear.

This analysis of the occupancy of d-states is extremely important for magnetic materials where the strength depends on the unpaired (d-band or f-band) electrons.

The same analysis can be done for the empty f bands of M--edges, which are also rather flat. Usually, the M$_{4,5}$ and the M$_{2,3}$ edges form doublets of white lines.

## ELNES

So far, we have only interpreted distinct features of the shape of the ionization edges. A general approach is to look at the shape of the edges in total and use this shape as a kind of fingerprint for the interpretation. Another one is to try to understand the different features as means of electronic structure calculations of various sophistication.

In order to understand the different approaches ( and their level of confidence in the results), we will discuss the most important edges one by one.

The shape of the ELNES is closely related to the density of states of the conduction band. The next chapters discuss the basics for an electronic structure interpretation of ELNES

### Transition matrix and electronic structure
The single scattering intensity of an ionization edge $J_k^1(E)$ is related to the band structure through Fermi's Golden Rule: The transition rate is proportional to the density of final states $N(E)$ multiplied with the square of an atomic transition matrix $M(E)$

 \begin{equation} \Large
 J_k^1(E) \propto |M(E)|^2 N(E)
 \end{equation}

  The transition matrix describes the transition probability between the core states and the final states (given by $N(E)$). Because the transition probability generally decreases with higher energies (above the edge threshold, the transition matrix gives the overall shape of the edge (sawtooth) and can be determined by atomic physics.

  The density of final states (conduction band) ($N(E)$) expresses the chemical environment and its symmetry.
  Because the core--levels are highly localized the final states $N(E)$ present the local density of states. This localization causes a different shape for different elements in the same compound, even if they are nearest neighbors (with a distance of  only a few Angstrom). The $N(E)$ will of course be different for elements in materials with different (local) symmetry, coordination or chemical composition.

### Life-time broadening
  For arbitrary excitations, the $N(E)$ is the joint density of states, which means a convolution of the initial and the final states. The density of final states $N(E)$ is broadened in the spectrum by the energy resolution of the experimental  setup $\delta E$ and the width of the initial state $\Gamma_i$. $\Gamma_i$ can be approximated with the uncertainty principle:

  \begin{equation} \Large
  \Gamma_i \pi_h \approx \hbar
  \end{equation}

   The lifetime of the core -hole $\pi_h$ is determined how fast the core--hole is filled and the additional energy is dissipated through emission of Auger electrons (for light elements) or X-ray photons (heavier atoms). The value of $\Gamma_i$ depends on the threshold energy of the edge and is calculated to be around 0.1 and 2 eV for K-edges of the first 40 elements.

 Further broadening of the $N(E)$ is induced by the lifetime of the final states $\pi_f$. The inelastic mean free path of the ejected electron is only a few nm (assuming a kinetic energy of less than 50eV).
  Using the free electron approximation ($E_{kin} = m_0 v^2 / 2$), we get for the energy broadening  of the final states:
  \begin{equation} \Large
  \Gamma_f \approx \frac {\hbar}{\pi_f} = \frac{\hbar v}{\lambda_i } = \frac{\hbar}{\lambda_i} \sqrt{\frac{2E_{kin}}{m_0}}
  \end{equation}

  Since the inelastic mean free path $\lambda_i$ varies inversely with kinetic energy $E_{kin}$ below 50 eV (and raises only slightly above 50 eV), the observed density of state structure is more and more broadened the further the distance from the onset.

  The next two chapters discuss the density of final states $N(E)$ and the transition matrix $M(E)$ in detail.

### Dipole-selection rule

Assuming a single electron approximation (and almost no electronic theory theory solves the many-particle problem fully) for the excitation, we can replace the many electron transition matrix elements with single electron matrix elements $M(\vec{q},E)$.

\begin{equation} \Large
M(\vec{q},E) =  <f|\exp(i\vec{q}\vec{r})|i>
 \end{equation}

 with the initial wave function $|i> = \phi_i$ and the complex conjugated final wave function $<f|=\phi_f$.
 The integration is over the volume $\tau$ surrounding the initial state (and we get a high localization again).

 We now expand the operator $\exp(i\vec{q}\vec{r})$ as :

\begin{equation} \Large
\exp(i\vec{q}\vec{r}) = 1 + i\vec{q}\vec{r} + \mbox{higher--order terms}
\end{equation}

Because the final and initial wave functions must be orthogonal to each other, the first term gives zero.
The higher--order terms are assumed to be negligible.

The second term has odd symmetry and determines the dipole selection rule:
- if the initial and final wave functions are both even or odd, then the result is zero.
- only if one of the  initial and final wave are odd and one of them is even the result is not zero.

 There is no transition probability for s to s transitions or p-p transitions or d-d or s-d transitions.
 However, transitions are possible from s to p states and from p to s and d states.

 The (optical) dipole selection rule says the angular quantum number $l$ of the initial and final state must differ by 1: $\delta l = \pm 1$.

 For the dipole selection rule to be valid, obviously the higher order terms must be very small, otherwise a third integral (of the dipole--forbidden reflections) will modify the observed ELNES.

 A rough estimates leads to the requirement $\vec{v}\vec{r} <<1$ for all $r$, which is equivalent to $ q<<q_d = 1/r_c$ where $r_c$ is the radius of the core state. This radius $r_c$ denotes the volume where the transition occurs.
 And we get for the carbon K-shells ($Z^* = Z- 0.3$) and 100kV acceleration energy that the dipole selection rule should be applicable for $\theta << \theta_d = 63 mrad$. For the Si L$_{3,2}$ edge the dipole selection rule should prevail up to about 12 mrad.

 Therefore, we must use rather small aperture ( $ < 10 - 20$ mrad) centered at the zero beam to insure the dipole selection rule to be valid.

### Core--hole effects and excitons

 So far, we can say that our ELNES can be calculated by the site and angular momentum projected density of states. This is true for almost all metals, but the core-hole effects (through excitonic interactions of the core--hole with the excited electron and valence electrons)  further modifies the final density of states $N(E)$.

The missing electron in the core hole and the extra electron in the conduction band causes changes in the band structure (in the core, valence and conduction states regions).
These relaxation processes of all the electrons involved can be rather complicated and we have to know when these complications have to be applied.

*A method to determine the edge onset from XPS data and the band gap, if core--hole effects are  negligible. This is only true for metals with an almost complete screening of the core--hole.*

As illustrated in the figure above (Pantelides, PRL 1975), if there is no influence of the core-electron, the edge onset of the ionization edge should be determined by:
\begin{eqnarray}
E_{\mbox{onset}} &=& E_{\mbox{core-state}} - E_{\mbox{CB}}\\
 &=& (E_{\mbox{core-state}} - E_{\mbox{vacuum}} ) + (E_{\mbox{vacuum}} - E_{\mbox{CB}} )\\
 &=& (E_{\mbox{core-state}} - E_{\mbox{vacuum}} ) + (E_{\mbox{vacuum}} - E_{\mbox{VB}} ) +(E_{\mbox{VB}} - E_{\mbox{CB}} )
\end{eqnarray}

The distances between core--states and vacuum and valence band (VB) and vacuum are directly accessible with XPS, while the band gap ($E_{\mbox{VB}} - E_{\mbox{CB}} $) is available from optical spectroscopy.

Therefore the calculated density of states not including core holes should start at that energy, If the agreement is not In all other cases, we must assume that the (excitonic) core-hole effects modify the $N(E)$.

This modification of $N(E)$ can be compared to doping effects in semiconductors as shown in the figure below, where the whole band structure shifts due to the extra electron. This shifting of bands (up and down) is due to the different screened potential of an excited atom compared to an atom in ground state.

 *Comparison of doping and core--hole effects on the electronic structure. Both effects modify the band structure.*

The modifications of the core hole effect not be predicted without extensive calculations.
Two methods have profen to be successful for ELNES calculations:
- explicit core hole in an all electron calculation (like FLAPW)
- Z+1 approximation in pseudo potential calculations (like VASP)

In the explicit core-hole approach, we just take one electron from a core level of one atom in the supercell and place it (energetically) on top of the other electrons of that (now excited) atom.
Now, the electronic structure calculations give the ELNES as the local and angular projected density of states of the excited atom.

In the Z+1 calculation, the atom with the explicit core-hole is replaced by an atom with an atomic number increased by one. The core hole is simulated by an nuclear charge that is increased by one and the excess electron is there naturally. This Z+1 approximation can be used in combination with the much faster pseudo potential methods. The results of this approximation are very close to the explicit core hole simulations.

## ELNES Examples

### Carbon

Carbon has 4 valence electrons.\\
As a free atom has two electron in 2s and two electron in 2p states.\\
In a compound, it is easier to understand the bonding assuming that there is only one electron in the 2s state and three electrons in the 2p state.

Carbon can bond in three modes and as elemental material stays in two modification:
 - Diamond:  sp$_3$ hybridization
 - lonsdaleite: sp$_3$ hybridization
 - Graphite:  sp$_2$ hybridization

The energy levels of the 2s and 2p levels are so close together that they can hybridize (as many elements in the first row of the periodic table). Hybridization means the states are indistinguishable (in energy). Carbon is unique in a sense as it can hybridize so that 2, 3 or 4 electrons hybridize. The 1s (occupied by one electron) and two or all three p states hybridize into states with p character, which means they are directional and form covalent bonds.

In the case of the sp$_2$ bonds, one of the p electrons does not contribute to bonding and is de-localized. This de-localized electron is the reason for graphite to be conducting, while in diamond all electrons are involved in covalent bonding and consequently diamond is an insulator. This extra (non-hybridized) 2p electron does the double bonding in molecules and polymers.

*Comparison between  $sp_2$ and $sp_3$ hybridization. The  $sp_2$ hybridization causes a layered structure and a de--localized electron. The  $sp_3$hybridization causes a bond angles of 109.5${\circ}$.*

Graphite has three covalent bonds and forms a layered structure with three nearest neighbors within the layers \ref{fig:sp2}. Within the layer, the $sp_2$ hybridization does the covalent bonding. The layers are bonded together with the weak Van-der-Waals force. Diamond has 4 covalent bonds and forms with its 4 neighbors a tetrahedron. The bond angle in diamond are 109.5 and fill all three dimensions.

```python
C_reference_spectra = eels.get_spectrum_eels_db(element='C',edge='K')
```

    8-pm
    8-pm
    B4C_B_K_Otmane_Abidate_41
    B4C_C_K_Otmane_Abidate_42
    C_C_K_V_Serin_17
    C_C_K_M__Jaouen,_G__Tourillon_20xray
    C_C_K_M__Jaouen,_G__Tourillon_21xray
    C_C_K_Otmane_Abidate_24
    C_C_K_Odile_Stephan_226
    C_C_K__243
    C_(graphite)_C_K_David_Muller_(at_Cornell_University)_1995_23
    7-pm
    E113 250-2-cal-kgd-pm-dm
    e113 k-pm
    255-1-pmmod
    C3N6H6_C_K_Susana_Trasobares_114
    C3N6H6_C_K_Susana_Trasobares_115
    CN_C_K_Susana_Trasobares_122
    Cr3C2_C_K_R__Leapman_11
    Cr7C3_C_K_R__Leapman_13
    CrC_C_K_Christina_Scheu_176
    FeCO3_O_K_A__Gloter,_A__Chen_206
    MgCO3_O_K_Irmgard_Rom_43
    MgCO3_C_K_Irmgard_Rom_44
    SiC_beta_Si_K_Flank_30xray
    found 24 spectra in EELS database)

```python
plt.figure()
for name, spectrum  in C_reference_spectra.items():
    if 'Core' in  spectrum['TITLE'] or 'L3' in  spectrum['TITLE']:
        #plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = spectrum['TITLE'])
        pass

for name, spectrum  in C_reference_spectra.items():
    if 'C_(graphite)_C_K_David_Muller_(at_Cornell_University)_1995_23' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: graphite')
    if 'C_C_K_M__Jaouen,_G__Tourillon_21xray' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: diamond')
    if 'C_C_K__243' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: ')
    if 'CN_C_K_Susana_Trasobares_122' in  spectrum['TITLE']:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'C-K: CN')
plt.legend()
plt.xlim(275, 310)
```

    <IPython.core.display.Javascript object>

    (275.0, 310.0)

Looking at the bonding of carbon in molecular orbital theory or (its predecessor) Ligand--Field theory the  non--hybridized p electron in graphite will form an occupied $\pi$ bond and an unoccupied $\pi^*$ bond. In figure \ref{fig:C-K} we see that the unoccupied $\pi^*$ state is visible in the graphite spectrum.
In diamond, there is no molecule like p electron and consequently there is no $\pi$ or $\pi^*$ bond.
The appearance of the $\pi^*$ state in a carbon spectrum is used as a fingerprint for $sp_2$ hybridization. In the case of so called diamond like carbon, an amorphous carbon with $sp_2$ and $sp_3$ bonds, the quality (amount of $sp_3$ bonds) of the diamond like carbon can be assessed by the  intensity of the  $\pi^*$ peak (or rather the lack of it).

Both spectra have a $\sigma_1^*$ and $\sigma_2^*$ peak which are associated with the molecule like s states.
The C-K edge should show only the p density of states due to the dipole selection rule. The $\sigma^*$ states show up in the C-K edge because of these states are already related to the s-p like free electron gas density of states (s-p hybridization) above the edge. The $\sigma^*$ states are the non-bonding states of the ( $sp_2$ or $sp_3$) hybridized states and are, therefore, present in any carbon compound.

The quantification of $sp_2$ versus $sp_3$ hybridization is also important in polymers (where the non hybridized p electron in a $sp_2$ configuration forms the  conducting double bonds. In Buckminster fullerens (bucky balls) and carbon nanotubes the $sp_3$ hybridization is always associated with a defect (dislocation like), where a carbon atom has now 4 nearest neighbors.

### Silicon

The calculation of the transition matrix $M(E)$ for
the Si-L$_{3,2}$ edge shows that the intensity of the ELNES consists almost exclusively of d-states. Less than 5\% of the intensity is from the also dipole--allowed s-DOS.

We can, therefore, assume that only d-states form the Si-L$_{3,2}$ ELNES.
The spin orbit splitting of the initial p states is 0.7 eV, and this means that the L$_3$ and the L$_2$ are separated by 0.7 eV, which cannot be resolved with most instrumental setups.
To the calculated (local) d-DOS the same DOS has  to be added (with a ratio of about 2:1)  but shifted by 0.7 eV.

*Comparison of experimental and theoretical data. While an effective mass exciton would explain the sharp raise, the effect is to small, the electronic structure  calculation without core hole effect, placed at the correct onset, does not agree with the experiment.*

The edge onset of the Si-L$_{3,2}$ of pure Si should be at 100 eV without core-hole effects.  A d--Dos calculated without influence of a core--hole is shown in figure \ref{fig:Si-L-pure} beginning at this value. We can clearly see that this DOS of state cannot reproduce the experimental ELNES. From this disagreement between experiment and DOS without core-hole effect, we conclude that the
	core-hole effects must be included.

The main feature of the Si-L$_{3,2}$ of pure Si is the extreme sharp raise of the edge at the onset.
This feature cannot be explained by the d-DOS calculated without core--hole effects, which raises not as steeply as the experimental ELNES.

This steep raise is another indication of the core--hole and must have its origin in an excitonic effect (an interaction of the excess electron in the conduction band an the hole in the core state).
In the figure above, the calculations of an effective mass electron (due to an state that is created just below the conduction band) is compared to the experimental ELNES. Such an effective mass electron must be considered delocalized. We see that the raise is steep enough to explain the experimental raise, we also see that the effect (intensity) is not change the ELNES.

Only the explicit inclusion of the core--hole or the Z+1 calculations in figure \ref{fig:Si-L-pure2} can explain this steep onset. We can, therefore, conclude on localized excitonic enhancement of the states at the bottom of conduction band. This is a rather localized excitonic effect.

We can also see in the comparison of the explicit inclusion of the core-hole and the Z+1 approximation that both simulations lead to the same ELNES, however only the explicit core-hole calculation can predict the exact intensity (cross section) of the Si-L$_{3,2}$ edge.

The same calculations are also successful for SiO$_2$ (quartz) as can be seen in figure \ref{fig:Si-L-sio2}. The experimental data show the spin--orbit splitting in the first peak, all other features are to smeared out to show a clear distinction between transitions originating from $2p_{3/2}$ and 2p$_1/2$.
Due to the simple addition of the shifted spectra, the splitting in the first peak is reproduced rather easily and cannot be used for further analysis. Again, this edge is completely dominated by the local d-DOS.

### Oxygen and Nitrogen

Oxygen and nitrogen edges are usually very similar. Here we will discuss mostly the oxygen edge, but this can be easily transferred to nitrogen.

The Si-SiO$_2$ interface shows oxygen deficiency in the oxide at the interface. In the following, I will show that the oxygen K edge ELNES cannot be used to probe the oxygen deficiency. Experimentally, the oxygen K edge has a chemical shift by about 1 eV. The structure of the edge is washed out at the interface as shown in figure . Higher spatial resolution experiments by Muller (nature 2003) show a completely structureless O-K edge. Simulation of the O-K edge show that this shift and the featureless structure is due to the dimer like structure (Si-O-Si) which is not avoidable at any Si-SiO$_2$ interface.

Another approach is the so called "finger-print" method. In this method, one compares edges from different but known materials and hopes that similar features are conclusive for different coordinations within the unknown material. This approach can be improved by using simulations of the ELNES of the edges as seen in the plot below.

```python
O_reference_spectra = eels.get_spectrum_eels_db(element='O',edge='K')
```

    (La0_7,Sr0_3)MnO3_O_K_imhoff_229
    8-pm
    Al2O3_Al_K_Flank_1xray
    Al2O3_(alpha)_Al_K_Stefan_Nufer_53
    AlPO4_Al_K_Flank_2xray
    B2O3_B_K_Karine_Varlot_8xray
    B2O3_B_K_Karine_Varlot_9xray
    7-pm
    E113 250-2-cal-kgd-pm-dm
    e113 k-pm
    255-1-pmmod
    CaCoO NT-2
    CaTiO3_Ti_L3_Saso_Sturm_133
    CoO_O_K_dominique_IMHOFF_235
    Cr2O3_(alpha)_O_K_Rik_Brydson_158
    Cu2O_O_K_Y_Kihn_130
    Profile Of Cu4O3-O-K
    CuO_O_K_Y_Kihn_126
    Fe2O3_(alpha)_O_K_A__Gloter,_A__Chen_201
    Fe2TiO5_Ti_L3_E_Fries_P_Perriat_36
    Fe3O4_O_K_A__Gloter,_A__Chen_205
    FeCO3_O_K_A__Gloter,_A__Chen_206
    FeOOH_O_K_A__Gloter,_A__Chen_204
    FeOOH__(alpha)_O_K_A__Gloter,_A__Chen_203
    FeOOH__(beta)_O_K_A__Gloter,_A__Chen_202
    lsmo2
    APL_article_LSMO_oxygen2
    APL_article_interface_oxygen2
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    LaCoO3_O_K_Irmgrad_Rom,_Ferdinand_Hofer_162
    MgCO3_O_K_Irmgard_Rom_43
    MgCO3_C_K_Irmgard_Rom_44
    MgO_O_K_Giovanni_Bertoni_112
    MgO_Mg_K_Giovanni_Bertoni_113
    MgO_O_K_dominique_IMHOFF_233
    MgO_Mg_K_dominique_IMHOFF_234
    Na2O_Na_K_Flank_34xray
    Nb2O5_Nb_M_David_Bach,_Wilfried_Sigle_215
    NbO2_Nb_M_David_Bach,_Wilfried_Sigle_217
    SiO2_Si_K_Flank_27xray
    SiO2_Si_K_Flank_28xray
    SiO2_Si_K_Flank_29xray
    Spectre centre EELSDB
    SrTiO3_O_K_imhoff_232
    srtio32
    APL_article_STO_oxygen2
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    y
    EELS Spectrum Image (high-loss)model from decomposition with 6 components
    TiO2_(amorphous)_O_K_Giovanni_Bertoni_106
    TiO2_(anatase)_O_K_Giovanni_Bertoni_103
    Y3Al5O12_O_K_Saso_Sturm_182
    Y3Al5O12_Al_K_Saso_Sturm_213
    Y4Al2O9_O_K_Saso_Sturm_178
    Y4Al2O9_Al_K_Saso_Sturm_179
    YAlO3_O_K_Saso_Sturm_186
    YAlO3_Al_K_Saso_Sturm_187
    ZnO_O_K_Wilfried_Sigle_169
    found 55 spectra in EELS database)

```python
O_reference_titles = ['SrTiO3_O_K_imhoff_232',
                      'MgO_O_K_Giovanni_Bertoni_112',
                      'ZnO_O_K_Wilfried_Sigle_169',
                      'Cr2O3_(alpha)_O_K_Rik_Brydson_158'
                     ]
O_reference_materials = ['SrTiO$_3$', 'MgO', 'ZnO', 'Cr$_2$O$_3$']
plt.figure()
i = 0
for name, spectrum  in O_reference_spectra.items():
    if spectrum['TITLE'] in O_reference_titles:
        plt.plot(spectrum['enery_scale'],spectrum['data']/np.max(spectrum['data']), label = 'O-K:'+O_reference_materials[i])
        i+=1

plt.legend();
plt.xlim(525,570)

```

    <IPython.core.display.Javascript object>

    (525.0, 570.0)

A comparison hows that the cubic oxides MgO and MgAl$_2$O$_4$ (spinel structure) have a similar ELNES which differs from the ELNES from the rhombohedral oxide Al$_2$O$_3$.

Correlation effects of valence electrons cause the so called Hubbard band. These bands dominate the electronic structure in high T$_c$ semiconductors, giant magnet resistors and other materials with high correlations.

In the figure below, we see that such a correlation effect takes place in the giant magneto-resistor LaMnO$_3$ but not in the perovskite LaAlO$_3$, which does not have this splitting of the d bands (Al does not have d electrons but Mn has), a precondition for the formation of a Hubbard band.

*The O-K edge in LaMnO$_3$ has a pre-peak that is interpreted as Hubbard band and is not present in LaAlO$_3$ even so it has the same structure.*

The same Hubbard band is visible in the high T$_c$ superconductor YBa$_2$Cu$_3$O$_{7-\delta}$.
In figure \ref{fig:O-K-YBCO-DL}, we see the probing of this Hubbard band at a dislocation in YBa$_2$Cu$_3$O$_{7-\delta}$.

*The O-K edge at a dislocation in YBa$_2$Cu$_3$O$_{7-\delta}$ has less of the signature of a Hubbard band than far away in the bulk. This lack of holes in the dislocation  disturbs the superconductivity and is utilized in grain boundaries for Josephson junctions but is detrimental in polycrystalline high T$_c$ wires.*

The prepeak that is caused by the Hubbard band is reduced in the vicinity of the dislocation and vanishes completely within the dislocation core. This lack of holes in the dislocation  disturbs the superconductivity and is utilized in grain boundaries for Josephson junctions but is detrimental in polycrystalline high T$_c$ wires.

## Spatial Resolution in EELS
% images/spat-dif-resolution.jpg}

### Methods to achieve spatial resolution
- Spot Mode
- Area Mode = Spatial Difference
- Lines Scan Mode
    - Each Spot of Line one Spectrum
    - Each Segement of Line one Spectrum
    - Each Area of Line one Spectrum
- Spectrum Imaging

### Spot mode
% {images/spat-dif-spot.jpg}

### Spatial difference
% images/spat-dif-spatdif1.jpg
% images/spat-dif-spatdif2.jpg
% images/spat-dif-spatdif3.jpg
*EELS at Bi  doped  Cu grain boundary*
% images/spat-dif-spatdif4.jpg
% images/spat-dif-ls1.jpg
*As segregation at Si/SiO$_2$ interface*
% images/spat-dif-ls2.jpg
*As segregation at Si/Sio$_2$ interface*

### Energy Filtered Imaging (EFTEM)
% images/spat-dif-eftem1.jpg}
% images/spat-dif-eftem2.jpg}
% images/spat-dif-eftem3.jpg}

## Summary

The core--loss part of the electron energy--loss spectrum allows us to determine:
- chemical composition
- bonding
- magnetic moment through ionic charge

>
>	with high spatial resolution!!!
>

## Navigation
-   **Up Chapter 4: [Imaging](CH4_00-Spectroscopy.ipynb)**
-   **Back: [Analysis of Core-Loss](CH4_09-Analysis_Core_Loss.ipynb)**
-   **List of Content: [Front](../_MSE672_Intro_TEM.ipynb)**

```python

```


# [Math-Bot] Siamese LSTM:  Detecting duplicates

Author: Alin-Andrei Georgescu 2021

Welcome to my notebook! It explores the Siamese networks applied to natural language processing. The model is intended to detect duplicates, in other words to check if two sentences are similar.
The model uses "Long short-term memory" (LSTM) neural networks, which are an artificial recurrent neural networks (RNNs). This version uses a custom embedding layer with weigths based on GloVe pretrained vectors.

## Outline

- [Overview](#0)
- [Part 1: Importing the Data](#1)
    - [1.1 Loading in the data](#1.1)
    - [1.2 Converting a sentence to a tensor](#1.2)
    - [1.3 Understanding and building the iterator](#1.3)
- [Part 2: Defining the Siamese model](#2)
    - [2.1 Understanding and building the Siamese Network](#2.1)
    - [2.2 Implementing Hard Negative Mining](#2.2)
- [Part 3: Training](#3)
- [Part 4: Evaluation](#4)
- [Part 5: Making predictions](#5)

### Overview

General ideas:
- Designing a Siamese networks model
- Implementing the triplet loss
- Evaluating accuracy
- Using cosine similarity between the model's outputted vectors
- Working with Trax and Numpy libraries in Python 3

The LSTM cell's architecture (source: https://www.researchgate.net/figure/The-structure-of-the-LSTM-unit_fig2_331421650):

I will start by preprocessing the data, then I will build a classifier that will identify whether two sentences are the same or not.

I tokenized the data, then split the dataset into training and testing sets. I loaded pretrained GloVe word embeddings and built a sentence's vector by averaging the composing word's vectors. The model takes in the two sentence embeddings, runs them through an LSTM, and then compares the outputs of the two sub networks using cosine similarity.

This notebook has been built based on Coursera's Natural Language Processing Specialization.

# Part 1: Importing the Data

### 1.1 Loading in the data

First step in building a model is building a dataset. I used three datasets in building my model:
- the Quora Question Pairs
- edited SICK dataset
- custom Maths duplicates dataset

Run the cell below to import some of the needed packages.

```python
import os
import re
import nltk
from nltk.corpus import stopwords
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

import numpy as np
import pandas as pd
import random as rnd

!pip install textcleaner
import textcleaner as tc

!pip install trax
import trax
from trax import layers as tl
from trax.supervised import training
from trax.fastmath import numpy as fastnp

from collections import defaultdict

# set random seeds
rnd.seed(34)
```

**Notice that in this notebook Trax's numpy is referred to as `fastnp`, while regular numpy is referred to as `np`.**

Now the dataset and word embeddings will get loaded and the data processed.

```python
data = pd.read_csv("data/merged_dataset.csv", encoding="utf-8")

N = len(data)
print("Number of sentence pairs: ", N)

data.head()

!wget -O data/glove.840B.300d.zip nlp.stanford.edu/data/glove.840B.300d.zip
!unzip -d data data/glove.840B.300d.zip
!rm data/glove.840B.300d.zip

!wget -O data/glove.6B.zip nlp.stanford.edu/data/glove.6B.zip
!unzip -d data data/glove.6B.zip
!rm data/glove.6B.zip
!rm data/glove.6B.200d.txt
!rm data/glove.6B.100d.txt
!rm data/glove.6B.50d.txt

# load the whole embedding into memory
embeddings_index = defaultdict(lambda: 1)
embeddings = dict()
index = 0

pad = "<PAD>"
embeddings[pad] = np.zeros(300)
embeddings_index[pad] = index
index += 1

unk = "<UNK>"
embeddings[unk] = np.asarray([0.22418134, -0.28881392, 0.13854356, 0.00365387, -0.12870757, 0.10243822, 0.061626635, 0.07318011, -0.061350107, -1.3477012, 0.42037755, -0.063593924, -0.09683349, 0.18086134, 0.23704372, 0.014126852, 0.170096, -1.1491593, 0.31497982, 0.06622181, 0.024687296, 0.076693475, 0.13851812, 0.021302193, -0.06640582, -0.010336159, 0.13523154, -0.042144544, -0.11938788, 0.006948221, 0.13333307, -0.18276379, 0.052385733, 0.008943111, -0.23957317, 0.08500333, -0.006894406, 0.0015864656, 0.063391194, 0.19177166, -0.13113557, -0.11295479, -0.14276934, 0.03413971, -0.034278486, -0.051366422, 0.18891625, -0.16673574, -0.057783455, 0.036823478, 0.08078679, 0.022949161, 0.033298038, 0.011784158, 0.05643189, -0.042776518, 0.011959623, 0.011552498, -0.0007971594, 0.11300405, -0.031369694, -0.0061559738, -0.009043574, -0.415336, -0.18870236, 0.13708843, 0.005911723, -0.113035575, -0.030096142, -0.23908928, -0.05354085, -0.044904727, -0.20228513, 0.0065645403, -0.09578946, -0.07391877, -0.06487607, 0.111740574, -0.048649278, -0.16565254, -0.052037314, -0.078968436, 0.13684988, 0.0757494, -0.006275573, 0.28693774, 0.52017444, -0.0877165, -0.33010918, -0.1359622, 0.114895485, -0.09744406, 0.06269521, 0.12118575, -0.08026362, 0.35256687, -0.060017522, -0.04889904, -0.06828978, 0.088740796, 0.003964443, -0.0766291, 0.1263925, 0.07809314, -0.023164088, -0.5680669, -0.037892066, -0.1350967, -0.11351585, -0.111434504, -0.0905027, 0.25174105, -0.14841858, 0.034635577, -0.07334565, 0.06320108, -0.038343467, -0.05413284, 0.042197507, -0.090380974, -0.070528865, -0.009174437, 0.009069661, 0.1405178, 0.02958134, -0.036431845, -0.08625681, 0.042951006, 0.08230793, 0.0903314, -0.12279937, -0.013899368, 0.048119213, 0.08678239, -0.14450377, -0.04424887, 0.018319942, 0.015026873, -0.100526, 0.06021201, 0.74059093, -0.0016333034, -0.24960588, -0.023739101, 0.016396184, 0.11928964, 0.13950661, -0.031624354, -0.01645025, 0.14079992, -0.0002824564, -0.08052984, -0.0021310581, -0.025350995, 0.086938225, 0.14308536, 0.17146006, -0.13943303, 0.048792403, 0.09274929, -0.053167373, 0.031103406, 0.012354865, 0.21057427, 0.32618305, 0.18015954, -0.15881181, 0.15322933, -0.22558987, -0.04200665, 0.0084689725, 0.038156632, 0.15188617, 0.13274793, 0.113756925, -0.095273495, -0.049490947, -0.10265804, -0.27064866, -0.034567792, -0.018810693, -0.0010360252, 0.10340131, 0.13883452, 0.21131058, -0.01981019, 0.1833468, -0.10751636, -0.03128868, 0.02518242, 0.23232952, 0.042052146, 0.11731903, -0.15506615, 0.0063580726, -0.15429358, 0.1511722, 0.12745973, 0.2576985, -0.25486213, -0.0709463, 0.17983761, 0.054027, -0.09884228, -0.24595179, -0.093028545, -0.028203879, 0.094398156, 0.09233813, 0.029291354, 0.13110267, 0.15682974, -0.016919162, 0.23927948, -0.1343307, -0.22422817, 0.14634751, -0.064993896, 0.4703685, -0.027190214, 0.06224946, -0.091360025, 0.21490277, -0.19562101, -0.10032754, -0.09056772, -0.06203493, -0.18876675, -0.10963594, -0.27734384, 0.12616494, -0.02217992, -0.16058226, -0.080475815, 0.026953284, 0.110732645, 0.014894041, 0.09416802, 0.14299914, -0.1594008, -0.066080004, -0.007995227, -0.11668856, -0.13081996, -0.09237365, 0.14741232, 0.09180138, 0.081735, 0.3211204, -0.0036552632, -0.047030564, -0.02311798, 0.048961394, 0.08669574, -0.06766279, -0.50028914, -0.048515294, 0.14144728, -0.032994404, -0.11954345, -0.14929578, -0.2388355, -0.019883996, -0.15917352, -0.052084364, 0.2801028, -0.0029121689, -0.054581646, -0.47385484, 0.17112483, -0.12066923, -0.042173345, 0.1395337, 0.26115036, 0.012869649, 0.009291686, -0.0026459037, -0.075331464, 0.017840583, -0.26869613, -0.21820338, -0.17084768, -0.1022808, -0.055290595, 0.13513643, 0.12362477, -0.10980586, 0.13980341, -0.20233242, 0.08813751, 0.3849736, -0.10653763, -0.06199595, 0.028849555, 0.03230154, 0.023856193, 0.069950655, 0.19310954, -0.077677034, -0.144811], dtype='float32')
embeddings_index[unk] = index
index += 1

f = open("data/glove.6B.300d.txt")
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype="float32")
    embeddings_index[word] = index
    index += 1
    embeddings[word] = coefs
f.close()

vocab_size = index + 1
print("Loaded %s word vectors." % vocab_size)
```

Then I split the data into a train and test set. The test set will be used later to evaluate the model.

```python
N_dups = len(data[data.is_duplicate == 1])

# Take 90% of the duplicates for the train set
N_train = int(N_dups * 0.9)
print(N_train)

# Take the rest of the duplicates for the test set + an equal number of non-dups
N_test = (N_dups - N_train) * 2
print(N_test)

data_train = data[: N_train]
# Shuffle the train set
data_train = data_train.sample(frac=1)

data_test = data[N_train : N_train + N_test]
# Shuffle the test set
data_test = data_test.sample(frac=1)

print("Train set: ", len(data_train), "; Test set: ", len(data_test))

# Remove the unneeded data to some memory
del(data)
```

```python
S1_train_words = np.array(data_train["sentence1"])
S2_train_words = np.array(data_train["sentence2"])

S1_test_words = np.array(data_test["sentence1"])
S2_test_words = np.array(data_test["sentence2"])
y_test  = np.array(data_test["is_duplicate"])

del(data_train)
del(data_test)
```

Above, you have seen that the model only takes the duplicated sentences for training.
All this has a purpose, as the data generator will produce batches $([s1_1, s1_2, s1_3, ...]$, $[s2_1, s2_2,s2_3, ...])$, where $s1_i$ and $s2_k$ are duplicate if and only if $i = k$.

An example of how the data looks is shown below.

```python
print("TRAINING SENTENCES:\n")
print("Sentence 1: ", S1_train_words[0])
print("Sentence 2: ", S2_train_words[0], "\n")
print("Sentence 1: ", S1_train_words[5])
print("Sentence 2: ", S2_train_words[5], "\n")

print("TESTING SENTENCES:\n")
print("Sentence 1: ", S1_test_words[0])
print("Sentence 2: ", S2_test_words[0], "\n")
print("is_duplicate =", y_test[0], "\n")
```

The first step is to tokenize the sentences using a custom tokenizer defined below.

```python
# Create arrays
S1_train = np.empty_like(S1_train_words)
S2_train = np.empty_like(S2_train_words)

S1_test = np.empty_like(S1_test_words)
S2_test = np.empty_like(S2_test_words)
```

```python
def data_tokenizer(sentence):
    """Tokenizer function - cleans and tokenizes the data

    Args:
        sentence (str): The input sentence.
    Returns:
        list: The transformed input sentence.
    """

    if sentence == "":
        return ""

    sentence = tc.lower_all(sentence)[0]

    # Change tabs to spaces
    sentence = re.sub(r"\t+_+", " ", sentence)
    # Change short forms
    sentence = re.sub(r"\'ve", " have", sentence)
    sentence = re.sub(r"(can\'t|can not)", "cannot", sentence)
    sentence = re.sub(r"n\'t", " not", sentence)
    sentence = re.sub(r"I\'m", "I am", sentence)
    sentence = re.sub(r" m ", " am ", sentence)
    sentence = re.sub(r"(\'re| r )", " are ", sentence)
    sentence = re.sub(r"\'d", " would ", sentence)
    sentence = re.sub(r"\'ll", " will ", sentence)
    sentence = re.sub(r"(\d+)(k)", r"\g<1>000", sentence)
    # Make word separations
    sentence = re.sub(r"(\+|-|\*|\/|\^|\.)", " $1 ", sentence)
    # Remove irrelevant stuff, nonprintable characters and spaces
    sentence = re.sub(r"(\'s|\'S|\'|\"|,|[^ -~]+)", "", sentence)
    sentence = tc.strip_all(sentence)[0]

    if sentence == "":
        return ""

    return tc.token_it(tc.lemming(sentence))[0]
```

```python
for idx in range(len(S1_train_words)):
    S1_train[idx] = data_tokenizer(S1_train_words[idx])

for idx in range(len(S2_train_words)):
    S2_train[idx] = data_tokenizer(S2_train_words[idx])

for idx in range(len(S1_test_words)):
    S1_test[idx] = data_tokenizer(S1_test_words[idx])

for idx in range(len(S2_test_words)):
    S2_test[idx] = data_tokenizer(S2_test_words[idx])
```

### 1.2 Converting a sentence to a tensor

The next step is to convert every sentence to a tensor, or an array of numbers, using the word embeddings loaded above.

```python
# Converting sentences to arrays of integers
sentence_of_indexes = []

for i in range(len(S1_train)):
    for word in S1_train[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S1_train[i] = sentence_of_indexes
    sentence_of_indexes = []

for i in range(len(S2_train)):
    for word in S2_train[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S2_train[i] = sentence_of_indexes
    sentence_of_indexes = []

for i in range(len(S1_test)):
    for word in S1_test[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S1_test[i] = sentence_of_indexes
    sentence_of_indexes = []

for i in range(len(S2_test)):
    for word in S2_test[i]:
        sentence_of_indexes += [embeddings_index[word]] if word in embeddings_index else [embeddings_index[unk]]

    S2_test[i] = sentence_of_indexes
    sentence_of_indexes = []
```

```python
print("FIRST SENTENCE IN TRAIN SET:\n")
print(S1_train_words[0], "\n")
print("ENCODED VERSION:")
print(S1_train[0],"\n")
del(S1_train_words)
del(S2_train_words)

print("FIRST SENTENCE IN TEST SET:\n")
print(S1_test_words[0], "\n")
print("ENCODED VERSION:")
print(S1_test[0])
del(S1_test_words)
del(S2_test_words)
```

Now, the train set must be split into a training/validation set so that it can be used to train and evaluate the Siamese model.

```python
# Splitting the data
cut_off = int(len(S1_train) * .8)
train_S1, train_S2 = S1_train[: cut_off], S2_train[: cut_off]
val_S1, val_S2 = S1_train[cut_off :], S2_train[cut_off :]
print("Number of duplicate sentences: ", len(S1_train))
print("The length of the training set is:  ", len(train_S1))
print("The length of the validation set is: ", len(val_S1))
```

### 1.3 Understanding and building the iterator

Given the compational limits, we need to split our data into batches. In this notebook, I built a data generator that takes in $S1$ and $S2$ and returned a batch of size `batch_size` in the following format $([s1_1, s1_2, s1_3, ...]$, $[s2_1, s2_2,s2_3, ...])$. The tuple consists of two arrays and each array has `batch_size` sentences. Again, $s1_i$ and $s2_i$ are duplicates, but they are not duplicates with any other elements in the batch.

The command `next(data_generator)` returns the next batch. This iterator returns a pair of arrays of sentences, which will later be used in the model.

**The ideas behind:**
- The generator returns shuffled batches of data. To achieve this without modifying the actual sentence lists, a list containing the indexes of the sentences is created. This list can be shuffled and used to get random batches everytime the index is reset.
- Append elements of $S1$ and $S2$ to `input1` and `input2` respectively.

```python
def data_generator(S1, S2, batch_size, pad=1, shuffle=False):
    """Generator function that yields batches of data

    Args:
        S1 (list): List of transformed (to tensor) sentences.
        S2 (list): List of transformed (to tensor) sentences.
        batch_size (int): Number of elements per batch.
        pad (int, optional): Pad character from the vocab. Defaults to 1.
        shuffle (bool, optional): If the batches should be randomnized or not. Defaults to False.
    Yields:
        tuple: Of the form (input1, input2) with types (numpy.ndarray, numpy.ndarray)
        NOTE: input1: inputs to your model [s1a, s2a, s3a, ...] i.e. (s1a,s1b) are duplicates
              input2: targets to your model [s1b, s2b,s3b, ...] i.e. (s1a,s2i) i!=a are not duplicates
    """

    input1 = []
    input2 = []
    idx = 0
    len_s = len(S1)
    sentence_indexes = [*range(len_s)]

    if shuffle:
        rnd.shuffle(sentence_indexes)

    while True:
        if idx >= len_s:
            # If idx is greater than or equal to len_q, reset it
            idx = 0
            # Shuffle to get random batches if shuffle is set to True
            if shuffle:
                rnd.shuffle(sentence_indexes)

        s1 = S1[sentence_indexes[idx]]
        s2 = S2[sentence_indexes[idx]]

        idx += 1

        input1.append(s1)
        input2.append(s2)

        if len(input1) == batch_size:
            # Determine max_len as the longest sentence in input1 & input 2
            max_len = max(max([len(s) for s in input1]), max([len(s) for s in input2]))
            # Pad to power-of-2
            max_len = 2 ** int(np.ceil(np.log2(max_len)))

            b1 = []
            b2 = []
            for s1, s2 in zip(input1, input2):
                # Add [pad] to s1 until it reaches max_len
                s1 = s1 + [pad] * (max_len - len(s1))
                # Add [pad] to s2 until it reaches max_len
                s2 = s2 + [pad] * (max_len - len(s2))

                # Append s1
                b1.append(s1)
                # Append s2
                b2.append(s2)

            # Use b1 and b2
            yield np.array(b1), np.array(b2)

            # reset the batches
            input1, input2 = [], []
```

```python
batch_size = 2
res1, res2 = next(data_generator(train_S1, train_S2, batch_size))
print("First sentences  :\n", res1, "\n Shape: ", res1.shape)
print("Second sentences :\n", res2, "\n Shape: ", res2.shape)
```

Now we can go ahead and start building the neural network, as we have a data generator.

# Part 2: Defining the Siamese model

### 2.1 Understanding and building the Siamese Network

A Siamese network is a neural network which uses the same weights while working in tandem on two different input vectors to compute comparable output vectors. The Siamese network model proposed in this notebook looks like this:

The sentences' embeddings are passed to an LSTM layer, the output vectors, $v_1$ and $v_2$, are normalized, and finally a triplet loss is used to get the corresponding cosine similarity for each pair of sentences. The triplet loss makes use of a baseline (anchor) input that is compared to a positive (truthy) input and a negative (falsy) input. The distance from the baseline (anchor) input to the positive (truthy) input is minimized, and the distance from the baseline (anchor) input to the negative (falsy) input is maximized. In math equations, the following is maximized:

$$\mathcal{L}(A, P, N)=\max \left(\|\mathrm{f}(A)-\mathrm{f}(P)\|^{2}-\|\mathrm{f}(A)-\mathrm{f}(N)\|^{2}+\alpha, 0\right)$$

$A$ is the anchor input, for example $s1_1$, $P$ the duplicate input, for example, $s2_1$, and $N$ the negative input (the non duplicate sentence), for example $s2_2$.

$\alpha$ is a margin - how much the duplicates are pushed from the non duplicates.

**The ideas behind:**
- Trax library is used in implementing the model.
- `tl.Serial`: Combinator that applies layers serially (by function composition) allowing the set up the overall structure of the feedforward.
- `tl.LSTM` The LSTM layer.
- `tl.Mean`: Computes the mean across a desired axis. Mean uses one tensor axis to form groups of values and replaces each group with the mean value of that group.
- `tl.Fn` Layer with no weights that applies the function f - vector normalization in this case.
- `tl.parallel`: It is a combinator layer (like `Serial`) that applies a list of layers in parallel to its inputs.

```python
def Siamese(vocab_size=vocab_size, d_model=128):
    """Returns a Siamese model.

    Args:
        vocab_size (int, optional): Length of the vocabulary. Defaults to len(vocab).
        d_model (int, optional): Depth of the model. Defaults to 128.
        mode (str, optional): "train", "eval" or "predict", predict mode is for fast inference. Defaults to "train".

    Returns:
        trax.layers.combinators.Parallel: A Siamese model.
    """

    global embedding_layer

    def normalize(x):  # normalizes the vectors to have L2 norm 1
        return x / fastnp.sqrt(fastnp.sum(x * x, axis=-1, keepdims=True))

    embedding_layer = tl.Embedding(vocab_size, d_model)

    s_processor = tl.Serial(                        # Processor will run on S1 and S2.
        embedding_layer,
        tl.LSTM(d_model),                          # LSTM layer
        tl.Mean(axis=1),                            # Mean over columns
        tl.Fn('Normalize', lambda x: normalize(x))  # Apply normalize function
    )  # Returns one vector of shape [batch_size, d_model].

    # Run on S1 and S2 in parallel.
    model = tl.Parallel(s_processor, s_processor)
    return model
```

Setup the Siamese network model.

```python
# Check the model
model = Siamese()
embedding_layer.weights = np.asarray(list(embeddings.values()))
print(model)
```

### 2.2 Implementing Hard  Negative Mining

Now it's the time to implement the `TripletLoss`.
As explained earlier, loss is composed of two terms. One term utilizes the mean of all the non duplicates, the second utilizes the *closest negative*. The loss expression is then:

\begin{align}
 \mathcal{Loss_1(A,P,N)} &=\max \left( -cos(A,P)  + mean_{neg} +\alpha, 0\right) \\
 \mathcal{Loss_2(A,P,N)} &=\max \left( -cos(A,P)  + closest_{neg} +\alpha, 0\right) \\
\mathcal{Loss(A,P,N)} &= mean(Loss_1 + Loss_2) \\
\end{align}

```python
def TripletLossFn(v1, v2, margin=0.25):
    """Custom Loss function.

    Args:
        v1 (numpy.ndarray): Array with dimension (batch_size, model_dimension) associated to S1.
        v2 (numpy.ndarray): Array with dimension (batch_size, model_dimension) associated to S2.
        margin (float, optional): Desired margin. Defaults to 0.25.

    Returns:
        jax.interpreters.xla.DeviceArray: Triplet Loss.
    """

    scores = fastnp.dot(v1, v2.T)       # pairwise cosine sim
    batch_size = len(scores)

    positive = fastnp.diagonal(scores)  # the positive ones (duplicates)
    negative_without_positive = scores - 2.0 * fastnp.eye(batch_size)

    closest_negative = fastnp.max(negative_without_positive, axis=1)
    negative_zero_on_duplicate = (1.0 - fastnp.eye(batch_size)) * scores
    mean_negative = fastnp.sum(negative_zero_on_duplicate, axis=1) / (batch_size - 1)

    triplet_loss1 = fastnp.maximum(0.0, margin - positive + closest_negative)
    triplet_loss2 = fastnp.maximum(0.0, margin - positive + mean_negative)
    triplet_loss = fastnp.mean(triplet_loss1 + triplet_loss2)

    return triplet_loss
```

```python
v1 = np.array([[0.26726124, 0.53452248, 0.80178373],[0.5178918 , 0.57543534, 0.63297887]])
v2 = np.array([[ 0.26726124,  0.53452248,  0.80178373],[-0.5178918 , -0.57543534, -0.63297887]])
TripletLossFn(v2,v1)
print("Triplet Loss:", TripletLossFn(v2,v1))
```

**Expected Output:**
```CPP
Triplet Loss: 0.5
```

```python
from functools import partial
def TripletLoss(margin=1):
    # Trax layer creation
    triplet_loss_fn = partial(TripletLossFn, margin=margin)
    return tl.Fn("TripletLoss", triplet_loss_fn)
```

# Part 3: Training

The next step is model training - defining the cost function and the optimizer, feeding in the built model. But first I will define the data generators used in the model.

```python
batch_size = 512
train_generator = data_generator(train_S1, train_S2, batch_size)
val_generator = data_generator(val_S1, val_S2, batch_size)
print("train_S1.shape ", train_S1.shape)
print("val_S1.shape   ", val_S1.shape)
```

Now, I will define the training step. Each training iteration is defined as an `epoch`, each epoch being an iteration over all the data, using the training iterator.

**The ideas behind:**
- Two tasks are needed: `TrainTask` and `EvalTask`.
- The training runs in a trax loop `trax.supervised.training.Loop`.
- Pass the other parameters to a loop.

```python
def train_model(Siamese, TripletLoss, lr_schedule, train_generator=train_generator, val_generator=val_generator, output_dir="trax_model/"):
    """Training the Siamese Model

    Args:
        Siamese (function): Function that returns the Siamese model.
        TripletLoss (function): Function that defines the TripletLoss loss function.
        lr_schedule (function): Trax multifactor schedule function.
        train_generator (generator, optional): Training generator. Defaults to train_generator.
        val_generator (generator, optional): Validation generator. Defaults to val_generator.
        output_dir (str, optional): Path to save model to. Defaults to "trax_model/".

    Returns:
        trax.supervised.training.Loop: Training loop for the model.
    """

    output_dir = os.path.expanduser(output_dir)

    train_task = training.TrainTask(
        labeled_data=train_generator,
        loss_layer=TripletLoss(),
        optimizer=trax.optimizers.Adam(0.01),
        lr_schedule=lr_schedule
    )

    eval_task = training.EvalTask(
        labeled_data=val_generator,
        metrics=[TripletLoss()]
    )

    training_loop = training.Loop(Siamese(),
                                  train_task,
                                  eval_tasks=[eval_task],
                                  output_dir=output_dir,
                                  random_seed=34)

    return training_loop
```

```python
train_steps = 1500
lr_schedule = trax.lr.warmup_and_rsqrt_decay(400, 0.01)
training_loop = train_model(Siamese, TripletLoss, lr_schedule)
training_loop.run(train_steps)
```

# Part 4:  Evaluation

To determine the accuracy of the model, the test set that was configured earlier is used. While the training used only positive examples, the test data, S1_test, S2_test and y_test, is setup as pairs of sentences, some of which are duplicates some are not.
This routine runs all the test sentences pairs through the model, computes the cosine simlarity of each pair, thresholds it and compares the result to  y_test - the correct response from the data set. The results are accumulated to produce an accuracy.

**The ideas behind:**
 - The model loops through the incoming data in batch_size chunks.
 - The output vectors are computed and their cosine similarity is thresholded.

```python
def classify(test_S1, test_S2, y, threshold, model, data_generator=data_generator, batch_size=64):
    """Function to test the model. Calculates some metrics, such as precision, accuracy, recall and F1 score.

    Args:
        test_S1 (numpy.ndarray): Array of S1 sentences.
        test_S2 (numpy.ndarray): Array of S2 sentences.
        y (numpy.ndarray): Array of actual target.
        threshold (float): Desired threshold.
        model (trax.layers.combinators.Parallel): The Siamese model.
        data_generator (function): Data generator function. Defaults to data_generator.
        batch_size (int, optional): Size of the batches. Defaults to 64.

    Returns:
        (float, float, float, float): Accuracy, precision, recall and F1 score of the model.
    """

    true_pos = 0
    true_neg = 0
    false_pos = 0
    false_neg = 0

    for i in range(0, len(test_S1), batch_size):
        to_process = len(test_S1) - i

        if to_process < batch_size:
            batch_size = to_process

        s1, s2 = next(data_generator(test_S1[i : i + batch_size], test_S2[i : i + batch_size], batch_size, shuffle=False))
        y_test = y[i : i + batch_size]

        v1, v2 = model((s1, s2))

        for j in range(batch_size):
            d = np.dot(v1[j], v2[j].T)
            res = d > threshold

            if res == 1:
                if y_test[j] == res:
                    true_pos += 1
                else:
                    false_pos += 1
            else:
                if y_test[j] == res:
                    true_neg += 1
                else:
                    false_neg += 1

    accuracy = (true_pos + true_neg) / (true_pos + true_neg + false_pos + false_neg)
    precision = true_pos / (true_pos + false_pos)
    recall = true_pos / (true_pos + false_neg)
    f1_score = 2 * precision * recall / (precision + recall)

    print("fn = " +  str(false_neg) + " fp = " + str(false_pos) + " tn = " + str(true_neg) + " tp = " + str(true_pos))

    return (accuracy, precision, recall, f1_score)
```

```python
print(len(S1_test))
```

```python
# Loading in the saved model
model = Siamese()
model.init_from_file("trax_model/model.pkl.gz")
# Evaluating it
accuracy, precision, recall, f1_score = classify(S1_test, S2_test, y_test, 0.7, model, batch_size=512)
print("Accuracy", accuracy)
print("Precision", precision)
print("Recall", recall)
print("F1 score", f1_score)
```

# Part 5: Making predictions

In this section the model will be put to work. It will be wrapped in a function called `predict` which takes two sentences as input and returns $1$ or $0$, depending on whether the pair is a duplicate or not.

But first, we need to embed the sentences.

```python
def predict(sentence1, sentence2, threshold, model, vocab, data_generator=data_generator, verbose=False):
    """Function for predicting if two sentences are duplicates.

    Args:
        sentence1 (str): First sentence.
        sentence2 (str): Second sentence.
        threshold (float): Desired threshold.
        model (trax.layers.combinators.Parallel): The Siamese model.
        vocab (collections.defaultdict): The vocabulary used.
        data_generator (function): Data generator function. Defaults to data_generator.
        verbose (bool, optional): If the results should be printed out. Defaults to False.

    Returns:
        bool: True if the sentences are duplicates, False otherwise.
    """

    s1 = data_tokenizer(sentence1)     # tokenize
    s2 = data_tokenizer(sentence2)     # tokenize

    S1 = [embeddings_index[word] for word in s1]  # encode s1
    S2 = [embeddings_index[word] for word in s2]  # encode s2

    S1, S2 = next(data_generator([S1], [S2], 1, vocab["<PAD>"]))

    v1, v2 = model((S1, S2))
    d = np.dot(v1[0], v2[0].T)
    res = d > threshold

    if verbose == True:
        print("S1  = ", S1, "\nS2  = ", S2)
        print("d   = ", d)
        print("res = ", res)

    return res
```

Now we can test the model's ability to make predictions.

```python
sentence1 = "I love running in the park."
sentence2 = "I like running in park?"
# 1 means it is duplicated, 0 otherwise
predict(sentence1 , sentence2, 0.7, model, verbose=True)
```

The Siamese network is capable of catching complicated structures. Concretely, it can identify sentence duplicates although the sentences do not have many words in common.