Rough volatility with Python
Jim Gatheral

For Python Quants, New York, Friday May 6, 2016

Acknowledgements
The code in this iPython notebook used to be in R. I am very grateful to Yves Hilpisch and Michael Schwed for translating my R-code to Python.

For slideshow functionality I use RISE by Damián Avila.

Outline of presentation
The time series of historical volatility

Scaling properties
The RFSV model

Pricing under rough volatility

Forecasting realized variance

The time series of variance swaps

Relating historical and implied

The time series of realized variance
Assuming an underlying variance process  vs , integrated variance  1δ∫t+δtvsds  may (in principle) be estimated arbitrarily accurately given enough price data.

In practice, market microstructure noise makes estimation harder at very high frequency.
Sophisticated estimators of integrated variance have been developed to adjust for market microstructure noise. See Gatheral and Oomen [6] (for example) for details of these.
The Oxford-Man Institute of Quantitative Finance makes historical realized variance (RV) estimates freely available at http://realized.oxford-man.ox.ac.uk. These estimates are updated daily.

Each day, for 21 different indices, all trades and quotes are used to estimate realized (or integrated) variance over the trading day from open to close.
Using daily RV estimates as proxies for instantaneous variance, we may investigate the time series properties of  vt  empirically.
First load all necessary Python libraries.

In [1]:
import warnings; warnings.simplefilter('ignore')
import datetime as dt
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
from matplotlib.mlab import stineman_interp
import pandas as pd
import pandas.io.data as web
import requests
import zipfile as zi
import StringIO as sio
from sklearn import datasets, linear_model
import scipy.special as scsp
import statsmodels.api as sm
import math
import seaborn as sns; sns.set()
%matplotlib inline
Then update and save the latest Oxford-Man data.

In [2]:
url = 'http://realized.oxford-man.ox.ac.uk/media/1366/'
url += 'oxfordmanrealizedvolatilityindices.zip'
data = requests.get(url, stream=True).content
z = zi.ZipFile(sio.StringIO(data))
z.extractall()
There are many different estimates of realized variance, all of them very similar. We will use the realized kernel estimates denoted by ".rk".

In [3]:
df = pd.read_csv('OxfordManRealizedVolatilityIndices.csv', index_col=0, header=2 )
rv1 = pd.DataFrame(index=df.index)
for col in df.columns:
    if col[-3:] == '.rk':
        rv1[col] = df[col]
rv1.index = [dt.datetime.strptime(str(date), "%Y%m%d") for date in rv1.index.values]
Let's plot SPX realized variance.

In [4]:
spx = pd.DataFrame(rv1['SPX2.rk'])
spx.plot(color='red', grid=True, title='SPX realized variance',
         figsize=(16, 9), ylim=(0,0.003));

Figure 1: Oxford-Man KRV estimates of SPX realized variance from January 2000 to the current date.

In [5]:
spx.head()
Out[5]:
SPX2.rk
2000-01-03	0.000161
2000-01-04	0.000264
2000-01-05	0.000305
2000-01-06	0.000149
2000-01-07	0.000123
In [6]:
spx.tail()
Out[6]:
SPX2.rk
2016-04-27	0.000031
2016-04-28	0.000031
2016-04-29	0.000048
2016-05-02	0.000028
2016-05-03	0.000041
We can get SPX data from Yahoo using the DataReader function:

In [7]:
SPX = web.DataReader(name = '^GSPC',data_source = 'yahoo', start='2000-01-01')
SPX = SPX['Adj Close']
SPX.plot(title='SPX',figsize=(14, 8));

The smoothness of the volatility process
For  q≥0 , we define the  q th sample moment of differences of log-volatility at a given lag  Δ .( ⟨⋅⟩  denotes the sample average):

m(q,Δ)=⟨|logσt+Δ−logσt|q⟩

For example

m(2,Δ)=⟨(logσt+Δ−logσt)2⟩

is just the sample variance of differences in log-volatility at the lag  Δ .

Scaling of  m(q,Δ)  with lag  Δ
In [8]:
spx['sqrt']= np.sqrt(spx['SPX2.rk'])
spx['log_sqrt'] = np.log(spx['sqrt'])

def del_Raw(q, x):
    return [np.mean(np.abs(spx['log_sqrt'] - spx['log_sqrt'].shift(lag)) ** q)
            for lag in x]
In [9]:
plt.figure(figsize=(8, 8))
plt.xlabel('$log(\Delta)$')
plt.ylabel('$log\  m(q.\Delta)$')
plt.ylim=(-3, -.5)

zeta_q = list()
qVec = np.array([.5, 1, 1.5, 2, 3])
x = np.arange(1, 100)
for q in qVec:
    plt.plot(np.log(x), np.log(del_Raw(q, x)), 'o')
    model = np.polyfit(np.log(x), np.log(del_Raw(q, x)), 1)
    plt.plot(np.log(x), np.log(x) * model[0] + model[1])
    zeta_q.append(model[0])

print zeta_q
[0.072526605081197543, 0.14178030350291049, 0.20760692212873791, 0.27007948205423049, 0.38534332343872962]

Figure 2:  logm(q,Δ)  as a function of  logΔ , SPX.

Monofractal scaling result
From the above log-log plot, we see that for each  q ,  m(q,Δ)∝Δζq .
How does  ζq  scale with  q ?
Scaling of  ζq  with  q
In [10]:
plt.figure(figsize=(8,8))
plt.xlabel('q')
plt.ylabel('$\zeta_{q}$')
plt.plot(qVec, zeta_q, 'or')

line = np.polyfit(qVec[:4], zeta_q[:4],1)
plt.plot(qVec, line[0] * qVec + line[1])
h_est= line[0]
print(h_est)
0.131697049909

Figure 3: Scaling of  ζq  with  q .

We find the monofractal scaling relationship

ζq=qH

with  H≈0.13 .

Note however that  H  does vary over time, in a narrow range.
Note also that our estimate of  H  is biased high because we proxied instantaneous variance  vt  with its average over each day  1T∫T0vtdt , where  T  is one trading day.
Estimated  H  for all indices
We now repeat this analysis for all 21 indices in the Oxford-Man dataset.

In [11]:
def dlsig2(sic, x, pr=False):
    if pr:
        a= np.array([(sig-sig.shift(lag)).dropna() for lag in x])
        a=a ** 2
        print a.info()
    return [np.mean((sig-sig.shift(lag)).dropna() ** 2) for lag in x]
In [12]:
h = list()
nu = list()

for col in rv1.columns:
    sig = rv1[col]
    sig = np.log(np.sqrt(sig))
    sig = sig.dropna()
    model = np.polyfit(np.log(x), np.log(dlsig2(sig, x)), 1)
    nu.append(np.sqrt(np.exp(model[1])))
    h.append(model[0]/2.)

OxfordH = pd.DataFrame({'names':rv1.columns, 'h_est': h, 'nu_est': nu})

In [13]:
OxfordH
Out[13]:
h_est	names	nu_est
0	0.133954	SPX2.rk	0.321337
1	0.142315	FTSE2.rk	0.270677
2	0.113366	N2252.rk	0.320396
3	0.150251	GDAXI2.rk	0.274873
4	NaN	RUT2.rk	NaN
5	0.083370	AORD2.rk	0.359025
6	0.131013	DJI2.rk	0.317327
7	NaN	IXIC2.rk	NaN
8	0.130485	FCHI2.rk	0.291967
9	0.103370	HSI2.rk	0.281453
10	0.125847	KS11.rk	0.274713
11	0.145671	AEX.rk	0.290187
12	0.178427	SSMI.rk	0.223249
13	0.128117	IBEX2.rk	0.281662
14	0.110092	NSEI.rk	0.324883
15	0.092252	MXX.rk	0.324180
16	0.106595	BVSP.rk	0.312907
17	NaN	GSPTSE.rk	NaN
18	0.119857	STOXX50E.rk	0.337045
19	0.127094	FTSTI.rk	0.228910
20	0.133361	FTSEMIB.rk	0.298712
Distributions of  (logσt+Δ−logσt)  for various lags  Δ
Having established these beautiful scaling results for the moments, how do the histograms look?

In [14]:
def plotScaling(j, scaleFactor):
    col_name = rv1.columns[j]
    v = rv1[col_name]
    x = np.arange(1,101)

    def xDel(x, lag):
        return x-x.shift(lag)

    def sdl(lag):
        return (xDel(np.log(v), lag)).std()

    sd1 = (xDel(np.log(v), 1)).std()
    h = OxfordH['h_est'][j]
    f, ax = plt.subplots(2,2,sharex=False, sharey=False, figsize=(10, 10))

    for i_0 in range(0, 2):
        for i_1 in range(0, 2):
            la = scaleFactor ** (i_1*1+i_0*2)

            hist_val = xDel(np.log(v), la).dropna()
            std = hist_val.std()
            mean = hist_val.mean()

            ax[i_0][i_1].set_title('Lag = %s Days' %la)
            n, bins, patches = ax[i_0][i_1].hist(hist_val.values, bins=100,
                                   normed=1, facecolor='green',alpha=0.2)
            ax[i_0][i_1].plot(bins, mlab.normpdf(bins,mean,std), "r")
            ax[i_0][i_1].plot(bins, mlab.normpdf(bins,0,sd1 * la ** h), "b--")
            hist_val.plot(kind='density', ax=ax[i_0][i_1])

In [15]:
 plotScaling(1,5)

Figure 4: Histograms of  (logσt+Δ−logσt)  for various lags  Δ ; normal fit in red;  Δ=1  normal fit scaled by  Δ0.14  in blue.

Universality?
[Gatheral, Jaisson and Rosenbaum][5] compute daily realized variance estimates over one hour windows for DAX and Bund futures contracts, finding similar scaling relationships.
We have also checked that Gold and Crude Oil futures scale similarly.

Although the increments  (logσt+Δ−logσt)  seem to be fatter tailed than Gaussian.
A natural model of realized volatility
As noted originally by [Andersen et al.][1], distributions of differences in the log of realized volatility are close to Gaussian.

This motivates us to model  σt  as a lognormal random variable.
Moreover, the scaling property of variance of RV differences suggests the model:
(1)
logσt+Δ−logσt=ν(WHt+Δ−WHt)

where  WH  is fractional Brownian motion.

In [Gatheral, Jaisson and Rosenbaum][5], we refer to a stationary version of (1) as the RFSV (for Rough Fractional Stochastic Volatility) model.
Fractional Brownian motion (fBm)
Fractional Brownian motion (fBm)  {WHt;t∈R}  is the unique Gaussian process with mean zero and autocovariance function
E[WHtWHs]=12{|t|2H+|s|2H−|t−s|2H}

where  H∈(0,1)  is called the Hurst index or parameter.

In particular, when  H=1/2 , fBm is just Brownian motion.

If  H>1/2 , increments are positively correlated.% so the process is trending.

If  H<1/2 , increments are negatively correlated.% so the process is reverting.
Representations of fBm
There are infinitely many possible representations of fBm in terms of Brownian motion. For example, with  γ=12−H ,

Mandelbrot-Van Ness

WHt=CH{∫t−∞dWs(t−s)γ−∫0−∞dWs(−s)γ}.

The choice

CH=2HΓ(3/2−H)Γ(H+1/2)Γ(2−2H)−−−−−−−−−−−−−−−−−−−√

ensures that

E[WHtWHs]=12{t2H+s2H−|t−s|2H}.

Does simulated RSFV data look real?

Figure 8: Volatility of SPX (above) and of the RFSV model (below).

Remarks on the comparison
The simulated and actual graphs look very alike.

Persistent periods of high volatility alternate with low volatility periods.

H∼0.1  generates very rough looking sample paths (compared with  H=1/2  for Brownian motion).

Hence rough volatility.

On closer inspection, we observe fractal-type behavior.

The graph of volatility over a small time period looks like the same graph over a much longer time period.

This feature of volatility has been investigated both empirically and theoretically in, for example, [Bacry and Muzy][3] .

In particular, their Multifractal Random Walk (MRW) is related to a limiting case of the RSFV model as  H→0 .

Pricing under rough volatility
The foregoing behavior suggest the following model (see [Bayer et al.][2] for volatility under the real (or historical or physical) measure  P :

logσt=νWHt.

Let  γ=12−H . We choose the Mandelbrot-Van Ness representation of fractional Brownian motion  WH  as follows:

WHt=CH{∫t−∞dWPs(t−s)γ−∫0−∞dWPs(−s)γ}.

Then

==:logvu−logvtνCH{∫ut1(u−s)γdWPs+∫t−∞[1(u−s)γ−1(t−s)γ]dWPs}2νCH[Mt(u)+Zt(u)].

Note that  EP[Mt(u)|Ft]=0  and  Zt(u)  is  Ft -measurable.

To price options, it would seem that we would need to know  Ft , the entire history of the Brownian motion  Ws  for $s
Pricing under  P
Let

W~Pt(u):=2H−−−√∫utdWPs(u−s)γ

With  η:=2νCH/2H−−−√  we have  2νCHMt(u)=ηW~Pt(u)  so denoting the stochastic exponential by  E(⋅) , we may write

vu==vtexp{ηW~Pt(u)+2νCHZt(u)}EP[vu|Ft]E(ηW~Pt(u)).

The conditional distribution of  vu  depends on  Ft  only through the variance forecasts  EP[vu|Ft] ,
To price options, one does not need to know  Ft , the entire history of the Brownian motion  WPs  for $s
Pricing under  Q
Our model under  P  reads:

(2)
vu=EP[vu|Ft]E(ηW~Pt(u)).

Consider some general change of measure

dWPs=dWQs+λsds,

where  {λs:s>t}  has a natural interpretation as the price of volatility risk.

We may then rewrite (2) as

vu=EP[vu|Ft]E(ηW~Qt(u))exp{η2H−−−√∫utλs(u−s)γds}.

Although the conditional distribution of  vu  under  P  is lognormal, it will not be lognormal in general under  Q .

The upward sloping smile in VIX options means  λs  cannot be deterministic in this picture.
The rough Bergomi (rBergomi) model
Let's nevertheless consider the simplest change of measure

dWPs=dWQs+λ(s)ds,

where  λ(s)  is a deterministic function of  s . Then from (2), we would have

vu==EP[vu|Ft]E(ηW~Qt(u))exp{η2H−−−√∫ut1(u−s)γλ(s)ds}ξt(u)E(ηW~Qt(u))

where the forward variances  ξt(u)=EQ[vu|Ft]  are (at least in principle) tradable and observed in the market.

ξt(u)  is the product of two terms:

EP[vu|Ft]  which depends on the historical path $\{W_s, s
a term which depends on the price of risk  λ(s) .
Features of the rough Bergomi model
The rBergomi model is a non-Markovian generalization of the Bergomi model:
E[vu|Ft]≠E[vu|vt].

The rBergomi model is Markovian in the (infinite-dimensional) state vector  EQ[vu|Ft]=ξt(u) .
We have achieved our aim from Session 1 of replacing the exponential kernels in the Bergomi model with a power-law kernel.
We may therefore expect that the rBergomi model will generate a realistic term structure of ATM volatility skew.
Re-interpretation of the conventional Bergomi model
A conventional  n -factor Bergomi model is not self-consistent for an arbitrary choice of the initial forward variance curve  ξt(u) .

ξt(u)=E[vu|Ft]  should be consistent with the assumed dynamics.
Viewed from the perspective of the fractional Bergomi model however:

The initial curve  ξt(u)  reflects the history $\{W_s; s
The exponential kernels in the exponent of the conventional Bergomi model approximate more realistic power-law kernels.
The conventional two-factor Bergomi model is then justified in practice as a tractable Markovian engineering approximation to a more realistic fractional Bergomi model.
The stock price process
The observed anticorrelation between price moves and volatility moves may be modeled naturally by anticorrelating the Brownian motion  W  that drives the volatility process with the Brownian motion driving the price process.
Thus
dStSt=vt−−√dZt

with
dZt=ρdWt+1−ρ2−−−−−√dW⊥t

where  ρ  is the correlation between volatility moves and price moves.
Simulation of the rBergomi model
We simulate the rBergomi model as follows:

Construct the joint covariance matrix for the Volterra process  W~  and the Brownian motion  Z  and compute its Cholesky decomposition.
For each time, generate iid normal random vectors {and multiply them by the lower-triangular matrix obtained by the Cholesky decomposition} to get a  m×2n  matrix of paths of  W~  and  Z  with the correct joint marginals.
With these paths held in memory, we may evaluate the expectation under  Q  of any payoff of interest.
This procedure is very slow!

Speeding up the simulation is work in progress.
Guessing rBergomi model parameters
The rBergomi model has only three parameters:  H ,  η  and  ρ .
If we had a fast simulation, we could just iterate on these parameters to find the best fit to observed option prices. But we don't.
However, the model parameters  H ,  η  and  ρ  have very direct interpretations:

H  controls the decay of ATM skew  ψ(τ)  for very short expirations.

The product  ρη  sets the level of the ATM skew for longer expirations.

Keeping  ρη  constant but decreasing  ρ  (so as to make it more negative) pushes the minimum of each smile towards higher strikes.
So we can guess parameters in practice.
As we will see, even without proper calibration (i.e. just guessing parameters), rBergomi model fits to the volatility surface are amazingly good.
SPX smiles in the rBergomi model
In Figures 9 and 10, we show how well a rBergomi model simulation with guessed parameters fits the SPX option market as of February 4, 2010, a day when the ATM volatility term structure happened to be pretty flat.

rBergomi parameters were:  H=0.07 ,  η=1.9 ,  ρ=−0.9 .
Only three parameters to get a very good fit to the whole SPX volatility surface!
rBergomi fits to SPX smiles as of 04-Feb-2010

Figure 9: Red and blue points represent bid and offer SPX implied volatilities; orange smiles are from the rBergomi simulation.

Shortest dated smile as of February 4, 2010

Figure 10: Red and blue points represent bid and offer SPX implied volatilities; orange smile is from the rBergomi simulation.

ATM volatilities and skews
In Figures 11 and 12, we see just how well the rBergomi model can match empirical skews and vols. Recall also that the parameters we used are just guesses!

Term structure of ATM skew as of February 4, 2010

Figure 11: Blue points are empirical skews; the red line is from the rBergomi simulation.

Term structure of ATM vol as of February 4, 2010

Figure 12: Blue points are empirical ATM volatilities; the red line is from the rBergomi simulation.

Another date
Now we take a look at another date: August 14, 2013, two days before the last expiration date in our dataset.
Options set at the open of August 16, 2013 so only one trading day left.
Note in particular that the extreme short-dated smile is well reproduced by the rBergomi model.
There is no need to add jumps!
SPX smiles as of August 14, 2013

Figure 13: Red and blue points represent bid and offer SPX implied volatilities; orange smiles are from the rBergomi simulation.

The forecast formula
In the RFSV model (1),  logσt≈νWHt+C  for some constant  C .
[Nuzman and Poor][7] show that  WHt+Δ  is conditionally Gaussian with conditional expectation
E[WHt+Δ|Ft]=cos(Hπ)πΔH+1/2∫t−∞WHs(t−s+Δ)(t−s)H+1/2ds

and conditional variance

Var[WHt+Δ|Ft]=cΔ2H.

where
c=Γ(3/2−H)Γ(H+1/2)Γ(2−2H).

The forecast formula
Thus, we obtain

Variance forecast formula

(3)
EP[vt+Δ|Ft]=exp{EP[log(vt+Δ)|Ft]+2cν2Δ2H}

where

EP[logvt+Δ|Ft]=cos(Hπ)πΔH+1/2∫t−∞logvs(t−s+Δ)(t−s)H+1/2ds.

Implement variance forecast in Python
In [16]:
def c_tilde(h):
    return scsp.gamma(3. / 2. - h) / scsp.gamma(h + 1. / 2.) * scsp.gamma(2. - 2. * h)

def forecast_XTS(rvdata, h, date, nLags, delta, nu):
    i = np.arange(nLags)
    cf = 1./((i + 1. / 2.) ** (h + 1. / 2.) * (i + 1. / 2. + delta))
    ldata = rvdata.truncate(after=date)
    l = len(ldata)
    ldata = np.log(ldata.iloc[l - nLags:])
    ldata['cf'] = np.fliplr([cf])[0]
    # print ldata
    ldata = ldata.dropna()
    fcst = (ldata.iloc[:, 0] * ldata['cf']).sum() / sum(ldata['cf'])

    return math.exp(fcst + 2 * nu ** 2 * c_tilde(h) * delta ** (2 * h))
SPX actual vs forecast variance
In [17]:
rvdata = pd.DataFrame(rv1['SPX2.rk'])
nu  = OxfordH['nu_est'][0] # Vol of vol estimate for SPX
h = OxfordH['h_est'][0]
n = len(rvdata)
delta = 1
nLags = 500
dates = rvdata.iloc[nLags:n-delta].index
rv_predict = [forecast_XTS(rvdata, h=h, date=d, nLags=nLags,
                           delta=delta, nu=nu) for d in dates]
rv_actual = rvdata.iloc[nLags+delta:n].values
Scatter plot of delta days ahead predictions
In [18]:
plt.figure(figsize=(8, 8))
plt.plot(rv_predict, rv_actual, 'bo');

Figure 14: Actual vols vs predicted vols.

Superimpose actual and predicted vols
In [19]:
plt.figure(figsize=(11, 6))
vol_actual = np.sqrt(np.multiply(rv_actual,252))
vol_predict = np.sqrt(np.multiply(rv_predict,252))
plt.plot(vol_actual, "b")
plt.plot(vol_predict, "r");

Figure 15: Actual volatilities in blue; predicted vols in red.

Forecasting the variance swap curve
Finally, we forecast the whole variance swap curve using the variance forecasting formula (3).

In [20]:
def xi(date, tt, nu,h, tscale):  # dt=(u-t) is in units of years
    rvdata = pd.DataFrame(rv1['SPX2.rk'])
    return [ forecast_XTS(rvdata,h=h,date=date,nLags=500,delta=dt*tscale,nu=nu) for dt in tt]

nu = OxfordH["nu_est"][0]
h = OxfordH["h_est"][0]

def varSwapCurve(date, bigT, nSteps, nu, h, tscale, onFactor):
  # Make vector of fwd variances
  tt = [ float(i) * (bigT) / nSteps for i in range(nSteps+1)]
  delta_t = tt[1]
  xicurve = xi(date, tt, nu, h, tscale)
  xicurve_mid = (np.array(xicurve[0:nSteps]) + np.array(xicurve[1:nSteps+1])) / 2
  xicurve_int = np.cumsum(xicurve_mid) * delta_t
  varcurve1 = np.divide(xicurve_int, np.array(tt[1:]))
  varcurve = np.array([xicurve[0],]+list(varcurve1))
  varcurve = varcurve * onFactor * tscale #  onFactor is to compensate for overnight moves
  res = pd.DataFrame({"texp":np.array(tt), "vsQuote":np.sqrt(varcurve)})
  return(res)
In [21]:
def varSwapForecast(date,tau,nu,h,tscale,onFactor):
  vsc = varSwapCurve(date, bigT=2.5, nSteps=100, nu=nu, h=h,
                    tscale=tscale, onFactor=onFactor) # Creates the whole curve
  x = vsc['texp']
  y = vsc['vsQuote']
  res = stineman_interp(tau,x,y,None)

  return(res)

# Test the function

tau = (.25,.5,1,2)
date = dt.datetime(2008,9,8)
varSwapForecast(date,tau,nu=nu,h=h,tscale=252,onFactor=1)
Out[21]:
array([ 0.21949454,  0.21398188,  0.2117466 ,  0.21262899])
'Constructing a time series of variance swap curves
For each of 2,658 days from Jan 27, 2003 to August 31, 2013:

We compute proxy variance swaps from closing prices of SPX options sourced from OptionMetrics (www.optionmetrics.com) via WRDS.
We form the forecasts  EP[vu|Ft]  using (3) with 500 lags of SPX RV data sourced from The Oxford-Man Institute of Quantitative Finance (http://realized.oxford-man.ox.ac.uk).
We note that the actual variance swap curve is a factor (of roughly 1.4) higher than the forecast, which we may attribute to a combination of overnight movements of the index and the price of volatility risk.
Forecasts must therefore be rescaled to obtain close-to-close realized variance forecasts.
3-month forecast vs actual variance swaps

Figure 16: Actual (proxy) 3-month variance swap quotes in blue vs forecast in red (with no scaling factor).

Ratio of actual to forecast

Figure 17: The ratio between 3-month actual variance swap quotes and 3-month forecasts.

The Lehman weekend
Empirically, it seems that the variance curve is a simple scaling factor times the forecast, but that this scaling factor is time-varying.

We can think of this factor as having two multiplicative components: the overnight factor, and the price of volatility risk.
Recall that as of the close on Friday September 12, 2008, it was widely believed that Lehman Brothers would be rescued over the weekend. By Monday morning, we knew that Lehman had failed.
In Figure 18, we see that variance swap curves just before and just after the collapse of Lehman are just rescaled versions of the RFSV forecast curves.
We need variance swap estimates for 12-Sep-2008 and 15-Sep-2008
We proxy these by taking SVI fits for the two dates and computing the log-strips.

In [22]:
varSwaps12 =(
    0.2872021, 0.2754535, 0.2601864, 0.2544684, 0.2513854, 0.2515314,
    0.2508418, 0.2520099, 0.2502763, 0.2503309, 0.2580933, 0.2588361,
    0.2565093)

texp12 = (
    0.01916496, 0.04654346, 0.09582478, 0.19164956, 0.26830938, 0.29842574,
    0.51745380, 0.54483231, 0.76659822, 0.79397673, 1.26488706, 1.76317591,
    2.26146475)

varSwaps15 = (
    0.4410505, 0.3485560, 0.3083603, 0.2944378, 0.2756881, 0.2747838,
    0.2682212, 0.2679770, 0.2668113, 0.2706713, 0.2729533, 0.2689598,
    0.2733176)

texp15 = (
    0.01095140, 0.03832991, 0.08761123, 0.18343600, 0.26009582, 0.29021218,
    0.50924025, 0.53661875, 0.75838467, 0.78576318, 1.25667351, 1.75496235,
    2.25325120)
Actual vs predicted over the Lehman weekend
In [23]:
nu = OxfordH['nu_est'][0]
h = OxfordH['h_est'][0]
date1 = dt.datetime(2008, 9, 12)
date2 = dt.datetime(2008, 9, 15)

# Variance curve fV model forecasts
tau1000 = [ float(i) * 2.5 / 1000. for i in range(1,1001)]
vs1 = varSwapForecast(date1, tau1000, nu=nu,h=h, tscale=252, onFactor=1.29)
vs2 = varSwapForecast(date2, tau1000, nu=nu,h=h, tscale=252, onFactor=1.29)
In [24]:
plt.figure(figsize=(11, 6))
plt.plot(texp12, varSwaps12, "r")
plt.plot(texp15, varSwaps15, "b")
plt.plot(tau1000, vs1, "r--")
plt.plot(tau1000, vs2, "b--");

Figure 18: SPX variance swap curves as of September 12, 2008 (red) and September 15, 2008 (blue). The dashed curves are RFSV model forecasts rescaled by the 3-month ratio ( 1.29 ) as of the Friday close.

Remarks
We note that

The actual variance swaps curves are very close to the forecast curves, up to a scaling factor.
We are able to explain the change in the variance swap curve with only one extra observation: daily variance over the trading day on Monday 15-Sep-2008.
The SPX options market appears to be backward-looking in a very sophisticated way.
The Flash Crash
The so-called Flash Crash of Thursday May 6, 2010 caused intraday realized variance to be much higher than normal.
In Figure 19, we plot the actual variance swap curves as of the Wednesday and Friday market closes together with forecast curves rescaled by the 3-month ratio as of the close on Wednesday May 5 (which was  2.52 ).
We see that the actual variance curve as of the close on Friday is consistent with a forecast from the time series of realized variance that includes the anomalous price action of Thursday May 6.
Variance swap estimates
We again proxy variance swaps for 05-May-2010, 07-May-2010 and 10-May-2010 by taking SVI fits (see [Gatheral and Jacquier][4] ) for the three dates and computing the log-strips.

In [25]:
varSwaps5 = (
    0.4250369, 0.2552473, 0.2492892, 0.2564899, 0.2612677, 0.2659618, 0.2705928, 0.2761203,
    0.2828139, 0.2841165, 0.2884955, 0.2895839, 0.2927817, 0.2992602, 0.3116500)

texp5 = (
    0.002737851, 0.043805613, 0.120465435, 0.150581793, 0.197125257, 0.292950034,
    0.369609856, 0.402464066, 0.618754278, 0.654346338, 0.867898700, 0.900752909,
    1.117043121, 1.615331964, 2.631074606)

varSwaps7 = (
    0.5469727, 0.4641713, 0.3963352, 0.3888213, 0.3762354, 0.3666858, 0.3615814, 0.3627013,
    0.3563324, 0.3573946, 0.3495730, 0.3533829, 0.3521515, 0.3506186, 0.3594066)

texp7 = (
    0.01642710, 0.03832991, 0.11498973, 0.14510609, 0.19164956, 0.28747433, 0.36413415,
    0.39698836, 0.61327858, 0.64887064, 0.86242300, 0.89527721, 1.11156742, 1.60985626,
    2.62559890)

varSwaps10 = (
    0.3718439, 0.3023223, 0.2844810, 0.2869835, 0.2886912, 0.2905637, 0.2957070, 0.2960737,
    0.3005086, 0.3031188, 0.3058492, 0.3065815, 0.3072041, 0.3122905, 0.3299425)

texp10 = (
    0.008213552, 0.030116359, 0.106776181, 0.136892539, 0.183436003, 0.279260780,
    0.355920602, 0.388774812, 0.605065024, 0.640657084, 0.854209446, 0.887063655,
    1.103353867, 1.601642710, 2.617385352)
In [26]:
date1 = dt.datetime(2010, 5, 5)
date2 = dt.datetime(2010, 5, 7)

vsf5 = varSwapCurve(date1, bigT=2.5, nSteps=100, nu=nu, h=h, tscale=252, onFactor=2.52)
vsf7 = varSwapCurve(date2, bigT=2.5, nSteps=100, nu=nu, h=h, tscale=252, onFactor=2.52)
In [27]:
plt.figure(figsize=(11, 6))
plt.plot(texp5, varSwaps5, "r", label='May 5')
plt.plot(texp7, varSwaps7, "g", label='May 7')

plt.legend()
plt.xlabel("Time to maturity")
plt.ylabel("Variance swap quote")

plt.plot(vsf5['texp'], vsf5['vsQuote'], "r--")
plt.plot(vsf7['texp'], vsf7['vsQuote'], "g--");

Figure 19: SPX variance swap curves as of May 5, 2010 (red) and May 7, 2010 (green). The dashed curves are RFSV model forecasts rescaled by the 3-month ratio ( 2.52 ) as of the close on Wednesday May 5. The curve as of the close on May 7 is consistent with the forecast including the crazy moves on May 6.

The weekend after the Flash Crash
Now we plot forecast and actual variance swap curves as of the close on Friday May 7 and Monday May 10.

In [28]:
date1 = dt.datetime(2010,5,7)
date2 = dt.datetime(2010,5,10)

vsf7 = varSwapCurve(date1, bigT=2.5, nSteps=100, nu=nu, h=h, tscale=252, onFactor=2.52)
vsf10 = varSwapCurve(date2, bigT=2.5, nSteps=100, nu=nu, h=h, tscale=252, onFactor=2.52)
In [29]:
plt.figure(figsize=(11, 6))
plt.plot(texp7, varSwaps7, "g", label='May 7')
plt.plot(texp10, varSwaps10, "m", label='May 10')

plt.legend()
plt.xlabel("Time to maturity")
plt.ylabel("Variance swap quote")

plt.plot(vsf7['texp'], vsf7['vsQuote'], "g--")
plt.plot(vsf10['texp'], vsf10['vsQuote'], "m--");

Figure 20: The May 10 actual curve is inconsistent with a forecast that includes the Flash Crash.

Now let's see what happens if we exclude the Flash Crash from the time series used to generate the variance curve forecast.

In [30]:
plt.figure(figsize=(11, 6))
ax = plt.subplot(111)
rvdata_p = rvdata.drop((dt.datetime(2010, 5, 6)), axis=0)
rvdata.loc["2010-05-04":"2010-05-10"].plot(ax=ax, legend=False)
rvdata_p.loc["2010-05-04":"2010-05-10"].plot(ax=ax, legend=False);

Figure 21: rvdata_p has the May 6 realized variance datapoint eliminated (green line). Notice the crazy realized variance estimate for May 6!

We need a new variance curve forecast function that uses the new time series.

In [31]:
def xip(date, tt, nu,h, tscale):  # dt=(u-t) is in units of years
    rvdata = pd.DataFrame(rv1['SPX2.rk'])
    rvdata_p = rvdata.drop((dt.datetime(2010, 5, 6)), axis=0)
    return [ forecast_XTS(rvdata_p, h=h, date=date,nLags=500,
                          delta=delta_t * tscale, nu=nu) for delta_t in tt]

nu = OxfordH["nu_est"][0]
h = OxfordH["h_est"][0]

def varSwapCurve_p(date, bigT, nSteps, nu, h, tscale, onFactor):
  # Make vector of fwd variances
  tt = [ float(i) * (bigT) / nSteps for i in range(nSteps+1)]
  delta_t = tt[1]
  xicurve = xip(date, tt, nu, h, tscale)
  xicurve_mid = (np.array(xicurve[0:nSteps]) + np.array(xicurve[1:nSteps + 1])) / 2
  xicurve_int = np.cumsum(xicurve_mid) * delta_t
  varcurve1 = np.divide(xicurve_int, np.array(tt[1:]))
  varcurve = np.array([xicurve[0],]+list(varcurve1))
  varcurve = varcurve * onFactor * tscale #  onFactor is to compensate for overnight moves
  res = pd.DataFrame({"texp":np.array(tt), "vsQuote":np.sqrt(varcurve)})
  return(res)

def varSwapForecast_p(date, tau, nu, h, tscale, onFactor):
  vsc = varSwapCurve_p(date, bigT=2.5, nSteps=100, nu=nu, h=h,
                    tscale=tscale, onFactor=onFactor) # Creates the whole curve
  x = vsc['texp']
  y = vsc['vsQuote']
  res = stineman_interp(tau, x, y, None)

  return(res)

# Test the function

tau = (.25, .5 ,1, 2)
date = dt.datetime(2010, 5, 10)
varSwapForecast_p(date, tau, nu=nu, h=h, tscale=252, onFactor=1. / (1 - .35))
Out[31]:
array([ 0.26077084,  0.25255902,  0.25299844,  0.26116175])
Finally, we compare our new forecast curves with the actuals.

In [32]:
date1 = dt.datetime(2010, 5, 7)
date2 = dt.datetime(2010, 5, 10)

vsf7 = varSwapCurve(date1, bigT=2.5, nSteps=100, nu=nu, h=h, tscale=252, onFactor=2.52)
vsf10p = varSwapCurve_p(date2, bigT=2.5, nSteps=100, nu=nu, h=h, tscale=252, onFactor=2.52)
In [33]:
plt.figure(figsize=(11, 6))
plt.plot(texp7, varSwaps7, "g", label='May 7')
plt.plot(texp10, varSwaps10, "m", label='May 10')

plt.legend()
plt.xlabel("Time to maturity")
plt.ylabel("Variance swap quote")

plt.plot(vsf7['texp'], vsf7['vsQuote'], "g--")
plt.plot(vsf10p['texp'], vsf10p['vsQuote'], "m--");

Figure 22: The May 10 actual curve is consistent with a forecast that excludes the Flash Crash.

Resetting of expectations over the weekend
In Figures 20 and 22, we see that the actual variance swap curve on Monday, May 10 is consistent with a forecast that excludes the Flash Crash.
Volatility traders realized that the Flash Crash should not influence future realized variance projections.
Summary
We uncovered a remarkable monofractal scaling relationship in historical volatility.

A corollary is that volatility is not a long memory process, as widely believed.
This leads to a natural non-Markovian stochastic volatility model under  P .
The simplest specification of  dQdP  gives a non-Markovian generalization of the Bergomi model.
The history of the Brownian motion $\lbrace W_s, s
This model fits the observed volatility surface surprisingly well with very few parameters.
For perhaps the first time, we have a simple consistent model of historical and implied volatility.
References

^ Torben G Andersen, Tim Bollerslev, Francis X Diebold, and Heiko Ebens, The distribution of realized stock return volatility, *Journal of Financial Economics* **61**(1) 43-76 (2001).
^ Christian Bayer, Peter Friz and Jim Gatheral, Pricing under rough volatility, *Quantitative Finance* forthcoming, available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2554754, (2015).
^ Emmanuel Bacry and Jean-François Muzy, Log-infinitely divisible multifractal processes, *Communications in Mathematical Physics* **236**(3) 449-475 (2003).
^ Jim Gatheral and Antoine Jacquier, Arbitrage-free SVI volatility surfaces, *Quantitative Finance* **14**(1) 59-71 (2014).
^ Jim Gatheral, Thibault Jaisson and Mathieu Rosenbaum, Volatility is rough, available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2509457, (2014).
^ Jim Gatheral and Roel Oomen, Zero-intelligence realized variance estimation, *Finance and Stochastics* **14**(2) 249-283 (2010).
^ Carl J. Nuzman and H. Vincent Poor, Linear estimation of self-similar processes via Lamperti’s transformation, *Journal of Applied Probability* **37**(2) 429-452 (2000).


#=
Julia can natively call C and Fortran functions, and with the help of packages it can also access code from other languages, such as Python and R.

Julia has a "no boilerplate" philosophy for C and Fortran code.
There's no "glue" code, code generation of compilation.
To exemplify this, let's consider two files, one in Fortran and one in C:
```
! File lu_fact.f90
subroutine lu_factorization (nrow, ncol, A)

  implicit none

  integer, intent(in) :: nrow, ncol
  real, intent(inout), dimension(nrow, ncol) :: A

  integer :: i, j, k

  do j = 1,nrow
    do i = j+1,nrow
      A(i,j) = A(i,j) / A(j,j)
      do k = j+1,ncol
        A(i,k) = A(i,k) - A(j,k) * A(i,j)
      enddo
    enddo
  enddo

end
```
The code above implements the [LU factorization](https://en.wikipedia.org/wiki/LU_decomposition), without pivoting or checks.
The input matrix is also used as output.
The factorization is used to solve linear systems.

The code below implements the [forward and back substitutions](https://en.wikipedia.org/wiki/Triangular_matrix#Forward_and_back_substitution), using the resulting matrix from above as input.
The input vector `x` corresponds to the right-hand side vector on input, and as the solution of the linear system on output.
```
// File lu_solve.c
#include <stdio.h>

void lu_solve(long long n, float *LU, float *x)
{
  int i, j;

  for (i = 0; i < n; i++)
    for (j = 0; j < i; j++)
      x[i] -= LU[i + j * n] * x[j];

  for (i = n - 1; i >= 0; i--)
  {
    for (j = i + 1; j < n; j++)
      x[i] -= LU[i + j * n] * x[j];
    x[i] /= LU[i + i * n];
  }
}
```

The file `lu_fact.f90` is in Fortran 90 and `lu_solve.c` is in C, as the extension implies.
To compile these file to use with Julia, we have to prepare them into a shared library.
To that end, we'll compile them using the GCC/GFortran compilers with the `-fPIC -shared` flags, and then link them with `ld`.

The following commands were tested on linux:

```
gfortran -fPIC -shared lu_fact.f90 -o lu_fact.o
gcc -fPIC -shared lu_solve.c -o lu_solve.o
ld -shared $PWD/lu_fact.o $PWD/lu_solve.o -o lu.so
```

The `$PWD` variable is necessary because the shared library should know where the compiled packages are.
There are other ways of doing this.

Now we can use the library `lu.so` on Julia.
Here's an example:
=#
using Libdl, LinearAlgebra

path = "assets/interoperability"
lib = joinpath(path, "lu.so")

n = 10
L = tril(rand(Float32, n, n), -1) + I
U = triu(rand(Float32, n, n))
A = L * U
b = A * ones(Float32, n)

LU = copy(A)
x = copy(b)

ccall(
  (:lu_factorization_, lib),
  Cvoid,
  (Ref{Int}, Ref{Int}, Ref{Cfloat}),
  n, n, LU,
)

ccall(
  (:lu_solve, lib),
  Cvoid,
  (Int, Ref{Cfloat}, Ref{Cfloat}),
  n, LU, x,
)

println("‖LU - (L + U - I)‖ = ", norm(LU - L - U + I))
println("‖x - 1‖ = ", norm(x .- 1))
println("‖A * x - b‖ = ", norm(A * x - b))

#=
The first part of the code defines a "good" matrix `A` and `b` such that the solution to the linear system $Ax = b$ is `x = ones(n)`.
The `LU = copy(A)` and `x = copy(b)` are so that we can use `A` and `b` for testing later.
One important part of the code is that since we used the default variables for Fortran, we used `float` for C, and we have to use an appropriate type for Julia as well.
`Float32` refers to a 32 bits real number, i.e. `float`.
`Cfloat` is just an alias for `Float32`.

The `ccall` signature is simple:
- The first argument is the function or a tuple with the function and it's library;
- The second argument is the return type;
- The third argument is a tuple with the types of every argument;
- The other arguments are the arguments to the C/Fortran function.

Noteworthy:
- Fortran with `gfortran` added a `_` to the function name;
- `Ref{T}` indicates that a reference will be passed, allowing the argument to be modified (`Ptr{T}` also exists, to indicate returned pointers by C);
- Fortran always passes variables by reference.

## FFTW

For a more substantial example, we can look at the [FFTW Library](https://www.fftw.org) for the discrete Fourier transform.
The compilation of this example takes some time and the installation may be less straightforward than in my system, so be warned.

- First, download the library. I'm using v3.3.10. Untar it too.
- Then, enter the folder and issue a `./configure --enable-shared`.
- Now, enter `make`.

This will produce a shared library `.libs/libfftw3.so`.

Now we can write a function to compute the FFT of a given vector using this lib.

We'll be create a simple version of the following C code, available in the [online docs of FFTW](https://www.fftw.org/fftw3_doc/Complex-One_002dDimensional-DFTs.html#Complex-One_002dDimensional-DFTs).
```
#include <fftw3.h>
...
{
    fftw_complex *in, *out;
    fftw_plan p;
    ...
    in = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * N);
    out = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * N);
    p = fftw_plan_dft_1d(N, in, out, FFTW_FORWARD, FFTW_ESTIMATE);
    ...
    fftw_execute(p); /* repeat as needed */
    ...
    fftw_destroy_plan(p);
    fftw_free(in); fftw_free(out);
}
```
=#
function myfft(x::Vector{ComplexF64})
  y = copy(x)
  FFTW_FORWARD = Int32(-1)
  FFTW_ESTIMATE = Int32(64)

  plan = ccall(
    (:fftw_plan_dft_1d, "assets/interoperability/fftw-3.3.10/.libs/libfftw3.so"),
    Ptr{Cvoid},
    (Int32, Ref{ComplexF64}, Ref{ComplexF64}, Int32, UInt32),
    length(x), x, y, FFTW_FORWARD, FFTW_ESTIMATE
  )
  ccall(
    (:fftw_execute, "assets/interoperability/fftw-3.3.10/.libs/libfftw3.so"),
    Cvoid,
    (Ptr{Cvoid},),
    plan
  )
  ccall(
    (:fftw_destroy_plan, "assets/interoperability/fftw-3.3.10/.libs/libfftw3.so"),
    Cvoid,
    (Ptr{Cvoid},),
    plan
  )
  return y
end

x = rand(ComplexF64, 8)
y = myfft(x)

#=
We can implement the a basic DFT to compare:
=#
function mydft(x)
  n = length(x)
  return [
    sum(x[i+1] * exp(-im * 2π * i * k / n) for i = 0:n-1)
    for k = 0:n-1
  ]
end
mydft(x)

#=
You may have noticed:
- Memory is handled by Julia in this case, so we don't use `malloc`.
- The C struct `fftw_complex` simply stores two `double`s. In Julia we would need a similar struct, and `ComplexF64` is exactly that, so it's extra smooth.
- We can use `Ptr{Cvoid}` to avoid dealing with the return type of `fftw_plan_dft_1d`, which is an opaque pointer.

### FFTW.jl (and where are the .so files?)

The ability to easily integrate C/Fortran functions is also one of the foundations of Julia.
In fact, the example above is adapted from a [2011 commit](https://github.com/JuliaLang/julia/commit/021afbfbac5ea941dc5402af2a9dea6d91fc7267) almost unmodified.
The FFTW package is now wrapped in the package `FFTW.jl`. You can add and use it easily:
=#
using FFTW
y = fft(x)
#=
If you install `FFTW.jl`, you will notice that you don't need to install the C library `FFTW` manually, i.e., the Julia package automatically does that for you.
Julia has a built-in [`Artifacts` system](https://pkgdocs.julialang.org/dev/artifacts/) and a commonly used package `BinaryBuilder.jl` that work in conjunction to allow binaries to be pre-built and distributed.
The details are complicated, but the user doesn't have to worry about them.
=#

#=
## Python

Julia can also can Python and R code, using `PyCall` and `RCall`, respectively.
We'll focus on Python for this example.

We're gonna use `PyCall`. Like `IJulia`,
=#
using Plots, PyCall, Random
pyplot()

Random.seed!(123)
n = 2^8
X = (2 * rand(n, 2) .- 1) * 4
y = [
  X[i,2]^2 * 16 - X[i,1]^2 * 16 ≤ 2 * randn() ? 1 : -1 for i = 1:n
]
y = [
  (X[i,1] - 1)^2 + (4 - randn()) * (X[i,2] - X[i,1]^2)^2 ≤ 10 ? 1 : -1 for i = 1:n
]
plot(leg=false)
idx = findall(y .== -1)
scatter!(X[idx,1], X[idx,2], m=(4,:red,:square))
idx = findall(y .== 1)
scatter!(X[idx,1], X[idx,2], m=(4,:blue,:circle))
png(joinpath(@OUTPUT, "pycall-1")) # hide

#=
\fig{pycall-1}
=#

svm = pyimport_conda("sklearn.svm", "scikit-learn")
clf = svm.SVC(C=1e-2, gamma=5.0, probability=true)
clf.fit(X, y)
x1g = range(extrema(X[:,1])..., length=100)
x2g = range(extrema(X[:,2])..., length=100)
Z = [
  clf.predict_proba([x1i x2j;])[2] for x2j in x2g, x1i in x1g
]
contourf(x1g, x2g, Z, c=cgrad([:pink,:magenta,:lightblue]), levels=50)
idx = findall(y .== -1)
scatter!(X[idx,1], X[idx,2], m=(4,:red,:square), lab="", opacity=0.5)
idx = findall(y .== 1)
scatter!(X[idx,1], X[idx,2], m=(4,:blue,:circle), lab="", opacity=0.5)
png(joinpath(@OUTPUT, "pycall-2")) # hide

#=
\fig{pycall-2}
=#


# # BAT.jl plotting tutorial

using BAT
using Distributions
using IntervalSets

# ## Generate samples to be plotted
likelihood = params -> begin

    r1 = logpdf.(
    MixtureModel(Normal[
    Normal(-10.0, 1.2),
    Normal(0.0, 1.8),
    Normal(10.0, 2.5)], [0.1, 0.3, 0.6]), params.a)

    r2 = logpdf.(
    MixtureModel(Normal[
    Normal(-5.0, 2.2),
    Normal(5.0, 1.5)], [0.3, 0.7]), params.b[1])

    r3 = logpdf.(Normal(2.0, 1.5), params.b[2])

    return LogDVal(r1+r2+r3)
end

prior = BAT.NamedTupleDist(
    a = Normal(-3, 4.5),
    b = [-20.0..20.0, -10..10]
)

posterior = PosteriorDensity(likelihood, prior);

samples, chains = bat_sample(posterior, MCMCSampling(mcalg = MetropolisHastings(), nsteps = 10^5));

# ## Set up plotting
# Set up plotting using the [Plots.jl](https://github.com/JuliaPlots/Plots.jl) package:
using Plots

# The BAT.jl plotting recipes have been tested with `pyplot()`, `gr()` and `plotly()` backends.
# All available plotting options work correctly with `pyplot()`.
# Only colored 2D contour plots are currently not correctly supported by the `gr()` and `plotly()` backends.
# The backend can be chosen using:

#gr() # Julia's default
#pyplot()
#plotly()

# ## Examples for 1D plots
# BAT.jl includes different plotting styles (seriestypes) for 1D representations of samples and priors:
#   * `:smallest_intervals` (alias `:HDR`): highlighting the smallest intervals containing a certain probability (highest density regions) by color
#   * `:central_intervals`: highlighting the central intervals containing a certain probability by color
#   * `:histogram` (alias `:steppost`): filled histogram
#   * `:stephist`: step histogram
# Examples of (marginalized) 1D representations of samples and priors are shown below.

# ### Default 1D plot of samples:
# Samples can be plotted either by their index or by using the parameter name (as a symbol or expression) as specified in the prior:

plot(samples, :a) #default seriestype = :smallest_intervals (alias :HDR)
#or: plot(samples, 2)
# The default seriestype for plotting samples is `:smallest_intervals` (alias `:HDR`), highlighting the smallest intervals (the highest density region) containing 68.3, 95.5 and 99.7 perecent of the posterior probability. By default, the marginal mode(s) of the histogram is(are) indicated as dotted black line(s).

# ### Default 1D plot of prior:
# Priors can be plotted either by their index or by using the parameter name:

plot(prior, :a)
#or: plot(prior, 1)

# ## Knowledge update plot
# The knowledge update after performing the sampling can be visualized by plotting the prior and the samples of the psterior together in one plot using `plot!()`:
plot(samples, :(b[1]))
plot!(prior, :(b[1]))

# ## Further seriestypes for 1D plots (samples and prior):
# ### central intervals:
plot(samples, :a, seriestype = :central_intervals)

# ### filled histogram:
plot(samples, :a, seriestype = :histogram) # alias :hist

# ### step histogram:
plot(samples, :a, seriestype = :stephist)

# ## Customizing 1D plots:

# ### Keyword arguments for [attributes supported by *Plots.jl*](https://docs.juliaplots.org/latest/attributes/#attributes-1) can be passed:
plot(samples, :a, seriestype = :stephist, bins=50, linecolor = :red, linewidth = 5, linealpha=0.4, xlim=(-10,10))

# ### Customizing interval plots:
# For `:smallest_intervals` and `:central_intervals` plot, the probability enclosed in the intervals to be highlighted can be specified using the `intervals` keyword.
# The keyword `interval_labels` allows to specify the legend entries for the corresponding intervals (in same order).
# The interval colors need to be specified (in same order) using the `colors` keyword argument.
plot(samples, :a, seriestype=:smallest_intervals,
    intervals=[0.5, 0.1, 0.3, 0.99],
    interval_labels = ["HDR 50%", "HDR 10%", "HDR 30%", "HDR 99%"],
    colors=[:grey, :red, :blue, :orange]
)

# ### Plotting estimators in 1D sample plots:
# It is possible to indicate *mean*, *standard deviation*, *marginalmode*, and *globalmode* when plotting samples:
plot(samples, :(b[1]), globalmode=true, marginalmode=true, mean=true, std=true)

# #### The style of the estimators can be customized:
# By passing `true`, the point estimators are plotted using their default styles shown above.
# The styles can be modified by passing dictionaries specifying `linestyle`, `linecolor`, `linewidth` and `alpha` for *mean*, *globalmode* and *marginalmode*.
# The style of the standard deviation can be modified by specifying `fillcolor` and `fillalpha`.
plot(samples, :a, marginalmode=false,
    mean=Dict("linestyle" => :dot, "linecolor"=> :red, "linewidth"=>2, "alpha" => 0.7),
    std=Dict("fillcolor" => :red, "fillalpha"=>0.2))

# ## Examples for 2D plots of samples and priors
# Below, all available seriestypes and plotting features for 2D representations of samples and priors are shown
#   * `:smallest_intervals` (alias `:HDR`): histogram of the smallest intervals containing a certain probability (highest density regions)
#   * `:smallest_intervals_contour`: (colored) contours of the smallest intervals containing a certain probability (highest density regions)
#   * `:smallest__intervals_contourf`: (colored) filled contours of the smallest intervals containing a certain probability (highest density regions)
#   * `:histogram2d` (alias `:histogram`, `:hist`): 2D histogram

# ### Default 2D plot  of samples:
pyplot()
plot(samples, (:a, :(b[1])), mean=true, std=true) #default seriestype = :smallest_intervals (alias :HDR)
# The default seriestype for plotting samples is a 3-color heatmap showing the smallest intervals (highest density regions) containing 68.3%, 95.5% and 99.7% of the posterior probability. By default, the marginal mode
# of the histogram is indicated by a black square.

# ### Default 2D plot  of priors:
plot(prior, (:a, :(b[1]))) # default seriestype :smallest_intervals_contour

# ## 2D knowledge update plot
plot(samples, (:a, :(b[2])))
plot!(prior, (:a, :(b[2])))

# ## Further seriestypes for 2D plots (samples and prior):
# ### histogram:
plot(samples, (:a,:(b[2])), seriestype = :histogram)

# ### smallest intervals as colored contour lines:
# (currently only correctly supported with `pyplot()` backend)
plot(samples, (:a,:(b[2])), seriestype=:smallest_intervals_contour, bins=40)

# ### smallest intervals as filled contours with smoothing:
# (currently only correctly supported with `pyplot()` backend)
plot(samples, (:a,:(b[2])), seriestype=:smallest_intervals_contourf, bins=40, smoothing=1)

# ### Customizing smallest interval plots:
# The probability intervals to be highlighted can be specified using the `intervals` keyword.
# The keyword `interval_labels` allows to specify the legend entries for the corresponding intervals (in same order).
# The interval colors need to be specified (in same order) using the `colors` keyword argument.
plot(samples, (:a,:(b[2])), seriestype=:smallest_intervals, bins=200,
intervals=[0.7, 0.2],
interval_labels = ["HDR 70%", "HDR 20%"],
colors=[:blue, :red])

# ### marginal plot:
plot(samples, (:a, :(b[2])), seriestype = :marginal)

# ### scatter plot:
# (Only plotting the first 10.000 sample points, as for a large number of samples plot takes a lot of time and resources.)
gr()
plot(samples[1:10^4], (:a,:(b[2])), seriestype = :scatter)

# ## Customizing 2D plots:
# ### Plotting point estimators in 2D plots:
plot(samples, (:a,:(b[2])), seriestype=:smallest_intervals, nbins=200,
    mean=true, std=true, marginalmode=true, globalmode=true)

# #### It is possible to customize the style of the estimators:
# By passing `true`, the point estimators are plotted using their default styles shown above.
# The style of the point estimators *mean*, *globalmode* and *marginalmode* can be modified by passing a dictionary specifying `markershape`, `markercolor`, `markersize`, `markeralpha`, `markerstrokecolor`, `markerstrokestyle`, `markerstrokewidth` and `markerstrokealpha`.
# If `std==true`, the standard deviation of the mean value will be displayed as x- and y-errorbars.
plot(samples, (:a,:(b[2])), seriestype=:smallest_intervals, bins=(200, -4:0.2:8),
    marginalmode=Dict("markershape"=> :diamond, "markeralpha"=>1, "markercolor"=>:red, "markersize"=>5),
    mean = true, std=true
)

# ### Customizing marginal plots:
# Marginal plots can be modified by passing dictionaries to the keyword arguments `upper`, `right` and `diagonal`.
# The dictionaries for `upper` and `right` can contain the seriestypes and plot options for 1D distributions shown above.
# The dictionary for `diagonal` can use the seriestypes and plot options for 2D plots shown above.
plot(samples, (:a,:(b[2])), seriestype = :marginal,
    diagonal = Dict("seriestype"=>:histogram),
    upper=Dict("seriestype" => :smallest_intervals, "colors"=>[:blue, :grey, :orange]),
    right=Dict("seriestype" => :stephist)
)

# ## Overview plots
# Without specifying the parameters to be plotted, the 1D and 2D plots of (up to) the first 5 parameters are displayed as a grid.
plot(samples)

plot(prior)

plot(samples)
plot!(prior)

# The keyword argument `vsel` allows to specify which parameters to consider in the overview plot by passing the names or indeces:
plot(samples, vsel=(:a, :(b[2])))
# or: plot(samples, vsel=(1, 3))

# The binning of the overview plots can be specified individually for each parameter:
plot(samples, bins=200) # all paramaters have 200 bins (default)
plot(samples, bins=(10, -5:0.5:8, 500)) #  specify different bin numbers or bin edges for each parameter (only those specified with vsel)
plot(prior, bins=(a=50,)) # use 50 bins for parameter :a and the default value (200) for all others

# ### Customizing overview plots:
# The overview plots can be modified by passing dictionaries to the keyword arguments `upper`, `lower` and `diagonal`.
# The dictionaries for `upper` and `lower` can contain the 2D seriestypes and plot options shown above.
# The dictionary for `diagonal` can use the 1D seriestypes and plot options shown above.
# Nested dictonaries are possible (e.g. for modifying point estimators)
plot(samples, mean=true, globalmode=true, legend=true, bins = (a = -15:2:15, ),
    diagonal=Dict("seriestype"=>:stephist, "mean"=>Dict("linecolor" => :green, "linewidth" => 8)),
    lower = Dict("mean" => false, "colors"=>[:orange, :green, :grey]))

# ## Plots for MCMC diagnostics
# The diagnostic plots allow to plot the samples, a kernel density estimate, the trace and the autocorrelation function for each parameter and each chain:
# The parameters to be considered can be chosen with the `vsel` keyword.
diagnostics = BAT.MCMCDiagnostics(samples, chains)
pyplot()
plot(diagnostics, vsel=[1])

# ### Customizing diagnostics plots:
# #### available keyword arguments:
# * `vsel` - list of parameters to be plotted
# * `chains` - list of chains to be plotted
# * `diagnostics` - list of MCMC diagnostics to be plotted
#   * `:histogram` - 1D histograms of samples
#   * `:kde` - Kernel density estimate (using [*KernelDensity.jl*](https://github.com/JuliaStats/KernelDensity.jl))
#   * `:trace` - Trace plot
#   * `:acf` - Autocorrelation function (using [*StatsBase.autocor*](http://juliastats.github.io/StatsBase.jl/stable/signalcorr/#StatsBase.autocor))
# * `description::Bool = true` - show description (current chain, parameter, number of samples) as first column of plots
# * `histogram::Dict` - options for histogram plots (supports all arguments for 1D plots for samples)
# * `kde::Dict` - options for kde plots
# * `trace::Dict` - options for trace plots
# * `acf::Dict` - options for acf plots
# #### special options arguments for `:kde` (see [*KernelDensity.jl*](https://github.com/JuliaStats/KernelDensity.jl))
# * `npoints`: number of interpolation points to use (default: npoints = 2048)
# * `boundary`: lower and upper limits of the kde as a tuple
# * `kernel`: the distributional family from [*Distributions.jl*](https://github.com/JuliaStats/Distributions.jl) to use as the kernel (default = Distributions.Normal)
# * `bandwidth`: bandwidth of the kernel
# #### special keyword arguments for `:acf` (see [*StatsBase.autocor*](http://juliastats.github.io/StatsBase.jl/stable/signalcorr/#StatsBase.autocor))
# * `lags` - list of lags to be considered for ACF plots
# * `demean` - denotes whether the mean should be subtracted before computing the ACF
plot(diagnostics,
    vsel=[1, 2],
    chains=[1, 2],
    diagnostics = [:histogram, :kde, :trace, :acf],
    histogram = Dict("seriestype" => :smallest_intervals, "legend" => :false),
    trace = Dict("linecolor" => :red),
    acf = Dict("lags" => collect(1:20), "title"=> "Autocorrelation"),
    description = true)


using StatsBase
using Test

n = 5000

# 1D integer counts

x = rand(1:5, n)
w = weights(rand(n))

c = counts(x, 5)
@test size(c) == (5,)
c0 = Int[count(v->v == i, x) for i in 1:5]
@test c == c0
@test counts(x .+ 1, 2:6) == c0
@test proportions(x, 1:5) ≈ (c0 ./ n)

c = counts(x)
@test size(c) == (5,)
c0 = Int[count(v->v == i, x) for i in 1:5]
@test c == c0
@test counts(x .+ 1, 2:6) == c0
@test proportions(x) ≈ (c0 ./ n)

c = counts(x, 5, w)
@test size(c) == (5,)
c0 = Float64[sum(w.values[x .== i]) for i in 1:5]
@test c                      ≈ c0
@test counts(x .+ 1, 2:6, w) ≈ c0
@test proportions(x, 1:5, w) ≈ (c0 ./ sum(w))

c = counts(x, w)
@test size(c) == (5,)
c0 = Float64[sum(w.values[x .== i]) for i in 1:5]
@test c                      ≈ c0
@test counts(x .+ 1, 2:6, w) ≈ c0
@test proportions(x, w)      ≈ (c0 ./ sum(w))

# 2D integer counts

x = rand(1:4, n)
y = rand(1:5, n)
w = weights(rand(n))

c = counts(x, y, (4, 5))
@test size(c) == (4, 5)
c0 = Int[count(t->t != 0,  (x .== i) .& (y .== j)) for i in 1:4, j in 1:5]
@test c == c0
@test counts(x .+ 2, y .+ 3, (3:6, 4:8)) == c0
@test proportions(x, y, (1:4, 1:5)) ≈ (c0 ./ n)

c = counts(x, y)
@test size(c) == (4, 5)
c0 = Int[count(t->t != 0, (x .== i) .& (y .== j)) for i in 1:4, j in 1:5]
@test c == c0
@test counts(x .+ 2, y .+ 3, (3:6, 4:8)) == c0
@test proportions(x, y,) ≈ (c0 ./ n)

c = counts(x, y, (4, 5), w)
@test size(c) == (4, 5)
c0 = Float64[sum(w.values[(x .== i) .& (y .== j)]) for i in 1:4, j in 1:5]
@test c                                     ≈ c0
@test counts(x .+ 2, y .+ 3, (3:6, 4:8), w) ≈ c0
@test proportions(x, y, (1:4, 1:5), w)      ≈ (c0 ./ sum(w))

c = counts(x, y, w)
@test size(c) == (4, 5)
c0 = Float64[sum(w.values[(x .== i) .& (y .== j)]) for i in 1:4, j in 1:5]
@test c                                     ≈ c0
@test counts(x .+ 2, y .+ 3, (3:6, 4:8), w) ≈ c0
@test proportions(x, y, w)                  ≈ (c0 ./ sum(w))

# count map

x = ["a", "b", "a", "a", "b", "c"]
w = [1.0, 1.5, 2.0, 2.5, 3.0, 3.5]

cm = countmap(x)
@test cm["a"] == 3
@test cm["b"] == 2
@test cm["c"] == 1
pm = proportionmap(x)
@test pm["a"] ≈ (1/2)
@test pm["b"] ≈ (1/3)
@test pm["c"] ≈ (1/6)

# testing the radixsort branch of countmap
xx = repeat([6, 1, 3, 1], outer=100_000)
cm = countmap(xx)
@test cm == Dict(1 => 200_000, 3 => 100_000, 6 => 100_000)

# testing the radixsort-based addcounts
xx = repeat([6, 1, 3, 1], outer=100_000)
cm = Dict{Int, Int}()
StatsBase.addcounts_radixsort!(cm,xx)
@test cm == Dict(1 => 200_000, 3 => 100_000, 6 => 100_000)
xx2 = repeat([7, 1, 3, 1], outer=100_000)
StatsBase.addcounts_radixsort!(cm,xx2)
@test cm == Dict(1 => 400_000, 3 => 200_000, 6 => 100_000, 7 => 100_000)

# testing the Dict-based addcounts
cm = Dict{Int, Int}()
StatsBase.addcounts_dict!(cm,xx)
@test cm == Dict(1 => 200_000, 3 => 100_000, 6 => 100_000)

cm = countmap(x, weights(w))
@test cm["a"] == 5.5
@test cm["b"] == 4.5
@test cm["c"] == 3.5

@test cm == countmap(x, w)

pm = proportionmap(x, weights(w))
@test pm["a"] ≈ (5.5 / 13.5)
@test pm["b"] ≈ (4.5 / 13.5)
@test pm["c"] ≈ (3.5 / 13.5)

# testing small bits type
bx = [true, false, true, true, false]
@test countmap(bx) == Dict(true => 3, false => 2)

for T in [UInt8, UInt16, Int8, Int16]
    tx = T[typemin(T), 8, typemax(T), 19, 8]
    @test countmap(tx) == Dict(typemin(T) => 1, typemax(T) => 1, 8 => 2, 19 => 1)
end

@testset "views" begin
    X = view([1,1,1,2,2], 1:5)
    @test countmap(X) == countmap(copy(X))
end


using IntervalArithmetic, IntervalRootFinding
using Base.Test

@testset "`bisect` function" begin
    X = 0..1
    @test bisect(X) == (0..0.5, 0.5..1)
    @test bisect(X, 0.25) == (0..0.25, 0.25..1)

    X = -∞..∞
    @test bisect(X) == (-∞..0, 0..∞)
    @test bisect(X, 0.75) == (-∞..0, 0..∞)

    X = 1..∞
    @test bisect(X) == (Interval(1, prevfloat(∞)), Interval(prevfloat(∞), ∞))

    X = (0..1) × (0..2)
    @test bisect(X) == ( (0..1) × (0..1), (0..1) × (1..2) )
    @test bisect(X, 0.25) == ( (0..1) × (0..0.5), (0..1) × (0.5..2) )
    @test bisect(X, 1, 0.5) == ( (0..0.5) × (0..2), (0.5..1) × (0..2) )
    @test bisect(X, 1, 0.25) == ( (0..0.25) × (0..2), (0.25..1) × (0..2) )

    X = (-∞..∞) × (-∞..∞)
    @test bisect(X) == ( (-∞..0) × (-∞..∞), (0..∞) × (-∞..∞))
end


"""
    sv = selectElement(v, sy, dw=[1,1,...,1], tri=0)

Select elements of a multi-element DAC to minimize the selection error,
subject to the constraint that the nominal DAC output is `v`, i.e. `v` = (`sv`)ᵀ(`dw`).
Assume that the preferred usage order is that given by `dw`.

Input
- `v`:    DSM output in -M:2:M
- `sy`:   M×1 desired usage vector
- `dw`:   M×1 vector of element weights
- `tri`:  Flag indicating the elements are tri-level, i.e. `sv` = 0 is allowed. NOT SUPPORTED YET
Output
- `sv`:   M×1 selection vector. ±1 if `tri` = 0, {0,±1} if `tri` = 1.
"""
function selectElement(v, sy, dw=[], tri=0)
    M = length(sy)
    if isempty(dw) || all(dw .== 1)
        dw_is_one = true
        sdw = M
    else
        dw_is_one = false
        sdw = sum(dw)
    end

    if abs(v) > sdw
        error("|v| is too large.");
    elseif v == sdw
        return ones(M)
    elseif v == -sdw
        return -ones(M)
    end

    sv = -ones(Int, M)
    possibilities = sortperm(-sy)  # Determine the element priority

    # Speed up selection in the usual case where all element weights are one.
    # Suggested 1997-03-05 by J. A. Cherry.
    if dw_is_one
        sv[possibilities[1:(M+v)÷2]] .= 1
        return sv
    end

    # Go through sv possibilities one by one, until one which meets the
    # v = sv'*dw constraint is found.
    i = 1                    # Selection level
    pointer = ones(Int, M)   # Array of pointers to selected elements
    selected = zeros(Int, M) # Selected elements
    dw0 = [0; dw]
    while true
        while pointer[i] > M
            # backtrack
            i = i - 1
            if i == 0
                break   # failure!
            end
            pointer[i] = pointer[i] + 1;
            selected[i+1:end] .= 0
        end
        selected[i] = possibilities[pointer[i]]
        dv = 2 * sum(dw0[selected.+1]) - sdw
        if dv == v
            break       #success!
        elseif dv < v
            # Proceed to the next level of selection
            pointer[i+1] = pointer[i] + 1
            i = i + 1
        else
            # Try the next element at the current level.
            pointer[i] = pointer[i] + 1
        end
    end
    selected = selected[selected .!= 0]
    sv[selected] .= 1

    return sv
end


###############################################################################
## LASSO Problem
###############################################################################
function get_randomlasso(n, m, sparsity; reg=regularizer_l1, seed=1234)
    # A is drawn from the standard normal distribution
    # b = Ax0 + e where e is taken from the normal distribution with standard deviation 0.001.
    # We set λ1 so that the original sparsity is ultimately recovered.

    Random.seed!(seed)
    A = rand(Normal(), m, n)

    Random.seed!(seed)
    x0 = rand(Normal(), n)

    if reg == regularizer_l1
        Random.seed!(seed)
        inds_nz = Set(randperm(n)[1:sparsity])

        for i in 1:n
            !(i in inds_nz) && (x0[i] = 0)
        end

    elseif reg <: regularizer_l12
        T = reg.parameters[1]
        ngroups = Int(ceil(n/T))

        Random.seed!(seed)
        inds_nz = Set(randperm(ngroups)[1:sparsity])

        for i in 1:ngroups-1
            !(i in inds_nz) && (x0[T*i-3:T*i] .= 0)
        end
        !(ngroups in inds_nz) && (x0[T*ngroups-3:end] .= 0)

    elseif reg == regularizer_lnuclear
        n_mat = Int(sqrt(n))

        Random.seed!(seed)
        basisvecs = rand(Normal(), n_mat, sparsity)

        x0 = zeros(n)
        for sp in 1:sparsity
            x0 += vec(basisvecs[:, sp] * transpose(basisvecs[:, sp]))
        end

    elseif reg <: regularizer_distball
        p = reg.parameters[1]

        Random.seed!(seed)
        x0 = rand(Normal(), n); x0 /= norm(x0, p)

    else
        @error "hunhandled regularizer $reg"
    end

    delta = 0.01
    Random.seed!(seed)
    e = rand(Normal(0, delta^2), m)

    y = A*x0+e
    pb = LassoPb{reg}(A, y, delta, n, x0)

    return pb
end

function get_IM_testcase(; n=80, m=130, sparsity=0.1)
    # A is drawn from the standard normal distribution
    # b = Ax0 + e where x0 is a 10% sparse vector taken from the normal distribution,
    # e is taken from the normal distribution with standard deviation 0.001.
    # We set λ1 so that the original sparsity is ultimately recovered.

    Random.seed!(23)
    A = rand(Normal(), m, n)

    Random.seed!(23)
    x0 = rand(Normal(), n)
    for i in 1:n
        rand() < sparsity && (x0[i]=0)
    end

    Random.seed!(23)
    e = rand(Normal(0, 0.001), m)

    y = A*x0+e

    return LassoPb{regularizer_l1}(A, y, 1., n, x0)
end

function get_randomlassopb_TV(; n, p, λ=1, TVsparsity=0.05)
    Random.seed!(23)
    A = rand(Float64, p, n)

	y_orig = zeros(n)
	y = zeros(n)
	y_orig[1] = 10

	## Ground truth vector
	Random.seed!(23); sparsity = floor.(rand(n-1) .+ TVsparsity)
	Random.seed!(23);
	aux = rand(Normal(0, 4.0), n-1) .* sparsity
	for i in 2:n
		y_orig[i] = y_orig[i-1] + aux[i-1]
	end

	## Noisy observation
	Random.seed!(23); pert = rand(Normal(0, 1.0), n)
    y = y_orig .+ pert

    y_obs = A * y

    return LassoPb{regularizer_TV}(A, y_obs, λ, n, y_orig)
end

function build_2dlasso_normdsol(;p=2, θ = π*8/(9*2))
    xcenter = [cos(θ), sin(θ)]
    xcenter /= norm(xcenter, p)

    diagscaling = [1, 10]
    A = Diagonal(diagscaling)
    y = diagscaling .* xcenter
    λ = 1

    return LassoPb{regularizer_distball{p}}(A, y, λ, 2, xcenter)
end

###############################################################################
## Logistic Problem
###############################################################################

## ionosphere problem
function get_ionopb()
    @assert isfile("data/ionosphere.data")

    rawdata = readdlm("data/ionosphere.data", ',')
    A = Matrix{Float64}(rawdata[:, 1:end-1])
    n = size(A, 2)

    y = map(x-> x=="g" ? 1 : -1, rawdata[:, end])

    return LogisticPb{regularizer_l1}(A, y, 0.1, n, zeros(1))
end

## Random sparse logit
function get_logit(;n=80, m=85, sparsity=0.5)

    A = rand(m, n)*10
    y = Vector([rand()>sparsity for i in 1:m])

    return LogisticPb{regularizer_l1}(A, y, 0.1, n, zeros(1))
end


#
# 1. Custom initial condition setup
#    - ClimateMachine.HydrostaticBoussinesq.ocean_init_state!
#

# Setup types and globals we need
using StaticArrays
og=ClimateMachine.HydrostaticBoussinesq.OceanGyre

"""
    ocean_init_state!(::og)

initialize u,v,η with 0 and θ linearly distributed between 9 at z=0 and 1 at z=H

# Arguments
- `p`: OceanGyre problem object, used to dispatch on and obtain ocean height H
- `Q`: state vector
- `A`: auxiliary state vector, not used
- `coords`: the coordidinates
- `t`: time to evaluate at, not used
"""
function ClimateMachine.HydrostaticBoussinesq.ocean_init_state!(p::OceanGyre, Q, A, coords, t)
    @inbounds y = coords[2]
    @inbounds z = coords[3]
    @inbounds H = p.H

    Q.u = @SVector [0, 0]
    Q.η = 0

    # Wavy
    # Q.θ = (5 + 4 * cos(y * π / p.Lʸ)) * (1 + z / H)

    # Constant everywhere
    # Q.θ = 9

    # Big number
    # Q.θ = 50

    # Unstable column
    #
    # Define function with inversion near surface and stable stratification below
    #
    # $ \theta(z) = Ae^{\frac{-(z+L)}{L}} - Ce^{\frac{-D(z+L)}{L}} + B + Ez$
    #
    H=4000.;L=H/10.;A=20.;C=50.;D=2.5;B=8.;E=5.e-4;
    ft(xx,L)=exp(-xx/L);
    th1(zz)=A*ft.(-zz .+ L, L);
    th2(zz)=C*ft(D*(-zz .+ L), L);
    phi1=th1.(z)
    phi2=th2.(z)
    Q.θ = phi1 .- phi2 .+ B .+ E .* z;

    return nothing
end


using Sundials
function lorenz!(du,u,p,t)
 du[1] = 10.0(u[2]-u[1])
 du[2] = u[1]*(28.0-u[3]) - u[2]
 du[3] = u[1]*u[2] - (8/3)*u[3]
end

u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz!,u0,tspan)

sol = solve(prob,CVODE_Adams())
sol(90:1:100, idxs=1).u isa Array{Float64}


using ConvexTests, CSDP

using MathOptInterface
const MOI = MathOptInterface
using Dualization

@info "Starting CSDP tests"

exclusions =  [r"mip", r"exp", r"sdp_Complex_Semidefinite_constraint"]

do_tests("CSDP", () -> CSDP.Optimizer(printlevel=0); exclude = exclusions, first = true, last = false)

function dual_opt()
    DualOptimizer(CSDP.Optimizer(printlevel=0))
end

do_tests("CSDP", dual_opt; exclude = exclusions, first = false, last = true, variant="(dualized)")


@testset "Misc/NumberField" begin
  @testset "issubfield" begin
    Qx, x = FlintQQ["x"]
    K, a = NumberField(x^2 + 1, "a")
    L, b = NumberField(x^4 + 1, "b")

    c, KtoL = Hecke.issubfield(K, L)
    @test c == true
    @test parent(KtoL(a)) == L

    c, KtoL = Hecke.issubfield_normal(K, L)
    @test c == true
    @test parent(KtoL(a)) == L

    OK = maximal_order(K)
    OL = maximal_order(L)
    c, KtoL = Hecke.issubfield(K, L)
    @test c == true
    @test parent(KtoL(a)) == L
  end

  @testset "isisomorphic" begin
    Qx, x = FlintQQ["x"]
    f = x^5 + 12x - 92
    K, a = NumberField(f, "a")

    g = x^5 - 172x^4 + 7024x^3 + 8656448x^2 + 55735552x + 45796197888
    K2, a2 = NumberField(g, "a2")

    c, KtoK2 = Hecke.isisomorphic(K, K2)
    @test c == true
    @test parent(KtoK2(a)) == K2

    OK = maximal_order(K)
    OK2 = maximal_order(K2)
    c, KtoK2 = Hecke.isisomorphic(K, K2)
    @test c == true
    @test parent(KtoK2(a)) == K2

    h = f - 1
    K3, a3 = NumberField(h, "a3")
    d, KtoK3 = Hecke.isisomorphic(K, K3)
    @test d == false
  end
end


mutable struct SpecificInfo <: InfoDist
    joint::Matrix{Int}
    m1::Vector{Int}
    m2::Vector{Int}
    b1::Int
    b2::Int
    N::Int
    function SpecificInfo(b1::Int, b2::Int)
        if b1 < 1 || b2 < 1
            throw(ArgumentError("the support of each random variable must be at least 2"))
        end
        new(zeros(Int, b1, b2), zeros(Int, b1), zeros(Int, b2), b1, b2, 0)
    end
end

function SpecificInfo(xs::AbstractVector{Int}, ys::AbstractVector{Int})
    if isempty(xs) || isempty(ys)
        throw(ArgumentError("arguments must not be empty"))
    end
    xmin, xmax = extrema(xs)
    ymin, ymax = extrema(ys)
    if xmin < 1 || ymin < 1
        throw(ArgumentError("observations must be positive, nonzero"))
    end
    observe!(SpecificInfo(max(2, xmax), max(2, ymax)), xs, ys)
end

function observe!(dist::SpecificInfo, xs::AbstractVector{Int}, ys::AbstractVector{Int})
    if length(xs) != length(ys)
        throw(ArgumentError("arguments must have the same length"))
    end
    dist.N += length(xs)
    for i in eachindex(xs)
        dist.joint[xs[i], ys[i]] += 1
        dist.m1[xs[i]] += 1
        dist.m2[ys[i]] += 1
    end
    dist
end

function estimate(dist::SpecificInfo)
    si = zeros(dist.b1)
    N = dist.N
    for x1 in eachindex(dist.m1)
        n1 = dist.m1[x1]
        for x2 in eachindex(dist.m2)
            if !iszero(dist.joint[x1,x2])
                j, n2 = dist.joint[x1,x2], dist.m2[x2]
                si[x1] += j * log2((N * j) / (n1 * n2))
            end
        end
        si[x1] /= n1
    end
    si
end

@inline function clear!(dist::SpecificInfo)
    dist.joint[:] .= 0
    dist.m1[:] .= 0
    dist.m2[:] .= 0
    dist.N = 0
    dist
end

function specificinfo!(si::SpecificInfo, stimulus::AbstractVector{Int},
                       responses::AbstractMatrix{Int})
    specificinfo!(si, stimulus, box(responses))
end

function specificinfo!(si::SpecificInfo, stimulus::AbstractVector{Int},
                       responses::AbstractMatrix{Int}, subset::AbstractVector{Int})
    specificinfo!(si, stimulus, @view responses[subset, :])
end

function specificinfo!(si::SpecificInfo, stimulus::AbstractVector{Int},
                       responses::AbstractVector{Int})
    estimate(observe!(dist, stimulus, responses))
end

function specificinfo(stimulus::AbstractVector{Int}, responses::AbstractMatrix{Int})
    specificinfo(stimulus, box(responses))
end

function specificinfo(stimulus::AbstractVector{Int}, responses::AbstractMatrix{Int},
                      subset::AbstractVector{Int})
    specificinfo(stimulus, @view responses[subset, :])
end

function specificinfo(stimulus::AbstractVector{Int}, responses::AbstractVector{Int})
    estimate(SpecificInfo(stimulus, responses))
end


using FractionalDiffEq, Plots, SpecialFunctions
analytical(x)=exp(1+x)*erfc(sqrt(1+x))
tspan = LinRange(-1, 1, 100)
prob = FIEProblem([1, 1], [0.5, 0], 1, tspan)
sol = solve(prob, 20, SpectralUltraspherical())
solanalytical = analytical.(tspan)
plot(sol, title="Second kind Abel integral equation", label="Numerical")
plot!(tspan, solanalytical, ls=:dash, label="Analytical")


using LineSearches
using LinearAlgebra

function ls(uk::Vector{Float64}, grad_L::Vector{Float64}, Lu::Function, grad_Lu::Function)
    s = copy(-grad_L)
    Lalpha(alpha) = Lu(uk .+ alpha.*s)
    function grad_Lalpha(alpha)
        return dot(grad_Lu(uk .+ alpha .* s), s)
    end
    function Lgrad_Lalpha(alpha)
        gvec = grad_Lu(uk .+ alpha .* s)
        phi = Lu(uk .+ alpha .*s)
        dphi = dot(gvec, s)
        return (phi, dphi)
    end
    dL_0 = dot(s, grad_L)
    obj = Lu(uk)
    ls_ = BackTracking()
    alpha, obj = ls_(Lalpha, grad_Lalpha, Lgrad_Lalpha, 1.0, obj, dL_0)
    return alpha
end


tri_s = MYPolygon([0, 1, 1], [0, 0, 1])
tri_m = MYPolygon([0, √2, √2], [0, 0, √2])
tri_l = MYPolygon([0, 2, 2], [0, 0, 2])
square_s = MYPolygon([0, 1, 1, 0], [0, 0, 1, 1])
parallelogram = MYPolygon([0, √2, √(2)*3/2, √(2)/2], [0, 0, √(2)/2, √(2)/2])

# tri_m, square_s
house2 = MYPolygon([0, 1, 1, 1.5, 0.5, -0.5, 0], [0, 0, 1, 1, 2, 1, 1])
p_house2 = ("house2", house2, [tri_m, square_s])

# tri_l, tri_m, tri_s, tri_s
house4 = MYPolygon([-0.5*√2, 0.5*√2, 0.5*√2, √2, 0, -√2, -0.5*√2], [0, 0, √2, √2, 2*√2, √2, √2])
p_house4 = ("house4", house4, [tri_l, tri_m, tri_s, tri_s])

# tri_l, square_s
arrow2 = MYPolygon([0, 3, 1, 1, 0], [0, 0, 2, 1, 1])
p_arrow2 = ("arrow2", arrow2, [tri_l, square_s])

# tri_s, parallelogram
trapezoid2_1 = MYPolygon([-√2, √2, 0.5*√2, -0.5*√2], [0, 0, 0.5*√2, 0.5*√2])
p_trapezoid2_1 = ("trapezoid2_1", trapezoid2_1, [tri_s, parallelogram])

# tri_s, parallelogram
trapezoid2_2 = MYPolygon([0, 0.5*√2, 0, -√2], [0, 0.5*√2, √2, 0])
p_trapezoid2_2 = ("trapezoid2_2", trapezoid2_2, [tri_s, parallelogram])

# tri_s, square_s
tri_above_square = MYPolygon([0, 1, 1, 0, 0, 1, 0, 0, 0], [0, 0, 1, 1, 2, 3, 3, 2, 1])
p_tri_above_square = ("tri_above_square", tri_above_square, [tri_s, square_s])

# tri_s, square_s
fish2 = MYPolygon([0, 0.5*√2, √2, 0.5*√2, 0, -0.5*√2, -0.5*√2], [0, -0.5*√2, 0, 0.5*√2, 0, 0.5*√2, -0.5*√2])
p_fish2 = ("fish2", fish2, [tri_s, square_s])

# tri_m, tri_s
small_stand = MYPolygon([0, 1, 1, √2, 0], [0, 0, 1, √2, √2])
p_small_stand = ("small_stand", small_stand, [tri_m, tri_s])

# tri_m, tri_s, tri_s
triforce3 = MYPolygon([0, -1, -0.5*√2 - 1, 0.5*√2 - 1, -1, 1, 1 - 0.5*√2, 1 + 0.5*√2, 1], [1, 0, -0.5*√2, -0.5*√2, 0, 0, -0.5*√2, -0.5*√2, 0])
p_triforce3 = ("triforce3", triforce3, [tri_m, tri_s, tri_s])

# tri_s, tri_s, parallelogram
# tri_s, tri_s, square_s
rect_s = MYPolygon([0, 2, 2, 0], [0, 0, 1, 1])
p_rect_s = ("rect_s", rect_s, [tri_s, tri_s, parallelogram])

# tri_s, tri_s, parallelogram
hurt3 = MYPolygon([0, 1, 1, 0, -1, -1], [0, 1, 2, 1, 2, 1])
p_hurt3 = ("hurt3", hurt3 , [tri_s, tri_s, parallelogram])

# tri_m, tri_s, tri_s, parallelogram
# tri_m, tri_s, tri_s, square_s
hexagon_m = MYPolygon([0, √2, 2.12132, √2, 0, -0.7071068], [0, 0, 0.7071068, √2, √2, 0.7071068])
p_hexagon_m1 = ("hexagon_m", hexagon_m, [tri_m, tri_s, tri_s, parallelogram])
p_hexagon_m2 = ("hexagon_m", hexagon_m, [tri_m, tri_s, tri_s, square_s])

# tri_m, tri_s, tri_s
square_3 = MYPolygon([0, √2, √2, 0], [0, 0, √2, √2])
p_square_3 = ("square_3", square_3, [tri_m, tri_s, tri_s])

# tri_l, tri_s, tri_s, square_s
# tri_l, tri_m, tri_s, square_s
# tri_l, tri_s, tri_s, parallelogram
square_4 = MYPolygon([0, 2, 2, 0], [0, 0, 2, 2])
p_square_4 = ("square_4", square_4, [tri_l, tri_s, tri_s, square_s])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
square_7 = MYPolygon([0, 2*√2, 2*√2, 0], [0, 0, 2*√2, 2*√2])
p_square_7 = ("square_7", square_7, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_m, tri_s, tri_s
butterfly_s = MYPolygon([0, -1, 1, 0, 1, -1], [0, -1, -1, 0, 1, 1])
p_butterfly_s = ("butterfly_s", butterfly_s, [tri_m, tri_s, tri_s])

# tri_l, tri_l
butterfly = MYPolygon([0, √2, 2*√2, 2*√2, √2, 0], [0, √2, 0, 2*√2, √2, 2*√2])
p_butterfly = ("butterfly", butterfly, [tri_l, tri_l])

# tri_l, tri_l
pesudo_butterfly = MYPolygon([-1.16, -0.25, 1.16, 1.16, 0.25, -1.16], [-1.16, -0.25, -1.67, 1.16, 0.25, 1.67])
p_pesudo_butterfly = ("pesudo_butterfly", pesudo_butterfly, [tri_l, tri_l])

# tri_l, tri_l, tri_s, tri_s
butterfly4 = MYPolygon([-√2, -0.5*√2, -0.5*√2, -√2, √2, 0.5*√2, 0.5*√2, √2], [√2, 0.5*√2, -0.5*√2, -√2, -√2, -0.5*√2, 0.5*√2, √2])
p_butterfly4 = ("butterfly4", butterfly4, [tri_l, tri_l, tri_s, tri_s])

# tri_s, square_s, parallelogram
note = MYPolygon([0, 0.5*√2, 0.5*√2, 1.70, 0, 0, -0.5*√2], [-0.5*√2, 0, 0.413, 0.413, 1.5*√2, 0.5*√2, 0])
p_note = ("note", note, [tri_s, square_s, parallelogram])

# tri_l, tri_s, parallelogram
diamond3 = MYPolygon([0, √2, 0.5*√2, -0.5*√2, -√2], [-√2, 0, 0.5*√2, 0.5*√2, 0])
p_diamond = ("diamond", diamond3, [tri_l, tri_s, parallelogram])

# tri_l, tri_s, tri_s, square_s
antena = MYPolygon([-√2, √2, 0, 0.5*√2, 0.5*√2, -√2, 0], [0, 0, √2, 1.5*√2, 3.53, √2, √2])
p_antena = ("antena", antena, [tri_l, tri_s, tri_s, square_s])

# tri_l, tri_m, tri_s, square_s
pencile4 = MYPolygon([0, √2, √2, 0.5*√2, 0], [0, 0, 2*√2, 2.5*√2, 2*√2])
p_pencile4 = ("pencile4", pencile4, [tri_l, tri_m, tri_s, square_s])

# tri_l, tri_s, tri_s
# tri_m, tri_s, tri_s, square_s
# tri_m, tri_s, tri_s, parallelogram
crown4 = MYPolygon([-√2, √2, √2, 0.5*√2, 0, -0.5*√2, -√2], [0, 0, √2, 0.5*√2, √2, 0.5*√2, √2])
p_crown = ("crown", crown4, [tri_l, tri_s, tri_s])

# tri_l, tri_l, tri_s, tri_s
snake = MYPolygon([-2, -1, -1, 1, 1, 0, 0, -2], [-2, -2, -1, -1, 1, 1, 0, 0])
p_snake = ("snake", snake, [tri_l, tri_l, tri_s, tri_s])

# tri_m, tri_s, tri_s, square_s, parallelogram
hexagon_5 = MYPolygon([-1.5, -0.5, 0.5, 1.5, 0.5, -0.5], [0, -1, -1, 0, 1, 1])
p_hexagon5 = ("hexagon5", hexagon_5, [tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_s, tri_s, tri_s, tri_s, tri_s, tri_s
egu_force = MYPolygon([-1, 0, -1, 0, 0, 1, -1, 0, 0, 1, 1, 2, -1], [-2, -1, -1, 0, -1, 0, 0, 1, 0, 1, 0, 1, 1])
p_egu_force = ("egu_force", egu_force, [tri_s, tri_s, tri_s, tri_s, tri_s, tri_s])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
faucet = MYPolygon([-2, -1, -1, 2, 2, 1, 1, 2, -1, 0, 0, -2], [-2, -2, -1, -1, 0, 0, 1, 2, 2, 1, 0, 0])
p_faucet = ("faucet", faucet, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_l, tri_m, tri_s, square_s, parallelogram
Taiikusuwaring_human = MYPolygon([1, -0.5*√2, -√2, -0.5*√2, -2, -√2, -1.5*√2, -0.5*√2, 1-√2, 1], [0, 1+0.5*√2, 1, 0.291, -1, -1, -1-0.5*√2, -1-0.5*√2, -√2, √2])
p_Taiikusuwari = ("Taiikusuwari", Taiikusuwaring_human, [tri_l, tri_m, tri_s, square_s, parallelogram])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
crab_scissor = MYPolygon([0, √2, √2, 1.5*√2, 0.5*√2, 0.5*√2, 0.5*√2-1, 0.5*√2-1, -1-0.5*√2, -0.5*√2, -0.5*√2, -√2, -0.5*√2], [0, 0, √2, 1.5*√2, 2.5*√2, 1+0.5*√2, 1+0.5*√2, 1+2.5*√2, 1+1.5*√2, 1.5*√2, 0.5*√2, 0, -0.5*√2])
p_crab_scissor = ("crab_scissor", crab_scissor, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
candy7 = MYPolygon([-1.5-√2, -1.5, -0.5, 0.5, 1.5, 1.5+√2, 1.5+√2, 1.5, 0.5, -0.5, -1.5, -1.5-√2], [-√2, 0, -1, -1, 0, -√2, √2, 0, 1, 1, 0, √2])
p_candy7 = ("candy7", candy7, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# ミスアリ tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
flame6 = MYPolygon([0, -0.5, -0.5, -1, 0, -√2, -0.5*√2, -0.5*√2, -√2, √2, 0.5*√2, 0.5*√2, √2, 0, 1, 0.5, 0.5], [2, 2, 1, 1, 0, 0, -0.5*√2, -1.5*√2, -2*√2, -2*√2, -1.5*√2, -0.5*√2, 0, 0, 1, 1, 2])
# p_flame6 = ("flame6", flame6, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
flame7 = MYPolygon([1, 0, 0, -0.5, -0.5, -1, 0, -√2, -0.5*√2, -0.5*√2, -√2, √2, 0.5*√2, 0.5*√2, √2, 0, 1, 0.5, 0.5, 0, 1], [4, 3, 2, 2, 1, 1, 0, 0, -0.5*√2, -1.5*√2, -2*√2, -2*√2, -1.5*√2, -0.5*√2, 0, 0, 1, 1, 2, 2, 3])
p_flame7 = ("flame7", flame7, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
hurt7 = MYPolygon([-2, -1, -1, -0.294, 0, 2, √2, 0.5*√2, 1, 1, 0, 0, -0.5*√2, -√2, -0.5*√2, -√2], [0, -1, -1-√2, -1-0.5*√2, -2, 0, 0, 0.5*√2, 1, 2, 1, √2, 1.5*√2, √2, 0.5*√2, 0])
p_hurt7 = ("hurt7", hurt7, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])

# tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram
katana = MYPolygon([-1, 0, 1, 2, 1.22, 5.63, 5.63, 4.62, 4.62, 3.20, 3.20, 1.21, 1.21, 0, 0, -1],
                   [0, 0, 1, 1, 1.79, 6.21, 7.21, 6.22, 5.21, 5.21, 3.79, 3.79, 1.79, 3, 2, 1])
p_katana = ("katana", katana, [tri_l, tri_l, tri_m, tri_s, tri_s, square_s, parallelogram])


function maximumpoly(poly::arb_poly,
                     (lower, upper)::Tuple{arb, arb};
                     absmax::Bool = false)
    x = setinterval(lower, upper)
    maybeabs = ifelse(absmax, abs, identity)

    # Enclose the zeros of the derivative of the polynomial
    deriv = derivative(poly)

    roots = isolateroots(deriv, lower, upper)[1]

    # Evaluate the polynomial on the zeros and the endpoints and take
    # the maximum
    m = max(maybeabs(evaluate(poly, lower)),
            maybeabs(evaluate(poly, upper)))

    # FIXME: If lower and upper are not tight then it can happen that
    # we find a root which is contained in x but not in the true
    # interval. If the maximum occurs at this root we would get wrong
    # results.
    for root in roots
        m = max(m,
                maybeabs(evaluate(poly, setinterval(root...))))
    end

    m
end


# rk4(

function lorenz(t, x_vect)

  sigma = 10
  beta = 8/3
  rho = 28

  x = x_vect[1]
  y = x_vect[2]
  z = x_vect[3]

  f = Array{Float64}(3)

  f[1] = sigma*(y - x)
  f[2] = x*(rho - z) - y
  f[3] = x*y - beta*z

  return f

end


@inline function abs_fp_dd(x::T) where {T<:AbstractFloat}
    return abs(x), zero(T)
end

@inline function negabs_fp_dd(x::T) where {T<:AbstractFloat}
    return -abs(x), zero(T)
end

@inline function neg_fp_dd(x::T) where {T<:AbstractFloat}
    return -x, zero(T)
end

@inline function inv_fp_dd(x::T) where {T<:AbstractFloat}
    hi, lo = two_inv(x)
    return hi, lo
end

@inline function square_fp_dd(a::T) where {T<:AbstractFloat}
    zhi = a * a
    zlo = fma(a, a, -zhi)
    return zhi, zlo
end

@inline function cube_fp_dd(a::T) where {T<:AbstractFloat}
    zhi, zlo = square_fp_dd(a)
    zhi, zlo = mul_dddd_dd((zhi, zlo), (a, zero(T)))
    return zhi, zlo
end

@inline function powr2_fp_dd(x::T) where {T<:AbstractFloat}
    return powr2_(x)
end

@inline function powr3_fp_dd(x::T) where {T<:AbstractFloat}
    return powr3_(x)
end

@inline function powr4_fp_dd(x::T) where {T<:AbstractFloat}
    hi, lo3 = powr3_fp_dd(x)
    hi, lo = mul_(x, hi)
    lo = fma(lo3, x, lo)
    return hi, lo
end

@inline function powr5_fp_dd(x::T) where {T<:AbstractFloat}
    hi, lo4 = powr4_fp_dd(x)
    hi, lo = mul_(x, hi)
    lo = fma(lo4, x, lo)
    return hi, lo
end

@inline function powr6_fp_dd(x::T) where {T<:AbstractFloat}
    hi, lo5 = powr5_fp_dd(x)
    hi, lo = mul_(x, hi)
    lo = fma(lo5, x, lo)
    return hi, lo
end

@inline function root2_fp_dd(x::T) where {T<:AbstractFloat}
    return two_sqrt(x)
end

@inline function root3_fp_dd(x::T) where {T<:AbstractFloat}
    return two_cbrt(x)
end

@inline function root4_fp_dd(x::T) where {T<:AbstractFloat}
     return two_sqrt(sqrt(x))
end

@inline function root5_fp_dd(x::T) where {T<:AbstractFloat}
    throw(ErrorException("not implemented"))
end

@inline function root6_fp_dd(x::T) where {T<:AbstractFloat}
    return two_cbrt(sqrt(x))
end


using JuMP
using GLPKMathProgInterface
using Ipopt

n = 2;
lowRes = -4;
hiRes  = 4;
m = Model()
@defVar(m, lowRes <= α[i=1:n]  <=hiRes)
@defVar(m, lowRes <= β1[i=1:n] <=hiRes)
@defVar(m, lowRes <= β2[i=1:n] <=hiRes)
@defVar(m, lowRes <= β3[i=1:n] <=hiRes)
@defVar(m, lowRes <= β4[i=1:n] <=hiRes)

@defVar(m, lowRes <= A011 <= hiRes)
@defVar(m, lowRes <= A012 <= hiRes)
@defVar(m, lowRes <= A021 <= hiRes)
@defVar(m, lowRes <= A022 <= hiRes)

@defVar(m, lowRes <= A111 <= hiRes)
@defVar(m, lowRes <= A112 <= hiRes)
@defVar(m, lowRes <= A121 <= hiRes)
@defVar(m, lowRes <= A122 <= hiRes)

@defVar(m, lowRes <= B011 <= hiRes)
@defVar(m, lowRes <= B012 <= hiRes)
@defVar(m, lowRes <= B021 <= hiRes)
@defVar(m, lowRes <= B022 <= hiRes)

@defVar(m, lowRes <= B111 <= hiRes)
@defVar(m, lowRes <= B112 <= hiRes)
@defVar(m, lowRes <= B121 <= hiRes)
@defVar(m, lowRes <= B122 <= hiRes)

A0 = [A011 A012
     A021 A022]
A1 = [A111 A112
     A121 A122]
B0 = [B011 B012
      B021 B022]
B1 = [B111 B112
      B121 B122]
e = [1 1]'

@setObjective(m, Max, 0)
@addConstraints(m,begin
                  sum{α[i], i=1:n}==1
                  sum{β1[i], i=1:n}==1
                  sum{β1[i], i=1:n}==1
                  sum{β2[i], i=1:n}==0
                  sum{β3[i], i=1:n}==0
                  sum{β4[i], i=1:n}==0
                  sum{β4[i], i=1:n}==0
                end)
@addNLConstraints(m,begin
                  B111*β1[1]+B112*β1[1]+B121*β1[2]+B122*β1[2]==0
                  B111*β2[1]+B112*β2[1]+B121*β2[2]+B122*β2[2]==1
                  B111*β3[1]+B112*β3[1]+B121*β3[2]+B122*β3[2]==0
                  B111*β4[1]+B112*β4[1]+B121*β4[2]+B122*β4[2]==0
                  A011*α[1] +A012*α[1] +A021*α[2] +A022*α[2] ==1//2
                  B011*α[1] +B012*α[1] +B021*α[2] +B022*α[2] ==1
                  A111*β1[1]+A112*β1[1]+A121*β1[2]+A122*β1[2]==1
                  A111*β2[1]+A112*β2[1]+A121*β2[2]+A122*β2[2]==0
                  A111*β3[1]+A112*β3[1]+A121*β3[2]+A122*β3[2]==-1
                  A111*β4[1]+A112*β4[1]+A121*β4[2]+A122*β4[2]==0
                end)

@addNLConstraint(m,α[1] * (B011+B012)*(B011+B012)  + α[2] * (B021+B022)*(B021+B022)  == 3//2)
@addNLConstraint(m,β1[1] * (B111+B112)*(B111+B112)  + β1[2] * (B121+B122)*(B121+B122)  == 1)
@addNLConstraint(m,β2[1] * (B111+B112)*(B111+B112)  + β2[2] * (B121+B122)*(B121+B122)  == 0)
@addNLConstraint(m,β3[1] * (B111+B112)*(B111+B112)  + β3[2] * (B121+B122)*(B121+B122)  == -1)
@addNLConstraint(m,β4[1] * (B111+B112)*(B111+B112)  + β4[2] * (B121+B122)*(B121+B122)  == 2)
@addNLConstraint(m,β1[1] * (B111*(B111+B112) + B112*(B121+B122))  + β1[2] * (B121*(B111+B112) + B122*(B121+B122))  == 0)
@addNLConstraint(m,β2[1] * (B111*(B111+B112) + B112*(B121+B122))  + β2[2] * (B121*(B111+B112) + B122*(B121+B122))  == 0)
@addNLConstraint(m,β3[1] * (B111*(B111+B112) + B112*(B121+B122))  + β3[2] * (B121*(B111+B112) + B122*(B121+B122))  == 0)
@addNLConstraint(m,β4[1] * (B111*(B111+B112) + B112*(B121+B122))  + β4[2] * (B121*(B111+B112) + B122*(B121+B122))  == 1)

@addNLConstraint(m,(1/2)*((A111*(B011+B012) + A112*(B021+B022))*β1[1] +
      (A121*(B011+B012) + A122*(B021+B022))β1[2]) +
      (1/3)*((A111*(B011+B012) + A112*(B021+B022))β3[1] +
      (A121*(B011+B012) + A122*(B021+B022))β3[2]) == 0)

print(m)

status = solve(m)

println("Objective value: ", getObjectiveValue(m))
println("β1 = ", getValue(β1))
println("β2 = ", getValue(β2))
println("β3 = ", getValue(β3))
println("β4 = ", getValue(β4))


function perf_lucompletepivCopy!(A)
    n = size(A, 1)
    rowpiv=zeros(Int, n-1)
    colpiv=zeros(Int, n-1)
    for k = 1:n-1
        As = abs.(A[k:n, k:n])
        μ, λ = argmax(A).I
        μ += k-1; λ += k-1
        rowpiv[k] = μ
        A[[k,μ], 1:n] = A[[μ,k], 1:n]
        colpiv[k] = λ
        A[1:n, [k,λ]] = A[1:n, [λ,k]]
        if A[k,k] ≠ 0
            ρ = k+1:n
            A[ρ, k] = A[ρ, k]/A[k, k]
            A[ρ, ρ] = A[ρ, ρ] - A[ρ, k:k] * A[k:k, ρ]
        end
    end
    return (A, rowpiv, colpiv)
end

function perf_lucompletepivSub!(A)
    n = size(A, 1)
    rowpiv=zeros(Int, n-1)
    colpiv=zeros(Int, n-1)
    for k = 1:n-1
        As = abs.(view(A, k:n, k:n))
        μ, λ = argmax(A).I
        μ += k-1; λ += k-1
        rowpiv[k] = μ
        A[[k,μ], 1:n] = view(A, [μ,k], 1:n)
        colpiv[k] = λ
        A[1:n, [k,λ]] = view(A, 1:n, [λ,k])
        if A[k,k] ≠ 0
            ρ = k+1:n
            A[ρ, k] = view(A, ρ, k)/A[k, k]
            A[ρ, ρ] = view(A, ρ, ρ) - view(A, ρ, k:k) * view(A, k:k, ρ)
        end
    end
    return (A, rowpiv, colpiv)
end

function perf_gramschmidt!(U)
    m = size(U, 2)
    @inbounds for k = 1:m
        uk = view(U,:,k)
        for j = 1:k-1
            uj = view(U,:,j)
            uk .-= (uj ⋅ uk) .* uj
        end
        uk ./= norm(uk)
    end
end


function solve_diag(λ, Δ, g̃, seuil, α, ϵ; M = [0.0])
    # printstyled("on est ici ↓ \n", color = :red)
    T = eltype(α)
    # @show T
    if M == [0.0]
        # M = ones(Δ) ;
        # M = fill(T(1.0), size(Δ))
        M = ones(T, size(Δ))
    end
    λin = λ
    λtry = seuil
    dtry = -(Δ .+ λtry * M) .\ g̃
    normdtry = sqrt(dtry⋅dtry)
    seuiltry = normdtry / α
    d̃ = dtry

    #println(" Biss $biss_iter : λ = $λtry  seuil = $seuiltry ")

    # secant like heuristic to help Newton process by providing a good starting point
    global iter_bis = 0
    if λin < seuil
        while (λtry > seuiltry && (iter_bis < 10))
            λtry = λ + (λtry - λ) / max((seuil - seuiltry), T(2.0))
            dtry = -(Δ .+ λtry * M) .\ g̃
            normdtry = sqrt(dtry⋅dtry)
            seuiltry = normdtry / α
            global iter_bis += 1
            #println(" λ computation, iter_bis : $iter_bis  λtry  $λtry")
        end
        λ = λtry
        d̃ = dtry
        normd̃ = normdtry
        seuil = seuiltry
    end

    # Newton iterations
    global iter_nwt = 0
    while (abs(seuil - λ)/max(seuil,λ) > ϵ) && (iter_nwt < 40)
        dotd̃ = (Δ .+ λ * M) .\ (M .* d̃)
        Δλ = λ * (seuil - λ) / (seuil + λ * (λ * (d̃ ⋅ dotd̃) / (d̃ ⋅ d̃)))
        λ = max(λ + Δλ, λin + T(1.0e-10))
        λ = max(λ + Δλ, λin + (sqrt(eps(T) ./ 100.0)))
        d̃ = -(Δ .+ λ * M) .\ g̃
        normd̃ = sqrt(d̃⋅d̃)
        seuil = normd̃ / α

        @assert (λ >= 0.0)
        global iter_nwt += 1
    end
    #println(" λ computation, iter_bis : $iter_bis  iter_nwt : $iter_nwt  λ  $λ")
    #try assert((g̃ + 0.5*Δ.*d̃)⋅d̃ <= 0.0)  catch  @bp  end
    #println("*******SolveDiag:  g̃⋅d̃ = $(g̃⋅d̃), 0.5 d̃'Δd̃ = $(0.5*(Δ .* d̃)⋅d̃)")

    return d̃,λ
end


using DiffEqBase, StochasticDiffEq, Test

f1(u,p,t) = zero(u)
g1(u,p,t) = ones(size(u))
dt = 1//2^(4)
prob1 = SDEProblem{false}(f1,g1,0.,(0.0,1.0))
integrator = init(prob1,EM(),dt=dt,save_noise=true)

k = integrator.W.dW
@test integrator.W.dW != 0
solve!(integrator)
@test integrator.sol.W[2] == k

prob1 = SDEProblem{false}(f1,g1,zeros(4),(0.0,1.0))
sol = solve(prob1,EM(),dt=dt,save_noise=true)
@test sol.W[2] != zeros(4)

f1(du,u,p,t) = du.=0.
g1(du,u,p,t) = du.=1.
dt = 1//2^(4)
prob1 = SDEProblem(f1,g1,zeros(4),(0.0,1.0))
sol = solve(prob1,EM(),dt=dt,save_noise=true)
@test sol.W[2] != zeros(4)


# Separable sum, using tuples of arrays as variables

export SeparableSum

"""
**Separable sum of functions**

    SeparableSum(f_1, ..., f_k)

Given functions `f_1` to `f_k`, returns their separable sum, that is
```math
g(x_1, ..., x_k) = \\sum_{i=1}^k f_i(x_i).
```
The object `g` constructed in this way can be evaluated at `Tuple`s of length
`k`. Likewise, the `prox` and `prox!` methods for `g` operate with
(input and output) `Tuple`s of length `k`.

Example:

    f = SeparableSum(NormL1(), NuclearNorm()); # separable sum of two functions
    x = randn(10); # some random vector
    Y = randn(20, 30); # some random matrix
    f_xY = f((x, Y)); # evaluates f at (x, Y)
    (u, V), f_uV = prox(f, (x, Y), 1.3); # computes prox at (x, Y)
"""
struct SeparableSum{T <: Tuple} <: ProximableFunction
    fs::T
end

SeparableSum(fs::Vararg{ProximableFunction}) = SeparableSum((fs...,))

is_prox_accurate(f::SeparableSum) = all(is_prox_accurate.(f.fs))
is_convex(f::SeparableSum) = all(is_convex.(f.fs))
is_set(f::SeparableSum) = all(is_set.(f.fs))
is_singleton(f::SeparableSum) = all(is_singleton.(f.fs))
is_cone(f::SeparableSum) = all(is_cone.(f.fs))
is_affine(f::SeparableSum) = all(is_affine.(f.fs))
is_smooth(f::SeparableSum) = all(is_smooth.(f.fs))
is_quadratic(f::SeparableSum) = all(is_quadratic.(f.fs))
is_generalized_quadratic(f::SeparableSum) = all(is_generalized_quadratic.(f.fs))
is_strongly_convex(f::SeparableSum) = all(is_strongly_convex.(f.fs))

function (f::SeparableSum)(x::TupleOfArrays{R}) where R <: Real
    sum = R(0)
    for k in eachindex(x)
        sum += f.fs[k](x[k])
    end
    return sum
end

function prox!(ys::TupleOfArrays{R}, fs::Tuple, xs::TupleOfArrays{R}, gamma::R=R(1)) where R <: Real
    sum = R(0)
    for k in eachindex(xs)
        sum += prox!(ys[k], fs[k], xs[k], gamma)
    end
    return sum
end

function prox!(ys::TupleOfArrays{R}, fs::Tuple, xs::TupleOfArrays{R}, gamma::Tuple) where R <: Real
    sum = R(0)
    for k in eachindex(xs)
        sum += prox!(ys[k], fs[k], xs[k], gamma[k])
    end
    return sum
end

prox!(ys::TupleOfArrays{R}, f::SeparableSum, xs::TupleOfArrays{R}, gamma=R(1)) where R <: Real = prox!(ys, f.fs, xs, gamma)

function gradient!(grad::TupleOfArrays{R}, fs::Tuple, x::TupleOfArrays{R}) where R <: Real
    val = R(0)
    for k in eachindex(fs)
        val += gradient!(grad[k], fs[k], x[k])
    end
    return val
end

gradient!(grad::TupleOfArrays, f::SeparableSum, x::TupleOfArrays) = gradient!(grad, f.fs, x)

fun_name(f::SeparableSum) = "separable sum"
fun_dom(f::SeparableSum) = "n/a"
fun_expr(f::SeparableSum) = "(x₁, …, xₖ) ↦ f₁(x₁) + … + fₖ(xₖ)"
fun_params(f::SeparableSum) = "n/a"

function prox_naive(f::SeparableSum, xs::TupleOfArrays{R}, gamma::Union{R, Tuple}=R(1)) where R <: Real
    fys = R(0)
    ys = []
    for k in eachindex(xs)
        y, fy = prox_naive(f.fs[k], xs[k], typeof(gamma) <: Real ? gamma : gamma[k])
        fys += fy
        append!(ys, [y])
    end
    return Tuple(ys), fys
end


"""
    SeisEnvelope(in; <keyword arguments>)

Calculate the envelope attribute of a group of input traces.

# Arguments
- `in`: input data. A 2D Array where the first dimension is time.

# Example
```julia
julia> using PyPlot
julia> dtsec = 0.002; w = Ricker(dt=dtsec);
julia> e = SeisEnvelope(w); t=dtsec*collect(0:1:length(w)-1);plot(t,w,t,e,"-r");xlabel("Time [s]")
```
*Credits: Aaron Stanton,2017*

"""
function SeisEnvelope(d)

	D = fft(d,1)
	D[1:Int(floor(size(d,1)/2)),:] .= 0.0
	return 2*abs.(ifft(D,1))

end


function composer(fns::Vector{GradientHook})
    return if length( fns ) == 0
        identity
    else
        reduce( ∘ , reverse( fns ) )
    end
end
composer(fns::Nothing) = identity

struct IdentityHook <: GradientHook; end
(a::IdentityHook)( x ) = x
@adjoint (a::IdentityHook)( x ) = x, Δ -> Δ
Base.show(io::IO, l::IdentityHook) = print(io, "Identity Gradient Transform")

struct L1Hook{F} <: GradientHook
    λ::F
end
ΔL1(Δ, θ, λ) = ( Δ .+ ( λ .* sign.( Δ ) ) ) .* ( abs.( θ ) .> λ )
( a::L1Hook )( x ) = x
@adjoint ( a::L1Hook )( θ ) = θ, Δ -> ( nothing, ΔL1( Δ, θ, a.λ ) )
Base.show(io::IO, l::L1Hook) = print(io, "L1 Penalty (λ = $(l.λ))")

struct L2Hook{F} <: GradientHook
    λ::F
end
ΔL2(Δ, θ, λ) = Δ .+ ( λ .* sum(abs2, θ ) )
( a::L2Hook )( x ) = x
@adjoint ( a::L2Hook )( θ ) = θ, Δ -> ( nothing, ΔL2( Δ, θ, a.λ ) )
Base.show(io::IO, l::L2Hook) = print(io, "L2 Penalty (λ = $(l.λ))")

struct LipschitzHook{F} <: GradientHook
    λ::F
end
LipschitzRegularization(Δ, θ, λ) = Δ .+ ( λ .* sum(abs2, θ ) )
( a::LipschitzHook )( x ) = x
@adjoint ( a::LipschitzHook )( θ ) = θ, Δ -> ( nothing, LipschitzRegularization( Δ, θ, a.λ ) )
Base.show(io::IO, l::LipschitzHook) = print(io, "Lipschitz Penalty (λ = $(l.λ))")

struct ReverseGradientHook <: GradientHook; end
( a::ReverseGradientHook )( x ) = x
@adjoint ( a::ReverseGradientHook )( θ ) = θ, Δ -> (nothing, -Δ)
Base.show(io::IO, l::ReverseGradientHook) = print(io, "Reverse Gradient")

struct ClampHook{F} <: GradientHook
    ϵ::F
end
( a::ClampHook )( x ) = x
@adjoint ( a::ClampHook )( θ ) = θ, Δ -> ( nothing, clamp.( Δ, -a.ϵ, a.ϵ ) )
Base.show(io::IO, a::ClampHook) = print(io, "Gradient Clamp (Δ ∈ [-$(a.ϵ),$(a.ϵ)])")


abstract type AbstractCN0Estimator end

"""
$(SIGNATURES)

MomentsCN0Estimator to estimate the CN0
"""
struct MomentsCN0Estimator{N} <: AbstractCN0Estimator
    prompt_buffer::SVector{N, ComplexF64}
    current_index::Int
    length::Int
end

function MomentsCN0Estimator(N)
    MomentsCN0Estimator{N}(zero(SVector{N, ComplexF64}), 0, 0)
end

length(cn0_state::MomentsCN0Estimator) = cn0_state.length
get_prompt_buffer(cn0_state::MomentsCN0Estimator) = cn0_state.prompt_buffer
get_current_index(cn0_state::MomentsCN0Estimator) = cn0_state.current_index

"""
$(SIGNATURES)

Updates the `cn0_estimator` to include the information of the current prompt value.
"""
function update(cn0_estimator::MomentsCN0Estimator{N}, prompt) where N
    next_index = mod(get_current_index(cn0_estimator), N) + 1
    next_prompt_buffer = setindex(get_prompt_buffer(cn0_estimator), prompt, next_index)
    next_length = min(length(cn0_estimator) + 1, N)
    MomentsCN0Estimator{N}(next_prompt_buffer, next_index, next_length)
end

"""
$(SIGNATURES)

Estimates the CN0 based on the struct `cn0_estimator`.
"""
function estimate_cn0(cn0_estimator::MomentsCN0Estimator, integration_time)
    length(cn0_estimator) == 0 && return 0.0dBHz
    abs2_prompt_buffer = abs2.(get_prompt_buffer(cn0_estimator))
    M₂ = 1 / length(cn0_estimator) * sum(abs2_prompt_buffer)
    M₄ = 1 / length(cn0_estimator) * sum(abs2_prompt_buffer .^ 2)
    Pd = sqrt(abs(2 * M₂^2 - M₄))
    noise_power = (M₂ - Pd)
    noise_power_non_neg = noise_power - 2 * (noise_power < 0) * noise_power
    SNR = Pd / noise_power_non_neg
    dBHz(SNR / integration_time)
end


# ---
# jupyter:
#   jupytext:
#     formats: ipynb,jl:hydrogen
#     text_representation:
#       extension: .jl
#       format_name: hydrogen
#       format_version: '1.3'
#       jupytext_version: 1.11.2
#   kernelspec:
#     display_name: Julia 1.6.2
#     language: julia
#     name: julia-1.6
# ---

# %%
using StatsBase

sample([2, 3, 5], ProbabilityWeights([0.2, 0.3, 0.5]))

# %%
using StatsBase

a = sample([2, 3, 5], ProbabilityWeights([0.2, 0.3, 0.5]), 10^6)
[(x, count(==(x), a)/length(a)) for x in sort(unique(a))]

# %%
?sample

# %%
using Distributions

rand(DiscreteNonParametric([2, 3, 5], [0.2, 0.3, 0.5]))

# %%
using Distributions

a = rand(DiscreteNonParametric([2, 3, 5], [0.2, 0.3, 0.5]), 10^6)
[(x, count(==(x), a)/length(a)) for x in sort(unique(a))]

# %%
?DiscreteNonParametric

# %%


export Block

type Block{T}
    length::Integer
end

Block(s::Symbol, n) = Block{s}(n)

@doc """bootstrap_index! produces an array of integers that produces a draw from
the bootstrap distribution when used to index the original data. The first
argument is a vector of length n or an n by m matrix that will store the index.
The integer n is the number of observations and m is the number of bootstrap
replications. The second argument is a `Block` object that specifies the
blocking method and the block length.

Only the circular and moving blocks bootstrap are supported, which can be
invoked by using `Block(:circular, n)` and `Block(:moving, n)` respectively.

Examples
--------
i = Array(Int, 90) ## initialization

bootstrap_index!(i, Block(:circular, 10))
bootstrap_index!(i, Block(:moving, 9)) ## Overwrites previous line's results
""" ->
function bootstrap_index!{T<:Integer}(index::AbstractVector{T}, block::Block{:circular})
    n = length(index)
    nblocks = cld(n, block.length)
    for i in 1:nblocks
        blockstart_i = rand(1:n)
        for t in ((i-1) * block.length + 1):min(i*block.length, n)
            ## We're being sort of sloppy with the offsets here. The
            ## key issue is that there is a single startpoint for each
            ## block, and that starting point is uniformly distributed
            ## on 1:n. Even though t increases as i increases,
            ## randomizing blockstart_i ensures this uniform
            ## distribution.
            index[t] = mod(blockstart_i + t, n) + 1
        end
    end
end

function bootstrap_index!{T<:Integer}(index::AbstractVector{T}, block::Block{:moving})
    n = length(index)
    nboot = 0
    while nboot < n
        blockstart_i = rand(0:n-1)
        ## Shorten the block if it is larger than the remaining spots
        ## in index or if it runs over the end of the dataset.
        blocklength_i = min(n - blockstart_i, n - nboot, block.length)
        index[nboot + (1:blocklength_i)] = blockstart_i + (1:blocklength_i)
        nboot += blocklength_i
    end
end

function bootstrap_index!{T<:Integer}(index::AbstractMatrix{T}, block::Block)
    for i in 1:size(index, 2)
        bootstrap_index!(sub(index,:,i), block)
    end
end


# Broyden-Fletcher-Goldfarb-Shanno.
function BFGS!(B::Matrix, y::Vector, s::Vector)
    n, m = size(B)
    Bs = B * s
    B[1:n, 1:m] = B - (Bs * Bs') / dot(s, Bs) + (y * y') / dot(s, y)
end

# Inverse Broyden-Fletcher-Goldfarb-Shanno.
function inv_BFGS(B::Matrix, y::Vector, s::Vector)
    ys = dot(y, s)
    By = B * y
    return B + 1.0 / ys * ((ys + dot(y, By)) / ys * (s * s') - By * s' - s * By')
end

# Systematic rank one.
function SR1!(B::Matrix, y::Vector, s::Vector)
    n, m = size(B)
    yBs = y - B * s
    B[1:n, 1:m] = B + (yBs * yBs') / (yBs' * s)
end