### A Pluto.jl notebook ###
# v0.15.1

using Markdown
using InteractiveUtils

# ╔═╡ acbbdcab-b678-4143-95f2-d2be5f83e925
begin
	using CairoMakie
	using DataFrameMacros
	using DataFrames
	using MixedModels
	using MixedModelsMakie
	using PlutoUI
	CairoMakie.activate!(type="svg")
	TableOfContents()
end

# ╔═╡ fbd6339a-10d4-11ec-3c2e-15dff05b3589
md"""
# Generalized linear mixed models

- A GLMM (Generalized Linear Mixed Model) is used instead of a LMM (Linear Mixed Model) when the response is binary or, perhaps, a count with a low expected count.

- The specification of the model includes the distribution family for the response and, possibly, the *link function*, g, relating the *mean response*, μ, to the value of the *linear predictor*, η.

- To explain the model it helps to consider the linear mixed model in some detail first.

## Matrix notation for the sleepstudy model
"""

# ╔═╡ c72485ee-c4a7-427c-b590-5d9682fb09b3
sleepstudy = DataFrame(MixedModels.dataset(:sleepstudy))

# ╔═╡ 9dea9eac-ba97-49b1-9003-09142d186b43
m1 = fit(
	MixedModel,
	@formula(reaction ~ 1+days+(1+days|subj)),
	sleepstudy,
)

# ╔═╡ ec752d1e-e14a-431a-82b8-32a6e1bd7b5f
md"""
The response vector, y, has 180 elements.  The fixed-effects coefficient vector, β, has 2 elements and the fixed-effects model matrix, X, is of size 180 × 2.
"""

# ╔═╡ 73e26ca3-51a4-4873-b561-f0b1e61879f1
m1.y

# ╔═╡ d366844b-7e04-49b8-8d49-0ef82274a7e5
m1.β

# ╔═╡ 0bbed5cf-1417-42ca-80f6-306200c50077
m1.X

# ╔═╡ 637d518b-dce3-40d5-87c8-bdc655570f8c
md"""
The second column of X is just the `days` vector and the first column is all 1's.

There are 36 random effects, 2 for each of the 18 levels of `subj`.
The "estimates" (technically, the conditional means or conditional modes) are returned as a vector of matrices, one matrix for each grouping factor.
In this case there is only one grouping factor for the random effects so there is one one matrix which contains 18 intercept random effects and 18 slope random effects.
"""

# ╔═╡ 2c1cfa14-99f9-4916-bdc4-8c8af3b34bc3
m1.b

# ╔═╡ 5fcb27d2-771c-417c-aa58-771cdd91a8ff
md"""
There is a model matrix, Z, for the random effects.
In general it has one chunk of columns for the first grouping factor, a chunk of columns for the second grouping factor, etc.

In this case there is only one grouping factor.
"""

# ╔═╡ b8359d5d-0a2b-4bf4-a8d9-cb4ffe0d5b09
first(m1.reterms)

# ╔═╡ 93f478c1-f3fd-4163-954d-2d52a7e7efce
md"""
The defining property of a linear model or linear mixed model is that the fitted values are linear combinations of the fixed-effects parameters and the random effects. We can write the fitted values as
"""

# ╔═╡ d8453ffc-cabf-44b5-9066-65f2502c12ad
m1.X * m1.β + only(m1.reterms) * vec(only(m1.b))

# ╔═╡ 3a879484-b1ff-4ae8-8db4-6e17e5e073c8
fitted(m1)   # just to check that these are indeed the same as calculated above

# ╔═╡ 21b519d9-2140-4af7-83c7-b851cdc18c18
md"""
In symbols we would write the *linear predictor expression* as
```math
\eta = {\bf X\beta + Z b}
```
where $\eta$ has 180 elements, $\beta$ has 2 elements, $\bf b$ has 36 elements, $\bf X$ is of size 180 × 2 and $\bf Z$ is of size 180 × 36.

For a linear model or linear mixed model the linear predictor **is** the mean response, $\mu$.
That is, we can write the probability model in terms of a 180-dimensional random variable, $\mathcal Y$, for the response and a 36-dimensional random variable, $\mathcal B$, for the random effects as
```math
\begin{aligned}
  (\mathcal{Y} | \mathcal{B}=\bf{b}) &\sim\mathcal{N}(\bf{ X\beta + Z b},\sigma^2\bf{I})\\\\
  \mathcal{B}&\sim\mathcal{N}(\bf{0},\Sigma_\theta) .
\end{aligned}
```
where $\Sigma_\theta$ is a 36 × 36 symmetric covariance matrix that has a special form - it consists of 18 diagonal blocks, each of size 2 × 2 and all the same.

Recall that this symmetric matrix can be constructed from the parameters θ, which generate the lower triangular matrix λ, and the estimate σ̂².
"""

# ╔═╡ 765baad6-b96f-4d13-b46c-bd848d8845c4
m1.θ

# ╔═╡ 7a2a3c47-44e1-4061-a186-7e3a9b0269be
λ = only(m1.λ)  # with multiple grouping factors there will be multiple λ's

# ╔═╡ cd024e2b-592a-4801-bfa0-1a58e3dded1e
Σ = varest(m1) * (λ * λ')

# ╔═╡ bce7177c-e4d9-4acf-9ac1-2ff8f244c038
md"Compare the diagonal elements to the *Variance* column of"

# ╔═╡ 8d2ae114-5780-4d38-ba40-95aac032337b
VarCorr(m1)

# ╔═╡ d90a8d29-afc8-470e-b5da-f35a4cadf1c5
md"""
## Linear predictors in LMMs and GLMMs

Writing the model for $\mathcal Y$ as
```math
(\mathcal{Y} | \mathcal{B}=\bf{b})\sim\mathcal{N}(\bf{ X\beta + Z b},\sigma^2\bf{I})
```
may seem like over-mathematization (or "overkill", if you prefer) relative to expressions like
```math
y_i = \beta_1 x_{i,1} + \beta_2 x_{i,2}+ b_1 z_{i,1} +\dots+b_{36} z_{i,36}+\epsilon_i
```
but this more abstract form is necessary for generalizations.

The way that I read the first form is
> The conditional distribution of the response vector, $\mathcal Y$, given that the random effects vector, $\mathcal B =\bf b$, is a multivariate normal (or *Gaussian*) distribution whose mean, $\mu$, is the linear predictor, $\eta=\bf X\beta+Zb$, and whose covariance matrix is $\sigma^2\bf I$.  That is, conditional on $\bf b$, the elements of $\mathcal Y$ are independent normal random variables with constant variance, $\sigma^2$, and means of the form $\mu = \eta = \bf X\beta+Zb$.

So the only things that differ in the distributions of the $y_i$'s are the means and they are determined by this linear predictor, $\eta = \bf X\beta+Zb$.
## Generalized Linear Mixed Models
Consider first a GLMM for a vector, $\bf y$, of binary (i.e. yes/no) responses.
The probability model for the conditional distribution $\mathcal Y|\mathcal B=\bf b$ consists of independent [Bernoulli distributions](https://en.wikipedia.org/wiki/Bernoulli_distribution) where the mean, $\mu_i$, for the i'th response is again determined by the i'th element of a *linear predictor*, $\eta = \bf X\beta+Zb$.

However, in this case we will run into trouble if we try to make $\mu=\eta$ because $\mu_i$ is the probability of "success" for the i'th response and must be between 0 and 1.  We can't guarantee that the i'th component of $\eta$ will be between 0 and 1.  To get around this problem we apply a transformation to take $\eta_i$ to $\mu_i$.
For historical reasons this transformation is called the *inverse link*, written $g^{-1}$, and the opposite transformation - from the probability scale to an unbounded scale - is called the *link*, g.

Each probability distribution in the [exponential family](https://en.wikipedia.org/wiki/Exponential_family) (which is most of the important ones), has a *canonical link* which comes from the form of the distribution itself.  The details aren't as important as recognizing that the distribution itself determines a preferred link function.

For the Bernoulli distribution, the canonical link is the *logit* or *log-odds* function,
```math
\eta = g(\mu) = \log\left(\frac{\mu}{1-\mu}\right),
```
(it's called *log-odds* because it is the logarithm of the odds ratio, $p/(1-p)$)
and the canonical inverse link is the *logistic*
```math
\mu=g^{-1}(\eta)=\frac{1}{1+\exp(-\eta)}.
```
This is why fitting a binary response is sometimes called *logistic regression*.

For later use we define a Julia `logistic` function.
See [this presentation](https://github.com/dmbates/JSM2021/blob/main/notebooks/2compilation.jl) for more information than you could possible want to know on how Julia converts code like this to run on the processor.
"""

# ╔═╡ 7d836c30-f099-4bae-921c-f5a9cf57c5cc
logistic(η) = inv(one(η) + exp(-η))

# ╔═╡ 7390872e-9c9a-4d77-9f59-bdc7d9e29ab5
md"""
To reiterate, the probability model for a Generalized Linear Mixed Model (GLMM) is
```math
\begin{aligned}
  (\mathcal{Y} | \mathcal{B}=\bf{b}) &\sim\mathcal{D}(\bf{g^{-1}(X\beta + Z b)},\phi)\\\\
  \mathcal{B}&\sim\mathcal{N}(\bf{0},\Sigma_\theta) .
\end{aligned}
```
where $\mathcal{D}$ is the distribution family (such as Bernoulli or Poisson), $g^{-1}$ is the inverse link and $\phi$ is a scale parameter for $\mathcal{D}$ if it has one.
The important cases of the Bernoulli and Poisson distributions don't have a scale parameter - once you know the mean you know everything you need to know about the distribution. (For those following the presentation, [this poem](https://www.poetryfoundation.org/poems/44477/ode-on-a-grecian-urn) by John Keats is the one with the couplet "Beauty is truth, truth beauty - that is all ye know on earth and all ye need to know.")

### An example of a Bernoulli GLMM

The `contra` dataset in the `MixedModels` package is from a survey on the use of artificial contraception by women in Bangladesh.
"""

# ╔═╡ a6553206-8116-49a5-8994-0fc7f3c6e2cd
contra = DataFrame(MixedModels.dataset(:contra))

# ╔═╡ cae81515-4a9c-44f9-a24e-3d5f08c7933c
combine(groupby(contra, :dist), nrow)

# ╔═╡ 57bc6bbf-e4e4-4abc-960e-3e89e9312d66
md"""
The information recorded included woman's age, the number of live children she has, whether she lives in an urban or rural setting, and the political district in which she lives.

The age was centered.
Unfortunately, the version of the data to which I had access did not record what the centering value was.

A data plot, drawn using `lattice` graphics in `R`, shows that the probability of contraception use is **not** linear in `age` - it is low for younger women, higher for women in the middle of the range (assumed to be women in late 20's to early 30's) and low again for older women (late 30's to early 40's in this survey).

If we fit a model with only the `age` term in the fixed effects, that term will not be significant.
This doesn't mean that there is no "age effect", it only means that there is no significant linear effect for `age`.
"""

# ╔═╡ b7a9dadc-b95e-4abc-89d1-e9dff22e0a40
Resource("https://github.com/RePsychLing/SMLP2021/blob/main/notebooks/contra.svg?raw=true")

# ╔═╡ f850d266-934c-400b-9f20-a848f08513de
gm1 = fit(MixedModel,
	@formula(use ~ 1+age+abs2(age)+urban+livch+(1|dist)),
	contra,
	Bernoulli();   # will use the canonical link
	nAGQ=9,        # use a more compute-intensive evaluation of the deviance
	contrasts=Dict(:dist => Grouping(), :urban => HelmertCoding()),
)

# ╔═╡ 19f94d31-9321-440b-87e4-f2b488a5c6ca
md"""
Notice that the linear term for `age` is not significant but the quadratic term for `age` is highly significant.
We usually retain the lower order term, even if it is not significant, if the higher order term is significant.

Notice also that the parameter estimates for the treatment contrasts for `livch` are similar.
Thus the distinction of 1, 2, or 3+ childen is not as important as the contrast between having any children and not having any.
Those women who already have children are more likely to use artificial contraception.

Furthermore, the women without children have a different probability vs age profile than the women with children.
To allow for this we define a binary `children` factor and incorporate an `age&children` interaction.
"""

# ╔═╡ f0d0fc9c-f817-4f3c-99de-c2442126509b
VarCorr(gm1)

# ╔═╡ f02f0407-8e38-4cef-a509-18abd0594b3a
md"""
Notice that there is no "residual" variance being estimated.
This is because the Bernoulli distribution doesn't have a scale parameter.

### Reducing the levels of `livch` to `no children` versus `children`.
"""

# ╔═╡ d68d3d90-3584-4d70-9d83-d203ba28049c
@transform!(contra, :children = :livch ≠ "0")

# ╔═╡ 63610f52-5067-4754-a86b-675711c4f36a
gm2 = fit(
	MixedModel,
	@formula(use ~ 1 + age*children + abs2(age) + children + urban + (1|dist)),
	contra,
	Bernoulli();
	nAGQ=9,
	contrasts=Dict(
		:dist => Grouping(),
		:urban => HelmertCoding(),
		:children => HelmertCoding(),
	),
)

# ╔═╡ 3e8a4b2e-4996-4756-9932-93fbc3efb3e6
let
	mods = [gm2, gm1]
	DataFrame(
		model=[:gm2,:gm1],
		npar=dof.(mods),
		deviance=deviance.(mods),
		AIC=aic.(mods),
		BIC=bic.(mods),
		AICc=aicc.(mods),
	)
end

# ╔═╡ f16be61e-3b9b-40ba-b109-8e46d3af5298
md"""
Because these models are not nested, we cannot do a likelihood ratio test.
Nevertheless we see that the deviance is much lower in the model with `age & children` even though the 3 levels of `livch` have been collapsed into a single level of `children`.
There is a substantial decrease in the deviance even though there are fewer parameters in model `gm2` than in `gm1`.
This decrease is because the flexibility of the model - its ability to model the behavior of the response - is being put to better use in `gm2` than in `gm1`.

At present the calculation of the `geomdof` as `sum(influence(m))` is not correctly defined in our code for a GLMM so we need to do some more work before we can examine those values.
"""

# ╔═╡ f83aaa94-4620-41f2-8f61-893593aecb61
md"""
### Using `urban&dist` as a grouping factor

It turns out that there can be more difference between urban and rural settings within the same political district than there is between districts.
To model this difference we build a model with `urban&dist` as a grouping factor.
"""

# ╔═╡ 3b582cbb-f7e1-42bf-aa8d-abfa96342a32
gm3 = fit(
	MixedModel,
	@formula(use ~ 1 + age*children + abs2(age) + children + urban + (1|urban&dist)),
	contra,
	Bernoulli();
	nAGQ=9,
	contrasts=Dict(
		:dist => Grouping(),
		:urban => HelmertCoding(),
		:children => HelmertCoding(),
	),
)

# ╔═╡ dc7f19bc-a6fe-4d3c-8e78-1426aa6081e7
begin
	using Effects
	design = Dict(:children => [true, false], :urban => ["Y", "N"], :age => [0.0])
	preds = effects(design, gm3)
end

# ╔═╡ 77e2e52d-6133-4631-9675-d6db1f079ead
let
	mods = [gm3, gm2, gm1]
	DataFrame(
		model=[:gm3, :gm2,:gm1],
		npar=dof.(mods),
		deviance=deviance.(mods),
		AIC=aic.(mods),
		BIC=bic.(mods),
		AICc=aicc.(mods),
	)
end

# ╔═╡ 179ee9eb-020b-4f63-8699-bd1a0368e6e9
md"""
Notice that the parameter count in `gm3` is the same as that of `gm2` - the thing that has changed is the number of levels of the grouping factor- resulting in a much lower deviance for `gm3`.
This reinforces the idea that a simple count of the number of parameters to be estimated does not always reflect the complexity of the model.
"""

# ╔═╡ 73332322-ffc9-4e83-9f7e-94b726405c97
with_terminal() do
	show(gm2)
	println()
	show(gm3)
end

# ╔═╡ aa274ce1-0d31-4701-a967-7d7889b2ac38
md"""
The coefficient for `age` may be regarded as insignificant but we retain it for two reasons: we have a term of `age²` (written `abs2(age)`) in the model and we have a significant interaction `age & children` in the model.
"""

# ╔═╡ 7204cac6-fad2-4ccf-86ff-2d91c426c6c6
md"""
### Predictions for some subgroups
For a "typical" district (random effect near zero) the predictions on the linear predictor scale for a woman whose age is near the centering value (i.e. centered age of zero) are:
"""

# ╔═╡ b74f0e46-1046-4f44-8c0b-235351251e86
md"Converting these η values to probabilities yields"

# ╔═╡ f2fd5625-cab1-4ce2-8592-f6215ea65b86
logistic.(preds.use)

# ╔═╡ 0a3351b2-7e1c-45bb-97a7-05bc7be4826f
md"""
## Summarizing the results

- From the data plot we can see a quadratic trend in the probability by age.
- The patterns for women with children are similar and we do not need to distinguish between 1, 2, and 3+ children.
- We do distinguish between those women who do not have children and those with children.  This shows up in a signficant `age & children` interaction term.

"""

# ╔═╡ 00000000-0000-0000-0000-000000000001
PLUTO_PROJECT_TOML_CONTENTS = """
[deps]
CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
DataFrameMacros = "75880514-38bc-4a95-a458-c2aea5a3a702"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Effects = "8f03c58b-bd97-4933-a826-f71b64d2cca2"
MixedModels = "ff71e718-51f3-5ec2-a782-8ffcbfa3c316"
MixedModelsMakie = "b12ae82c-6730-437f-aff9-d2c38332a376"
PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8"

[compat]
CairoMakie = "~0.6.5"
DataFrameMacros = "~0.1.0"
DataFrames = "~1.2.2"
Effects = "~0.1.2"
MixedModels = "~4.1.1"
MixedModelsMakie = "~0.3.8"
PlutoUI = "~0.7.9"
"""

# ╔═╡ 00000000-0000-0000-0000-000000000002
PLUTO_MANIFEST_TOML_CONTENTS = """
# This file is machine-generated - editing it directly is not advised

[[AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "485ee0867925449198280d4af84bdb46a2a404d0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.0.1"

[[AbstractTrees]]
git-tree-sha1 = "03e0550477d86222521d254b741d470ba17ea0b5"
uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
version = "0.3.4"

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "84918055d15b3114ede17ac6a7182f68870c16f7"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.1"

[[Animations]]
deps = ["Colors"]
git-tree-sha1 = "e81c509d2c8e49592413bfb0bb3b08150056c79d"
uuid = "27a7e980-b3e6-11e9-2bcd-0b925532e340"
version = "0.4.1"

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[ArrayInterface]]
deps = ["IfElse", "LinearAlgebra", "Requires", "SparseArrays", "Static"]
git-tree-sha1 = "019303a0f26d6012f35ecdfa4618551d145fb9f2"
uuid = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9"
version = "3.1.31"

[[Arrow]]
deps = ["ArrowTypes", "BitIntegers", "CodecLz4", "CodecZstd", "DataAPI", "Dates", "Mmap", "PooledArrays", "SentinelArrays", "Tables", "TimeZones", "UUIDs"]
git-tree-sha1 = "b00e6eaba895683867728e73af78a00218f0db10"
uuid = "69666777-d1a9-59fb-9406-91d4454c9d45"
version = "1.6.2"

[[ArrowTypes]]
deps = ["UUIDs"]
git-tree-sha1 = "a0633b6d6efabf3f76dacd6eb1b3ec6c42ab0552"
uuid = "31f734f8-188a-4ce0-8406-c8a06bd891cd"
version = "1.2.1"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[Automa]]
deps = ["Printf", "ScanByte", "TranscodingStreams"]
git-tree-sha1 = "d50976f217489ce799e366d9561d56a98a30d7fe"
uuid = "67c07d97-cdcb-5c2c-af73-a7f9c32a568b"
version = "0.8.2"

[[AxisAlgorithms]]
deps = ["LinearAlgebra", "Random", "SparseArrays", "WoodburyMatrices"]
git-tree-sha1 = "a4d07a1c313392a77042855df46c5f534076fab9"
uuid = "13072b0f-2c55-5437-9ae7-d433b7a33950"
version = "1.0.0"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[BenchmarkTools]]
deps = ["JSON", "Logging", "Printf", "Statistics", "UUIDs"]
git-tree-sha1 = "42ac5e523869a84eac9669eaceed9e4aa0e1587b"
uuid = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
version = "1.1.4"

[[BitIntegers]]
deps = ["Random"]
git-tree-sha1 = "f50b5a99aa6ff9db7bf51255b5c21c8bc871ad54"
uuid = "c3b6d118-76ef-56ca-8cc7-ebb389d030a1"
version = "0.2.5"

[[Bzip2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "19a35467a82e236ff51bc17a3a44b69ef35185a2"
uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0"
version = "1.0.8+0"

[[CEnum]]
git-tree-sha1 = "215a9aa4a1f23fbd05b92769fdd62559488d70e9"
uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82"
version = "0.4.1"

[[Cairo]]
deps = ["Cairo_jll", "Colors", "Glib_jll", "Graphics", "Libdl", "Pango_jll"]
git-tree-sha1 = "d0b3f8b4ad16cb0a2988c6788646a5e6a17b6b1b"
uuid = "159f3aea-2a34-519c-b102-8c37f9878175"
version = "1.0.5"

[[CairoMakie]]
deps = ["Base64", "Cairo", "Colors", "FFTW", "FileIO", "FreeType", "GeometryBasics", "LinearAlgebra", "Makie", "SHA", "StaticArrays"]
git-tree-sha1 = "8664989955daccc90002629aa80193e44893bb45"
uuid = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
version = "0.6.5"

[[Cairo_jll]]
deps = ["Artifacts", "Bzip2_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"]
git-tree-sha1 = "f2202b55d816427cd385a9a4f3ffb226bee80f99"
uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a"
version = "1.16.1+0"

[[ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "30ee06de5ff870b45c78f529a6b093b3323256a3"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.3.1"

[[CodecBzip2]]
deps = ["Bzip2_jll", "Libdl", "TranscodingStreams"]
git-tree-sha1 = "2e62a725210ce3c3c2e1a3080190e7ca491f18d7"
uuid = "523fee87-0ab8-5b00-afb7-3ecf72e48cfd"
version = "0.7.2"

[[CodecLz4]]
deps = ["Lz4_jll", "TranscodingStreams"]
git-tree-sha1 = "59fe0cb37784288d6b9f1baebddbf75457395d40"
uuid = "5ba52731-8f18-5e0d-9241-30f10d1ec561"
version = "0.4.0"

[[CodecZlib]]
deps = ["TranscodingStreams", "Zlib_jll"]
git-tree-sha1 = "ded953804d019afa9a3f98981d99b33e3db7b6da"
uuid = "944b1d66-785c-5afd-91f1-9de20f533193"
version = "0.7.0"

[[CodecZstd]]
deps = ["TranscodingStreams", "Zstd_jll"]
git-tree-sha1 = "d19cd9ae79ef31774151637492291d75194fc5fa"
uuid = "6b39b394-51ab-5f42-8807-6242bab2b4c2"
version = "0.7.0"

[[ColorBrewer]]
deps = ["Colors", "JSON", "Test"]
git-tree-sha1 = "61c5334f33d91e570e1d0c3eb5465835242582c4"
uuid = "a2cac450-b92f-5266-8821-25eda20663c8"
version = "0.4.0"

[[ColorSchemes]]
deps = ["ColorTypes", "Colors", "FixedPointNumbers", "Random"]
git-tree-sha1 = "9995eb3977fbf67b86d0a0a0508e83017ded03f2"
uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4"
version = "3.14.0"

[[ColorTypes]]
deps = ["FixedPointNumbers", "Random"]
git-tree-sha1 = "32a2b8af383f11cbb65803883837a149d10dfe8a"
uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
version = "0.10.12"

[[ColorVectorSpace]]
deps = ["ColorTypes", "Colors", "FixedPointNumbers", "LinearAlgebra", "SpecialFunctions", "Statistics", "StatsBase"]
git-tree-sha1 = "4d17724e99f357bfd32afa0a9e2dda2af31a9aea"
uuid = "c3611d14-8923-5661-9e6a-0046d554d3a4"
version = "0.8.7"

[[Colors]]
deps = ["ColorTypes", "FixedPointNumbers", "Reexport"]
git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40"
uuid = "5ae59095-9a9b-59fe-a467-6f913c188581"
version = "0.12.8"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "6071cb87be6a444ac75fdbf51b8e7273808ce62f"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.35.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[Contour]]
deps = ["StaticArrays"]
git-tree-sha1 = "9f02045d934dc030edad45944ea80dbd1f0ebea7"
uuid = "d38c429a-6771-53c6-b99e-75d170b6e991"
version = "0.5.7"

[[Crayons]]
git-tree-sha1 = "3f71217b538d7aaee0b69ab47d9b7724ca8afa0d"
uuid = "a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f"
version = "4.0.4"

[[DataAPI]]
git-tree-sha1 = "bec2532f8adb82005476c141ec23e921fc20971b"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.8.0"

[[DataFrameMacros]]
deps = ["DataFrames"]
git-tree-sha1 = "508d57ef7b78551cf69c2837d80af5017ce57217"
uuid = "75880514-38bc-4a95-a458-c2aea5a3a702"
version = "0.1.0"

[[DataFrames]]
deps = ["Compat", "DataAPI", "Future", "InvertedIndices", "IteratorInterfaceExtensions", "LinearAlgebra", "Markdown", "Missings", "PooledArrays", "PrettyTables", "Printf", "REPL", "Reexport", "SortingAlgorithms", "Statistics", "TableTraits", "Tables", "Unicode"]
git-tree-sha1 = "d785f42445b63fc86caa08bb9a9351008be9b765"
uuid = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
version = "1.2.2"

[[DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "7d9d316f04214f7efdbb6398d545446e246eff02"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.10"

[[DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[Distributions]]
deps = ["ChainRulesCore", "FillArrays", "LinearAlgebra", "PDMats", "Printf", "QuadGK", "Random", "SparseArrays", "SpecialFunctions", "Statistics", "StatsBase", "StatsFuns"]
git-tree-sha1 = "f4efaa4b5157e0cdb8283ae0b5428bc9208436ed"
uuid = "31c24e10-a181-5473-b8eb-7969acd0382f"
version = "0.25.16"

[[DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "a32185f5428d3986f47c2ab78b1f216d5e6cc96f"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.5"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[EarCut_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "3f3a2501fa7236e9b911e0f7a588c657e822bb6d"
uuid = "5ae413db-bbd1-5e63-b57d-d24a61df00f5"
version = "2.2.3+0"

[[Effects]]
deps = ["DataFrames", "LinearAlgebra", "Statistics", "StatsBase", "StatsModels", "Tables"]
git-tree-sha1 = "989a9cb80952d2d3d36bccbe7c0001fd9ef5dfa2"
uuid = "8f03c58b-bd97-4933-a826-f71b64d2cca2"
version = "0.1.2"

[[EllipsisNotation]]
deps = ["ArrayInterface"]
git-tree-sha1 = "8041575f021cba5a099a456b4163c9a08b566a02"
uuid = "da5c29d0-fa7d-589e-88eb-ea29b0a81949"
version = "1.1.0"

[[Expat_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b3bfd02e98aedfa5cf885665493c5598c350cd2f"
uuid = "2e619515-83b5-522b-bb60-26c02a35a201"
version = "2.2.10+0"

[[ExprTools]]
git-tree-sha1 = "b7e3d17636b348f005f11040025ae8c6f645fe92"
uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04"
version = "0.1.6"

[[FFMPEG]]
deps = ["FFMPEG_jll"]
git-tree-sha1 = "b57e3acbe22f8484b4b5ff66a7499717fe1a9cc8"
uuid = "c87230d0-a227-11e9-1b43-d7ebe4e7570a"
version = "0.4.1"

[[FFMPEG_jll]]
deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "Pkg", "Zlib_jll", "libass_jll", "libfdk_aac_jll", "libvorbis_jll", "x264_jll", "x265_jll"]
git-tree-sha1 = "d8a578692e3077ac998b50c0217dfd67f21d1e5f"
uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5"
version = "4.4.0+0"

[[FFTW]]
deps = ["AbstractFFTs", "FFTW_jll", "LinearAlgebra", "MKL_jll", "Preferences", "Reexport"]
git-tree-sha1 = "f985af3b9f4e278b1d24434cbb546d6092fca661"
uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341"
version = "1.4.3"

[[FFTW_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "3676abafff7e4ff07bbd2c42b3d8201f31653dcc"
uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a"
version = "3.3.9+8"

[[FileIO]]
deps = ["Pkg", "Requires", "UUIDs"]
git-tree-sha1 = "3c041d2ac0a52a12a27af2782b34900d9c3ee68c"
uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549"
version = "1.11.1"

[[FillArrays]]
deps = ["LinearAlgebra", "Random", "SparseArrays", "Statistics"]
git-tree-sha1 = "a3b7b041753094f3b17ffa9d2e2e07d8cace09cd"
uuid = "1a297f60-69ca-5386-bcde-b61e274b549b"
version = "0.12.3"

[[FixedPointNumbers]]
deps = ["Statistics"]
git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc"
uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
version = "0.8.4"

[[Fontconfig_jll]]
deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "21efd19106a55620a188615da6d3d06cd7f6ee03"
uuid = "a3f928ae-7b40-5064-980b-68af3947d34b"
version = "2.13.93+0"

[[Formatting]]
deps = ["Printf"]
git-tree-sha1 = "8339d61043228fdd3eb658d86c926cb282ae72a8"
uuid = "59287772-0a20-5a39-b81b-1366585eb4c0"
version = "0.4.2"

[[FreeType]]
deps = ["CEnum", "FreeType2_jll"]
git-tree-sha1 = "cabd77ab6a6fdff49bfd24af2ebe76e6e018a2b4"
uuid = "b38be410-82b0-50bf-ab77-7b57e271db43"
version = "4.0.0"

[[FreeType2_jll]]
deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "87eb71354d8ec1a96d4a7636bd57a7347dde3ef9"
uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7"
version = "2.10.4+0"

[[FreeTypeAbstraction]]
deps = ["ColorVectorSpace", "Colors", "FreeType", "GeometryBasics", "StaticArrays"]
git-tree-sha1 = "19d0f1e234c13bbfd75258e55c52aa1d876115f5"
uuid = "663a7486-cb36-511b-a19d-713bb74d65c9"
version = "0.9.2"

[[FriBidi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "aa31987c2ba8704e23c6c8ba8a4f769d5d7e4f91"
uuid = "559328eb-81f9-559d-9380-de523a88c83c"
version = "1.0.10+0"

[[Future]]
deps = ["Random"]
uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820"

[[GLM]]
deps = ["Distributions", "LinearAlgebra", "Printf", "Reexport", "SparseArrays", "SpecialFunctions", "Statistics", "StatsBase", "StatsFuns", "StatsModels"]
git-tree-sha1 = "f564ce4af5e79bb88ff1f4488e64363487674278"
uuid = "38e38edf-8417-5370-95a0-9cbb8c7f171a"
version = "1.5.1"

[[GeometryBasics]]
deps = ["EarCut_jll", "IterTools", "LinearAlgebra", "StaticArrays", "StructArrays", "Tables"]
git-tree-sha1 = "58bcdf5ebc057b085e58d95c138725628dd7453c"
uuid = "5c1252a2-5f33-56bf-86c9-59e7332b4326"
version = "0.4.1"

[[Gettext_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"]
git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046"
uuid = "78b55507-aeef-58d4-861c-77aaff3498b1"
version = "0.21.0+0"

[[Glib_jll]]
deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "7bf67e9a481712b3dbe9cb3dac852dc4b1162e02"
uuid = "7746bdde-850d-59dc-9ae8-88ece973131d"
version = "2.68.3+0"

[[Graphics]]
deps = ["Colors", "LinearAlgebra", "NaNMath"]
git-tree-sha1 = "2c1cf4df419938ece72de17f368a021ee162762e"
uuid = "a2bd30eb-e257-5431-a919-1863eab51364"
version = "1.1.0"

[[Graphite2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "344bf40dcab1073aca04aa0df4fb092f920e4011"
uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472"
version = "1.3.14+0"

[[GridLayoutBase]]
deps = ["GeometryBasics", "InteractiveUtils", "Match", "Observables"]
git-tree-sha1 = "e2f606c87d09d5187bb6069dab8cee0af7c77bdb"
uuid = "3955a311-db13-416c-9275-1d80ed98e5e9"
version = "0.6.1"

[[Grisu]]
git-tree-sha1 = "53bb909d1151e57e2484c3d1b53e19552b887fb2"
uuid = "42e2da0e-8278-4e71-bc24-59509adca0fe"
version = "1.0.2"

[[HarfBuzz_jll]]
deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg"]
git-tree-sha1 = "8a954fed8ac097d5be04921d595f741115c1b2ad"
uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566"
version = "2.8.1+0"

[[IfElse]]
git-tree-sha1 = "28e837ff3e7a6c3cdb252ce49fb412c8eb3caeef"
uuid = "615f187c-cbe4-4ef1-ba3b-2fcf58d6d173"
version = "0.1.0"

[[ImageCore]]
deps = ["AbstractFFTs", "Colors", "FixedPointNumbers", "Graphics", "MappedArrays", "MosaicViews", "OffsetArrays", "PaddedViews", "Reexport"]
git-tree-sha1 = "db645f20b59f060d8cfae696bc9538d13fd86416"
uuid = "a09fc81d-aa75-5fe9-8630-4744c3626534"
version = "0.8.22"

[[ImageIO]]
deps = ["FileIO", "Netpbm", "PNGFiles"]
git-tree-sha1 = "0d6d09c28d67611c68e25af0c2df7269c82b73c7"
uuid = "82e4d734-157c-48bb-816b-45c225c6df19"
version = "0.4.1"

[[IndirectArrays]]
git-tree-sha1 = "012e604e1c7458645cb8b436f8fba789a51b257f"
uuid = "9b13fd28-a010-5f03-acff-a1bbcff69959"
version = "1.0.0"

[[IntelOpenMP_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d979e54b71da82f3a65b62553da4fc3d18c9004c"
uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0"
version = "2018.0.3+2"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[Interpolations]]
deps = ["AxisAlgorithms", "ChainRulesCore", "LinearAlgebra", "OffsetArrays", "Random", "Ratios", "Requires", "SharedArrays", "SparseArrays", "StaticArrays", "WoodburyMatrices"]
git-tree-sha1 = "61aa005707ea2cebf47c8d780da8dc9bc4e0c512"
uuid = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59"
version = "0.13.4"

[[IntervalSets]]
deps = ["Dates", "EllipsisNotation", "Statistics"]
git-tree-sha1 = "3cc368af3f110a767ac786560045dceddfc16758"
uuid = "8197267c-284f-5f27-9208-e0e47529a953"
version = "0.5.3"

[[InvertedIndices]]
git-tree-sha1 = "bee5f1ef5bf65df56bdd2e40447590b272a5471f"
uuid = "41ab1584-1d38-5bbf-9106-f11c6c58b48f"
version = "1.1.0"

[[IrrationalConstants]]
git-tree-sha1 = "f76424439413893a832026ca355fe273e93bce94"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.1.0"

[[Isoband]]
deps = ["isoband_jll"]
git-tree-sha1 = "f9b6d97355599074dc867318950adaa6f9946137"
uuid = "f1662d9f-8043-43de-a69a-05efc1cc6ff4"
version = "0.1.1"

[[IterTools]]
git-tree-sha1 = "05110a2ab1fc5f932622ffea2a003221f4782c18"
uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e"
version = "1.3.0"

[[IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "8076680b162ada2a031f707ac7b4953e30667a37"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.2"

[[JSON3]]
deps = ["Dates", "Mmap", "Parsers", "StructTypes", "UUIDs"]
git-tree-sha1 = "b3e5984da3c6c95bcf6931760387ff2e64f508f3"
uuid = "0f8b85d8-7281-11e9-16c2-39a750bddbf1"
version = "1.9.1"

[[KernelDensity]]
deps = ["Distributions", "DocStringExtensions", "FFTW", "Interpolations", "StatsBase"]
git-tree-sha1 = "591e8dc09ad18386189610acafb970032c519707"
uuid = "5ab0869b-81aa-558d-bb23-cbf5423bbe9b"
version = "0.6.3"

[[LAME_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "f6250b16881adf048549549fba48b1161acdac8c"
uuid = "c1c5ebd0-6772-5130-a774-d5fcae4a789d"
version = "3.100.1+0"

[[LZO_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "e5b909bcf985c5e2605737d2ce278ed791b89be6"
uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac"
version = "2.10.1+0"

[[LaTeXStrings]]
git-tree-sha1 = "c7f1c695e06c01b95a67f0cd1d34994f3e7db104"
uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
version = "1.2.1"

[[LazyArtifacts]]
deps = ["Artifacts", "Pkg"]
uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[Libffi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "761a393aeccd6aa92ec3515e428c26bf99575b3b"
uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490"
version = "3.2.2+0"

[[Libgcrypt_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll", "Pkg"]
git-tree-sha1 = "64613c82a59c120435c067c2b809fc61cf5166ae"
uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4"
version = "1.8.7+0"

[[Libgpg_error_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "c333716e46366857753e273ce6a69ee0945a6db9"
uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8"
version = "1.42.0+0"

[[Libiconv_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "42b62845d70a619f063a7da093d995ec8e15e778"
uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531"
version = "1.16.1+1"

[[Libmount_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "9c30530bf0effd46e15e0fdcf2b8636e78cbbd73"
uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9"
version = "2.35.0+0"

[[Libuuid_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "7f3efec06033682db852f8b3bc3c1d2b0a0ab066"
uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700"
version = "2.36.0+0"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[LogExpFunctions]]
deps = ["ChainRulesCore", "DocStringExtensions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "86197a8ecb06e222d66797b0c2d2f0cc7b69e42b"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.2"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[Lz4_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "5d494bc6e85c4c9b626ee0cab05daa4085486ab1"
uuid = "5ced341a-0733-55b8-9ab6-a4889d929147"
version = "1.9.3+0"

[[MKL_jll]]
deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"]
git-tree-sha1 = "c253236b0ed414624b083e6b72bfe891fbd2c7af"
uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7"
version = "2021.1.1+1"

[[Makie]]
deps = ["Animations", "Base64", "ColorBrewer", "ColorSchemes", "ColorTypes", "Colors", "Contour", "Distributions", "DocStringExtensions", "FFMPEG", "FileIO", "FixedPointNumbers", "Formatting", "FreeType", "FreeTypeAbstraction", "GeometryBasics", "GridLayoutBase", "ImageIO", "IntervalSets", "Isoband", "KernelDensity", "LaTeXStrings", "LinearAlgebra", "MakieCore", "Markdown", "Match", "MathTeXEngine", "Observables", "Packing", "PlotUtils", "PolygonOps", "Printf", "Random", "RelocatableFolders", "Serialization", "Showoff", "SignedDistanceFields", "SparseArrays", "StaticArrays", "Statistics", "StatsBase", "StatsFuns", "StructArrays", "UnicodeFun"]
git-tree-sha1 = "7e49f989e7c7f50fe55bd92d45329c9cf3f2583d"
uuid = "ee78f7c6-11fb-53f2-987a-cfe4a2b5a57a"
version = "0.15.2"

[[MakieCore]]
deps = ["Observables"]
git-tree-sha1 = "7bcc8323fb37523a6a51ade2234eee27a11114c8"
uuid = "20f20a25-4f0e-4fdf-b5d1-57303727442b"
version = "0.1.3"

[[MappedArrays]]
git-tree-sha1 = "e8b359ef06ec72e8c030463fe02efe5527ee5142"
uuid = "dbb5928d-eab1-5f90-85c2-b9b0edb7c900"
version = "0.4.1"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[Match]]
git-tree-sha1 = "5cf525d97caf86d29307150fcba763a64eaa9cbe"
uuid = "7eb4fadd-790c-5f42-8a69-bfa0b872bfbf"
version = "1.1.0"

[[MathOptInterface]]
deps = ["BenchmarkTools", "CodecBzip2", "CodecZlib", "JSON", "LinearAlgebra", "MutableArithmetics", "OrderedCollections", "Printf", "SparseArrays", "Test", "Unicode"]
git-tree-sha1 = "a1f9933fa00624d8c97301253d14710b80fa08ee"
uuid = "b8f27783-ece8-5eb3-8dc8-9495eed66fee"
version = "0.10.1"

[[MathProgBase]]
deps = ["LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "9abbe463a1e9fc507f12a69e7f29346c2cdc472c"
uuid = "fdba3010-5040-5b88-9595-932c9decdf73"
version = "0.7.8"

[[MathTeXEngine]]
deps = ["AbstractTrees", "Automa", "DataStructures", "FreeTypeAbstraction", "GeometryBasics", "LaTeXStrings", "REPL", "RelocatableFolders", "Test"]
git-tree-sha1 = "f5c8789464aed7058107463e5cef53e6ad3f1f3e"
uuid = "0a4f8689-d25c-4efe-a92b-7142dfc1aa53"
version = "0.2.0"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "2ca267b08821e86c5ef4376cffed98a46c2cb205"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.0.1"

[[MixedModels]]
deps = ["Arrow", "DataAPI", "Distributions", "GLM", "JSON3", "LazyArtifacts", "LinearAlgebra", "Markdown", "NLopt", "PooledArrays", "ProgressMeter", "Random", "SparseArrays", "StaticArrays", "Statistics", "StatsBase", "StatsFuns", "StatsModels", "StructTypes", "Tables"]
git-tree-sha1 = "f318e42a48ec0a856292bafeec6b07aed3f6d600"
uuid = "ff71e718-51f3-5ec2-a782-8ffcbfa3c316"
version = "4.1.1"

[[MixedModelsMakie]]
deps = ["DataFrames", "Distributions", "KernelDensity", "LinearAlgebra", "Makie", "MixedModels", "Printf", "SpecialFunctions", "StatsBase"]
git-tree-sha1 = "e2ae99b3b4675ddc91e9c421a775eff52a784696"
uuid = "b12ae82c-6730-437f-aff9-d2c38332a376"
version = "0.3.8"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[Mocking]]
deps = ["ExprTools"]
git-tree-sha1 = "748f6e1e4de814b101911e64cc12d83a6af66782"
uuid = "78c3b35d-d492-501b-9361-3d52fe80e533"
version = "0.7.2"

[[MosaicViews]]
deps = ["MappedArrays", "OffsetArrays", "PaddedViews", "StackViews"]
git-tree-sha1 = "b34e3bc3ca7c94914418637cb10cc4d1d80d877d"
uuid = "e94cdb99-869f-56ef-bcf0-1ae2bcbe0389"
version = "0.3.3"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[MutableArithmetics]]
deps = ["LinearAlgebra", "SparseArrays", "Test"]
git-tree-sha1 = "3927848ccebcc165952dc0d9ac9aa274a87bfe01"
uuid = "d8a4904e-b15c-11e9-3269-09a3773c0cb0"
version = "0.2.20"

[[NLopt]]
deps = ["MathOptInterface", "MathProgBase", "NLopt_jll"]
git-tree-sha1 = "f115030b9325ca09ef1619ba0617b2a64101ce84"
uuid = "76087f3c-5699-56af-9a33-bf431cd00edd"
version = "0.6.4"

[[NLopt_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "2b597c46900f5f811bec31f0dcc88b45744a2a09"
uuid = "079eb43e-fd8e-5478-9966-2cf3e3edb778"
version = "2.7.0+0"

[[NaNMath]]
git-tree-sha1 = "bfe47e760d60b82b66b61d2d44128b62e3a369fb"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "0.3.5"

[[Netpbm]]
deps = ["ColorVectorSpace", "FileIO", "ImageCore"]
git-tree-sha1 = "09589171688f0039f13ebe0fdcc7288f50228b52"
uuid = "f09324ee-3d7c-5217-9330-fc30815ba969"
version = "1.0.1"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[Observables]]
git-tree-sha1 = "fe29afdef3d0c4a8286128d4e45cc50621b1e43d"
uuid = "510215fc-4207-5dde-b226-833fc4488ee2"
version = "0.4.0"

[[OffsetArrays]]
deps = ["Adapt"]
git-tree-sha1 = "c870a0d713b51e4b49be6432eff0e26a4325afee"
uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
version = "1.10.6"

[[Ogg_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "7937eda4681660b4d6aeeecc2f7e1c81c8ee4e2f"
uuid = "e7412a2a-1a6e-54c0-be00-318e2571c051"
version = "1.3.5+0"

[[OpenSSL_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "15003dcb7d8db3c6c857fda14891a539a8f2705a"
uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95"
version = "1.1.10+0"

[[OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[Opus_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "51a08fb14ec28da2ec7a927c4337e4332c2a4720"
uuid = "91d4177d-7536-5919-b921-800302f37372"
version = "1.3.2+0"

[[OrderedCollections]]
git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.1"

[[PCRE_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b2a7af664e098055a7529ad1a900ded962bca488"
uuid = "2f80f16e-611a-54ab-bc61-aa92de5b98fc"
version = "8.44.0+0"

[[PDMats]]
deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"]
git-tree-sha1 = "4dd403333bcf0909341cfe57ec115152f937d7d8"
uuid = "90014a1f-27ba-587c-ab20-58faa44d9150"
version = "0.11.1"

[[PNGFiles]]
deps = ["Base64", "CEnum", "ImageCore", "IndirectArrays", "OffsetArrays", "libpng_jll"]
git-tree-sha1 = "e14c485f6beee0c7a8dcf6128bf70b85f1fe201e"
uuid = "f57f5aa1-a3ce-4bc8-8ab9-96f992907883"
version = "0.3.9"

[[Packing]]
deps = ["GeometryBasics"]
git-tree-sha1 = "1155f6f937fa2b94104162f01fa400e192e4272f"
uuid = "19eb6ba3-879d-56ad-ad62-d5c202156566"
version = "0.4.2"

[[PaddedViews]]
deps = ["OffsetArrays"]
git-tree-sha1 = "646eed6f6a5d8df6708f15ea7e02a7a2c4fe4800"
uuid = "5432bcbf-9aad-5242-b902-cca2824c8663"
version = "0.5.10"

[[Pango_jll]]
deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "FriBidi_jll", "Glib_jll", "HarfBuzz_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "9bc1871464b12ed19297fbc56c4fb4ba84988b0d"
uuid = "36c8627f-9965-5494-a995-c6b170f724f3"
version = "1.47.0+0"

[[Parsers]]
deps = ["Dates"]
git-tree-sha1 = "438d35d2d95ae2c5e8780b330592b6de8494e779"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.0.3"

[[Pixman_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b4f5d02549a10e20780a24fce72bea96b6329e29"
uuid = "30392449-352a-5448-841d-b1acce4e97dc"
version = "0.40.1+0"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[PlotUtils]]
deps = ["ColorSchemes", "Colors", "Dates", "Printf", "Random", "Reexport", "Statistics"]
git-tree-sha1 = "9ff1c70190c1c30aebca35dc489f7411b256cd23"
uuid = "995b91a9-d308-5afd-9ec6-746e21dbc043"
version = "1.0.13"

[[PlutoUI]]
deps = ["Base64", "Dates", "InteractiveUtils", "JSON", "Logging", "Markdown", "Random", "Reexport", "Suppressor"]
git-tree-sha1 = "44e225d5837e2a2345e69a1d1e01ac2443ff9fcb"
uuid = "7f904dfe-b85e-4ff6-b463-dae2292396a8"
version = "0.7.9"

[[PolygonOps]]
git-tree-sha1 = "c031d2332c9a8e1c90eca239385815dc271abb22"
uuid = "647866c9-e3ac-4575-94e7-e3d426903924"
version = "0.1.1"

[[PooledArrays]]
deps = ["DataAPI", "Future"]
git-tree-sha1 = "a193d6ad9c45ada72c14b731a318bedd3c2f00cf"
uuid = "2dfb63ee-cc39-5dd5-95bd-886bf059d720"
version = "1.3.0"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.2"

[[PrettyTables]]
deps = ["Crayons", "Formatting", "Markdown", "Reexport", "Tables"]
git-tree-sha1 = "0d1245a357cc61c8cd61934c07447aa569ff22e6"
uuid = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
version = "1.1.0"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[ProgressMeter]]
deps = ["Distributed", "Printf"]
git-tree-sha1 = "afadeba63d90ff223a6a48d2009434ecee2ec9e8"
uuid = "92933f4c-e287-5a05-a399-4b506db050ca"
version = "1.7.1"

[[QuadGK]]
deps = ["DataStructures", "LinearAlgebra"]
git-tree-sha1 = "12fbe86da16df6679be7521dfb39fbc861e1dc7b"
uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc"
version = "2.4.1"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[Ratios]]
deps = ["Requires"]
git-tree-sha1 = "7dff99fbc740e2f8228c6878e2aad6d7c2678098"
uuid = "c84ed2f1-dad5-54f0-aa8e-dbefe2724439"
version = "0.4.1"

[[RecipesBase]]
git-tree-sha1 = "44a75aa7a527910ee3d1751d1f0e4148698add9e"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.1.2"

[[Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[RelocatableFolders]]
deps = ["SHA", "Scratch"]
git-tree-sha1 = "0529f4188bc8efee85a7e580aca1c7dff6b103f8"
uuid = "05181044-ff0b-4ac5-8273-598c1e38db00"
version = "0.1.0"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.1.3"

[[Rmath]]
deps = ["Random", "Rmath_jll"]
git-tree-sha1 = "bf3188feca147ce108c76ad82c2792c57abe7b1f"
uuid = "79098fc4-a85e-5d69-aa6a-4863f24498fa"
version = "0.7.0"

[[Rmath_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "68db32dff12bb6127bac73c209881191bf0efbb7"
uuid = "f50d1b31-88e8-58de-be2c-1cc44531875f"
version = "0.3.0+0"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[SIMD]]
git-tree-sha1 = "9ba33637b24341aba594a2783a502760aa0bff04"
uuid = "fdea26ae-647d-5447-a871-4b548cad5224"
version = "3.3.1"

[[ScanByte]]
deps = ["Libdl", "SIMD"]
git-tree-sha1 = "9cc2955f2a254b18be655a4ee70bc4031b2b189e"
uuid = "7b38b023-a4d7-4c5e-8d43-3f3097f304eb"
version = "0.3.0"

[[Scratch]]
deps = ["Dates"]
git-tree-sha1 = "0b4b7f1393cff97c33891da2a0bf69c6ed241fda"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.1.0"

[[SentinelArrays]]
deps = ["Dates", "Random"]
git-tree-sha1 = "54f37736d8934a12a200edea2f9206b03bdf3159"
uuid = "91c51154-3ec4-41a3-a24f-3f23e20d615c"
version = "1.3.7"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[ShiftedArrays]]
git-tree-sha1 = "22395afdcf37d6709a5a0766cc4a5ca52cb85ea0"
uuid = "1277b4bf-5013-50f5-be3d-901d8477a67a"
version = "1.0.0"

[[Showoff]]
deps = ["Dates", "Grisu"]
git-tree-sha1 = "91eddf657aca81df9ae6ceb20b959ae5653ad1de"
uuid = "992d4aef-0814-514b-bc4d-f2e9a6c4116f"
version = "1.0.3"

[[SignedDistanceFields]]
deps = ["Random", "Statistics", "Test"]
git-tree-sha1 = "d263a08ec505853a5ff1c1ebde2070419e3f28e9"
uuid = "73760f76-fbc4-59ce-8f25-708e95d2df96"
version = "0.4.0"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.0.1"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[SpecialFunctions]]
deps = ["ChainRulesCore", "LogExpFunctions", "OpenSpecFun_jll"]
git-tree-sha1 = "a322a9493e49c5f3a10b50df3aedaf1cdb3244b7"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "1.6.1"

[[StackViews]]
deps = ["OffsetArrays"]
git-tree-sha1 = "46e589465204cd0c08b4bd97385e4fa79a0c770c"
uuid = "cae243ae-269e-4f55-b966-ac2d0dc13c15"
version = "0.1.1"

[[Static]]
deps = ["IfElse"]
git-tree-sha1 = "854b024a4a81b05c0792a4b45293b85db228bd27"
uuid = "aedffcd0-7271-4cad-89d0-dc628f76c6d3"
version = "0.3.1"

[[StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "3240808c6d463ac46f1c1cd7638375cd22abbccb"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.2.12"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[StatsAPI]]
git-tree-sha1 = "1958272568dc176a1d881acb797beb909c785510"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.0.0"

[[StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "8cbbc098554648c84f79a463c9ff0fd277144b6c"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.10"

[[StatsFuns]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "Reexport", "Rmath", "SpecialFunctions"]
git-tree-sha1 = "46d7ccc7104860c38b11966dd1f72ff042f382e4"
uuid = "4c63d2b9-4356-54db-8cca-17b64c39e42c"
version = "0.9.10"

[[StatsModels]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Printf", "ShiftedArrays", "SparseArrays", "StatsBase", "StatsFuns", "Tables"]
git-tree-sha1 = "3fa15c1f8be168e76d59097f66970adc86bfeb95"
uuid = "3eaba693-59b7-5ba5-a881-562e759f1c8d"
version = "0.6.25"

[[StructArrays]]
deps = ["Adapt", "DataAPI", "StaticArrays", "Tables"]
git-tree-sha1 = "1700b86ad59348c0f9f68ddc95117071f947072d"
uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a"
version = "0.6.1"

[[StructTypes]]
deps = ["Dates", "UUIDs"]
git-tree-sha1 = "8445bf99a36d703a09c601f9a57e2f83000ef2ae"
uuid = "856f2bd8-1eba-4b0a-8007-ebc267875bd4"
version = "1.7.3"

[[SuiteSparse]]
deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"]
uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9"

[[Suppressor]]
git-tree-sha1 = "a819d77f31f83e5792a76081eee1ea6342ab8787"
uuid = "fd094767-a336-5f1f-9728-57cf17d0bbfb"
version = "0.2.0"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.1"

[[Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"]
git-tree-sha1 = "368d04a820fe069f9080ff1b432147a6203c3c89"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.5.1"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[TimeZones]]
deps = ["Dates", "Future", "LazyArtifacts", "Mocking", "Pkg", "Printf", "RecipesBase", "Serialization", "Unicode"]
git-tree-sha1 = "6c9040665b2da00d30143261aea22c7427aada1c"
uuid = "f269a46b-ccf7-5d73-abea-4c690281aa53"
version = "1.5.7"

[[TranscodingStreams]]
deps = ["Random", "Test"]
git-tree-sha1 = "216b95ea110b5972db65aa90f88d8d89dcb8851c"
uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
version = "0.9.6"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[UnicodeFun]]
deps = ["REPL"]
git-tree-sha1 = "53915e50200959667e78a92a418594b428dffddf"
uuid = "1cfade01-22cf-5700-b092-accc4b62d6e1"
version = "0.4.1"

[[WoodburyMatrices]]
deps = ["LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "59e2ad8fd1591ea019a5259bd012d7aee15f995c"
uuid = "efce3f68-66dc-5838-9240-27a6d6f5f9b6"
version = "0.5.3"

[[XML2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "1acf5bdf07aa0907e0a37d3718bb88d4b687b74a"
uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a"
version = "2.9.12+0"

[[XSLT_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"]
git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a"
uuid = "aed1982a-8fda-507f-9586-7b0439959a61"
version = "1.1.34+0"

[[Xorg_libX11_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll", "Xorg_xtrans_jll"]
git-tree-sha1 = "5be649d550f3f4b95308bf0183b82e2582876527"
uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc"
version = "1.6.9+4"

[[Xorg_libXau_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4e490d5c960c314f33885790ed410ff3a94ce67e"
uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec"
version = "1.0.9+4"

[[Xorg_libXdmcp_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4fe47bd2247248125c428978740e18a681372dd4"
uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05"
version = "1.1.3+4"

[[Xorg_libXext_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "b7c0aa8c376b31e4852b360222848637f481f8c3"
uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3"
version = "1.3.4+4"

[[Xorg_libXrender_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "19560f30fd49f4d4efbe7002a1037f8c43d43b96"
uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa"
version = "0.9.10+4"

[[Xorg_libpthread_stubs_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "6783737e45d3c59a4a4c4091f5f88cdcf0908cbb"
uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74"
version = "0.1.0+3"

[[Xorg_libxcb_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"]
git-tree-sha1 = "daf17f441228e7a3833846cd048892861cff16d6"
uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b"
version = "1.13.0+3"

[[Xorg_xtrans_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "79c31e7844f6ecf779705fbc12146eb190b7d845"
uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10"
version = "1.4.0+3"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[Zstd_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "cc4bf3fdde8b7e3e9fa0351bdeedba1cf3b7f6e6"
uuid = "3161d3a3-bdf6-5164-811a-617609db77b4"
version = "1.5.0+0"

[[isoband_jll]]
deps = ["Libdl", "Pkg"]
git-tree-sha1 = "a1ac99674715995a536bbce674b068ec1b7d893d"
uuid = "9a68df92-36a6-505f-a73e-abb412b6bfb4"
version = "0.2.2+0"

[[libass_jll]]
deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "HarfBuzz_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "5982a94fcba20f02f42ace44b9894ee2b140fe47"
uuid = "0ac62f75-1d6f-5e53-bd7c-93b484bb37c0"
version = "0.15.1+0"

[[libfdk_aac_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "daacc84a041563f965be61859a36e17c4e4fcd55"
uuid = "f638f0a6-7fb0-5443-88ba-1cc74229b280"
version = "2.0.2+0"

[[libpng_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "94d180a6d2b5e55e447e2d27a29ed04fe79eb30c"
uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f"
version = "1.6.38+0"

[[libvorbis_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Ogg_jll", "Pkg"]
git-tree-sha1 = "c45f4e40e7aafe9d086379e5578947ec8b95a8fb"
uuid = "f27f6e37-5d2b-51aa-960f-b287f2bc3b7a"
version = "1.3.7+0"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

[[x264_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4fea590b89e6ec504593146bf8b988b2c00922b2"
uuid = "1270edf5-f2f9-52d2-97e9-ab00b5d0237a"
version = "2021.5.5+0"

[[x265_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "ee567a171cce03570d77ad3a43e90218e38937a9"
uuid = "dfaa095f-4041-5dcd-9319-2fabd8486b76"
version = "3.5.0+0"
"""

# ╔═╡ Cell order:
# ╠═acbbdcab-b678-4143-95f2-d2be5f83e925
# ╟─fbd6339a-10d4-11ec-3c2e-15dff05b3589
# ╠═c72485ee-c4a7-427c-b590-5d9682fb09b3
# ╠═9dea9eac-ba97-49b1-9003-09142d186b43
# ╟─ec752d1e-e14a-431a-82b8-32a6e1bd7b5f
# ╠═73e26ca3-51a4-4873-b561-f0b1e61879f1
# ╠═d366844b-7e04-49b8-8d49-0ef82274a7e5
# ╠═0bbed5cf-1417-42ca-80f6-306200c50077
# ╟─637d518b-dce3-40d5-87c8-bdc655570f8c
# ╠═2c1cfa14-99f9-4916-bdc4-8c8af3b34bc3
# ╟─5fcb27d2-771c-417c-aa58-771cdd91a8ff
# ╠═b8359d5d-0a2b-4bf4-a8d9-cb4ffe0d5b09
# ╟─93f478c1-f3fd-4163-954d-2d52a7e7efce
# ╠═d8453ffc-cabf-44b5-9066-65f2502c12ad
# ╠═3a879484-b1ff-4ae8-8db4-6e17e5e073c8
# ╟─21b519d9-2140-4af7-83c7-b851cdc18c18
# ╠═765baad6-b96f-4d13-b46c-bd848d8845c4
# ╠═7a2a3c47-44e1-4061-a186-7e3a9b0269be
# ╠═cd024e2b-592a-4801-bfa0-1a58e3dded1e
# ╟─bce7177c-e4d9-4acf-9ac1-2ff8f244c038
# ╠═8d2ae114-5780-4d38-ba40-95aac032337b
# ╟─d90a8d29-afc8-470e-b5da-f35a4cadf1c5
# ╠═7d836c30-f099-4bae-921c-f5a9cf57c5cc
# ╟─7390872e-9c9a-4d77-9f59-bdc7d9e29ab5
# ╠═a6553206-8116-49a5-8994-0fc7f3c6e2cd
# ╠═cae81515-4a9c-44f9-a24e-3d5f08c7933c
# ╟─57bc6bbf-e4e4-4abc-960e-3e89e9312d66
# ╟─b7a9dadc-b95e-4abc-89d1-e9dff22e0a40
# ╠═f850d266-934c-400b-9f20-a848f08513de
# ╟─19f94d31-9321-440b-87e4-f2b488a5c6ca
# ╠═f0d0fc9c-f817-4f3c-99de-c2442126509b
# ╟─f02f0407-8e38-4cef-a509-18abd0594b3a
# ╠═d68d3d90-3584-4d70-9d83-d203ba28049c
# ╠═63610f52-5067-4754-a86b-675711c4f36a
# ╠═3e8a4b2e-4996-4756-9932-93fbc3efb3e6
# ╟─f16be61e-3b9b-40ba-b109-8e46d3af5298
# ╟─f83aaa94-4620-41f2-8f61-893593aecb61
# ╠═3b582cbb-f7e1-42bf-aa8d-abfa96342a32
# ╠═77e2e52d-6133-4631-9675-d6db1f079ead
# ╟─179ee9eb-020b-4f63-8699-bd1a0368e6e9
# ╠═73332322-ffc9-4e83-9f7e-94b726405c97
# ╠═aa274ce1-0d31-4701-a967-7d7889b2ac38
# ╟─7204cac6-fad2-4ccf-86ff-2d91c426c6c6
# ╠═dc7f19bc-a6fe-4d3c-8e78-1426aa6081e7
# ╟─b74f0e46-1046-4f44-8c0b-235351251e86
# ╠═f2fd5625-cab1-4ce2-8592-f6215ea65b86
# ╟─0a3351b2-7e1c-45bb-97a7-05bc7be4826f
# ╟─00000000-0000-0000-0000-000000000001
# ╟─00000000-0000-0000-0000-000000000002


using Statistics
using RDatasets
data = dataset("datasets", "iris")

# Summary Statistics
describe(data)

# Unique elements in Species
unique(data.Species)
# Standard Deviation of PetalWidth
std(data.PetalWidth)

# Variance of  PetalWidth
var(data.PetalWidth)

#Mean of PetalWidth
mean(data.PetalWidth)

#Median of PetalWidth
median(data.PetalWidth)

#Correlation between PetalWidth & PetalLength
cor(data.PetalWidth, data.PetalLength)


import Random
using Test: @test, @testset, @inferred
using Statistics: mean, var
using Random: bitrand, rand!, randn!
using RestrictedBoltzmannMachines: RBM, Binary, visible, hidden, weights, free_energy
using RestrictedBoltzmannMachines: Gaussian, ReLU, dReLU, pReLU, xReLU
using RestrictedBoltzmannMachines: sample_v_from_v, sample_h_from_h, mean_from_inputs, var_from_inputs
using RestrictedBoltzmannMachines: rescale_hidden!, rescale_activations!

Random.seed!(23)

@testset "rescale_activations!" begin
    N = (7,2)
    layers = (
        Gaussian(randn(N...), randn(N...)),
        ReLU(randn(N...), randn(N...)),
        dReLU(randn(N...), randn(N...), randn(N...), randn(N...)),
        pReLU(randn(N...), randn(N...), randn(N...), 2rand(N...) .- 1),
        xReLU(randn(N...), randn(N...), randn(N...), randn(N...))
    )
    λ = rand(N...)
    for layer in layers
        μ = mean_from_inputs(layer)
        ν = var_from_inputs(layer)
        rescale_activations!(layer, λ)
        @test mean_from_inputs(layer) ≈ μ ./ λ
        @test var_from_inputs(layer) ≈ ν ./ λ.^2
    end
end

@testset "rescale_hidden!" begin
    rbm = RBM(Binary(2), ReLU(1), randn(2,1))
    randn!(rbm.visible.θ)
    randn!(rbm.hidden.θ)
    rand!(rbm.hidden.γ)
    rbm.hidden.γ .+= 0.5

    v = sample_v_from_v(rbm, bitrand(size(visible(rbm))..., 20000); steps=100)
    h = sample_h_from_h(rbm, rand(size(hidden(rbm))..., 20000); steps=100)
    ave_v = mean(v; dims=2)
    ave_h = mean(h; dims=2)
    var_v = var(v; dims=2)
    var_h = var(h; dims=2)
    F = free_energy(rbm, v)

    λ = [1 + rand()]
    rescale_hidden!(rbm, λ)

    @test free_energy(rbm, v) ≈ F .+ sum(log, λ)

    v = sample_v_from_v(rbm, bitrand(size(visible(rbm))..., 20000); steps=100)
    h = sample_h_from_h(rbm, rand(size(hidden(rbm))..., 20000); steps=100)

    @test mean(v; dims=2) ≈ ave_v rtol=0.1
    @test mean(h; dims=2) ≈ ave_h ./ λ rtol=0.1
    @test var(v; dims=2) ≈ var_v rtol=0.1
    @test var(h; dims=2) ≈ var_h ./ λ.^2 rtol=0.1
end


# Michelle Medina
# August 5, 2020

using PyPlot

# x = collect(range(0, stop=1, length = 1025))
# level = 4
# for l in 0:level, j in 0:2^l
# 	plot(x, basis.(l, j, x), "m")
# 	plot(x, derivBasis.(l, j, x))
# 	xlabel("x")
# 	ylabel("phi'(x)")
# 	show()
# end

# plot(x, derivBasis.(0, 0, x))
# plot(x, derivBasis.(0, 1, x))
# plot(x, derivBasis.(1, 0, x))
# plot(x, derivBasis.(1, 1, x))
# plot(x, derivBasis.(1, 2, x))
# show()

# xVal = collect(range(0.0, stop=1.0, length = 129))
# y = calcNodal(x -> sin(4*pi*x), xVal)
# cosx = calcNodal(x -> 4*pi*cos(4*pi*x), xVal)
# modal = Nodal_2_H(y)
#
# dy = zeros(length(xVal))
# for index in CartesianIndices(xVal)
# 	dy[index] = derivEvaluate(modal, xVal[index])
# end

# plot(xVal,y.values)
# plot(xVal,dy, "g .")
# plot(xVal, cosx.values, "r")
#

error = []
Num = []

level = 13
for l in 0:level
	N = 2^l + 1

	xOld = collect(range(0.0, stop=1.0, length = N))
	U = calcNodal(x -> exp(x), xOld)
	dU = calcNodal(x -> exp(x), xOld)
	modal_U = Nodal_2_H(U)

	# Compute the interpolation error
	le = 10 * (2^l+1)
	h = 1/le
	xNew = collect(range(0.0 + h/2, stop=1.0 - h/2, length = le))
	dUNew = calcNodal(x -> exp(x), xNew)
	dU_from_modal_U = calcNodal(x -> derivEvaluate(modal_U, x), xNew)

	error_up_to = sqrt(sum((dUNew.values - dU_from_modal_U.values).^2)/length(dUNew.values))

	append!(error, error_up_to)
	append!(Num, N)

	# @show l, N, error_up_to
end

loglog(Num, error)
loglog(Num, 1 ./Num)
loglog(Num, 1 ./(Num.^2))
show()


# TODO: documment all these methods

degree(g::Graph{N,E,D}, n::Integer) where {N,E,D} = ptr_length(g.nodes[n], dir_xor(D, Both))
degree(g::Graph) = [degree(g, i) for i = nodes(g)]

indegree(g::Graph) = [indegree(g, i) for i = nodes(g)]
indegree(g::Graph{N,E,D}, n::Integer) where {N,E,D} = ptr_length(g.nodes[n], dir_xor(D, Reverse))

outdegree(g::Graph{N,E,D}, n::Integer) where {N,E,D} = ptr_length(g.nodes[n], dir_xor(D, Forward))
outdegree(g::Graph) = [outdegree(g, i) for i = nodes(g)]

"""
    is_isolate(g::Graph, n::Integer)

Is node `n` an isolate, i.e. not connected to any other node?
"""
is_isolate(g::Graph, n::Integer) = degree(g, n) == 0
isolates(g::Graph) = [i for i = nodes(g) if is_isolate(g, i)]
isolatecount(g::Graph) = count(i -> is_isolate(g, i), nodes(g))

"""
    is_leaf(g::Graph, n::Integer)

Is node `n` a leaf node (i.e. a source or sink)?
See the [reference](https://en.wikipedia.org/wiki/Vertex_(graph_theory)#Types_of_vertices)
for more information.
"""
is_leaf(g::Graph, n::Integer) = degree(g, n) == 1
leaves(g::Graph) = [i for i = nodes(g) if is_leaf(g, i)]
leafcount(g::Graph) = count(i -> is_leaf(g, i), nodes(g))

"""
    is_source(g::Graph, n::Integer)

Is node `n` a source? Sources can only have outgoing edges.
In undirected graphs, there can't be any sources.
"""
is_source(g::DirectedGraph, n::Integer)  = indegree(g, n) == 0 && outdegree(g, n) > 0
is_source(g::UndirectedGraph, ::Integer) = false
sources(g::Graph) = [i for i = nodes(g) if is_source(g, i)]
sourcecount(g::Graph) = count(i -> is_source(g, i), nodes(g))

"""
    is_sink(g::Graph, n::Integer)

Is node `n` a sink? Sinks can only have incoming edges.
In undirected graphs, there can't be any sinks.
"""
is_sink(g::DirectedGraph, n::Integer)  = outdegree(g, n) == 0 && indegree(g, n) > 0
is_sink(g::UndirectedGraph, ::Integer) = false
sinks(g::Graph) = [i for i = nodes(g) if is_sink(g, i)]
sinkcount(g::Graph) = count(i -> is_sink(g, i), nodes(g))


"""
    Certify(optimizer)

Certify uses semidefinite programming to compute over-approximated certificates for a neural network with only one hidden layer.

# Problem requirement
1. Network: one hidden layer, any activation that is differentiable almost everywhere whose derivative is bound by 0 and 1
2. Input: hypercube
3. Output: halfspace

# Return
`BasicResult`

# Method
Semindefinite programming.
Default `optimizer` is `SCSSolver()`.

# Property
Sound but not complete.

# Reference
[A. Raghunathan, J. Steinhardt, and P. Liang,
"Certified Defenses against Adversarial Examples,"
*ArXiv Preprint ArXiv:1801.09344*, 2018.](https://arxiv.org/abs/1801.09344)
"""
@with_kw struct Certify{O<:AbstractMathProgSolver}
    optimizer::O  = SCSSolver()
end

# can pass keyword args to optimizer
# Certify(optimizer::DataType = SCSSolver; kwargs...) =  Certify(optimizer(;kwargs...))

function solve(solver::Certify, problem::Problem)
    @assert length(problem.network.layers) == 2 "Certify only handles Networks that have one hidden layer. Got $(length(problem.network.layers)) total layers"
    model = Model(solver = solver.optimizer)
    c, d = tosimplehrep(problem.output)
    v = c * problem.network.layers[2].weights
    W = problem.network.layers[1].weights
    M = get_M(v[1, :], W)
    n = size(M, 1)
    P = @variable(model, [1:n, 1:n], SDP)
    # Compute value
    output = c * compute_output(problem.network, problem.input.center) .- d[1]
    epsilon = problem.input.radius[1]
    o = output + epsilon/4 * tr(M*P)
    # Specify problem
    @constraint(model, diag(P) .<= ones(n))
    @objective(model, Max, o[1])
    status = solve(model, suppress_warnings = true)
    return interpret_result(solver, status, o[1])
end

# True if o < 0
# Undertermined if otherwise
function interpret_result(solver::Certify, status, o)
    # println("Upper bound: ", getvalue(o[1]))
    if getvalue(o) <= 0
        return BasicResult(:SAT)
    else
        return BasicResult(:Unknown)
    end
end

# M is used in the semidefinite program
function get_M(v::Vector{Float64}, W::Matrix{Float64})
    m = W' * Diagonal(v)
    mxs, mys = size(m)
    o = ones(size(W, 2), 1)
    # TODO Mrow2 and Mrow3 look suspicious (dims don't match in the general case).
    Mrow1 = [zeros(1, 1+mxs)    o'*m]
    Mrow2 = [zeros(mxs, 1+mxs)     m]
    Mrow3 = [m'*o  m' zeros(mys, mys)]

    M = [Mrow1; Mrow2; Mrow3]
    return M
end


function ising_local_MPO(::Type{T}, shift::Number=0.0; hz::Number=1.0, hx::Number=0.0)::MPO_open{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]

    h1_11 = -hz*Z - hx*X + shift*I

    h1 = zeros(eltype(h1_11), 2,2,1,2)
    h2 = zeros(eltype(h1_11), 2,2,2,1)

    h1[:,:,1,1] = h1_11
    h1[:,:,1,2] = -X

    h2[:,:,1,1] = E
    h2[:,:,2,1] = X

    h1 = permutedims(h1, (3,2,4,1)) #[m1,ket,m2,bra]
    h2 = permutedims(h2, (3,2,4,1)) #[m1,ket,m2,bra]

    h1 = convert(MPOTensor{T}, h1)
    h2 = convert(MPOTensor{T}, h2)

    MPOTensor{T}[h1, h2]
end

function ising_PBC_MPO(::Type{T}; hz::Real=1.0, hx::Real=0.0)::MPO_PBC_uniform{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]

    hM = zeros(Float64, 2,2,3,3)
    hB = zeros(Float64, 2,2,3,3)

    hM[:,:,1,1] = E
    hM[:,:,2,1] = X
    hM[:,:,3,1] = -hz*Z - hx*X
    hM[:,:,3,2] = -X
    hM[:,:,3,3] = E

    hB[:,:,1,:] = hM[:,:,3,:]
    hB[:,:,2,3] = X

    hM = permutedims(hM, (3,2,4,1)) #[m1,ket,m2,bra]
    hB = permutedims(hB, (3,2,4,1)) #[m1,ket,m2,bra]

    hM = convert(MPOTensor{T}, hM)
    hB = convert(MPOTensor{T}, hB)

    (hB, hM)
end

function ising_OBC_MPO(::Type{T}; hz::Real=1.0, hx::Real=0.0)::MPO_open_uniform{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]

    hM = zeros(Float64, 2,2,3,3)

    hM[:,:,1,1] = E
    hM[:,:,2,1] = X
    hM[:,:,3,1] = -hz*Z - hx*X
    hM[:,:,3,2] = -X
    hM[:,:,3,3] = E

    hL = zeros(Float64, 2,2,1,3)
    hR = zeros(Float64, 2,2,3,1)

    hL[:,:,1,:] = hM[:,:,3,:]
    hR[:,:,:,1] = hM[:,:,:,1]

    hM = permutedims(hM, (3,2,4,1)) #[m1,ket,m2,bra]
    hL = permutedims(hL, (3,2,4,1)) #[m1,ket,m2,bra]
    hR = permutedims(hR, (3,2,4,1)) #[m1,ket,m2,bra]

    hM = convert(MPOTensor{T}, hM)
    hL = convert(MPOTensor{T}, hL)
    hR = convert(MPOTensor{T}, hR)

    (hL, hM, hR)
end

function ising_PBC_MPO_split(::Type{T}; hz::Real=1.0, hx::Real=0.0)::MPO_PBC_uniform_split{T} where {T}
    X = [0.0 1.0; 1.0 0.0]
    hL = reshape(-X, (2,2,1,1))
    hR = reshape(X, (2,2,1,1))
    hL = convert(MPOTensor{T}, permutedims(hL, (3,2,4,1)))
    hR = convert(MPOTensor{T}, permutedims(hR, (3,2,4,1)))

    h_B = MPOTensor{T}[hL, hR]

    (ising_OBC_MPO(T, hz=hz, hx=hx), h_B)
end

function ising_Hn_MPO_split(::Type{T}, n::Integer, N::Integer; hz::Real=1.0, hx::Real=0.0) where {T}
    (hL, hM, hR), (hb1, hb2) = ising_PBC_MPO_split(T, hz=hz, hx=hx)

    hL[1,:,1,:] *= cis(n*2π/N)
    hL[1,:,2,:] *= cis(n*1.5*2π/N)

    hR[3,:,1,:] *= cis(n*N*2π/N) #I realise this is not strictly necessary, but it shows intent! :)

    get_hM = (j::Integer)->begin
        hM_j = copy(hM)
        hM_j[3,:,1,:] *= cis(n*j*2π/N)
        hM_j[3,:,2,:] *= cis(n*(j+0.5)*2π/N)
        hM_j
    end

    Hn_OBC = MPOTensor{T}[hL, (get_hM(n) for n in 2:N-1)..., hR]

    hb1 *= cis(n*(N+0.5)*2π/N)

    Hn_b = MPOTensor{T}[hb1, hb2]

    n, (Hn_OBC, Hn_b)
end

#------------------------------
function heis_local_MPO(::Type{T}; J::Number=1, spin::Real=1//2)::MPO_open{T} where {T}
    if spin == 1//2
        E = Matrix{ComplexF64}(I,2,2)
        X = complex([0.0 1.0; 1.0 0.0])
        Y = 1.0im*[0.0 -1.0; 1.0 0.0]
        Z = complex([1.0 0.0; 0.0 -1.0])
        zer = zero(E)
    else
        ValueError()
    end

    h1 = J*[X Y Z]
    h2 = [X;
          Y;
          Z]

    h1 = reshape(h1, (2,1,2,3))
    h2 = reshape(h2, (2,3,2,1))

    h1 = permutedims(h1, (2,3,4,1))
    h2 = permutedims(h2, (2,3,4,1))

    MPOTensor{T}[convert(MPOTensor{T}, h1), convert(MPOTensor{T}, h2)]
end

#------------------------------

"""
-(lamda*X1*X2 + delta1*X1*X3 + delta2*Z1*Z2 + hz*Z1)
"""
function ANNNI_local_MPO(::Type{T}; hz::Real=1.0, delta1::Real=0.0, delta2::Real=0.0, lambda::Real=1.0)::MPO_open{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]

    h1 = zeros(Float64, 2,2,1,2)
    h2 = zeros(Float64, 2,2,2,2)
    h3 = zeros(Float64, 2,2,2,1)

    h1[:,:,1,1] = -X
    h1[:,:,1,2] = -Z

    h2[:,:,1,1] = lambda*X
    h2[:,:,1,2] = E
    h2[:,:,2,1] = delta2 * Z + I*hz

    h3[:,:,1,1] = E
    h3[:,:,2,1] = delta1 * X

    h1 = permutedims(h1, (3,2,4,1)) #[m1,ket,m2,bra]
    h2 = permutedims(h2, (3,2,4,1)) #[m1,ket,m2,bra]
    h3 = permutedims(h3, (3,2,4,1)) #[m1,ket,m2,bra]

    h1 = convert(MPOTensor{T}, h1)
    h2 = convert(MPOTensor{T}, h2)
    h3 = convert(MPOTensor{T}, h3)

    MPOTensor{T}[h1, h2, h3]
end

function ANNNI_OBC_MPO(::Type{T}; hz::Real=1.0, delta1::Real=0.0, delta2::Real=0.0, lambda::Real=1.0)::MPO_open_uniform{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]

    hM = zeros(Float64, 2,2,5,5)

    hM[:,:,1,1] = E
    hM[:,:,2,1] = X
    hM[:,:,3,1] = Z
    hM[:,:,4,2] = E
    hM[:,:,5,2] = -lambda*X
    hM[:,:,5,3] = -delta2*Z-hz*I
    hM[:,:,5,4] = -delta1*X
    hM[:,:,5,5] = E

    hL = zeros(Float64, 2,2,1,5)
    hR = zeros(Float64, 2,2,5,1)

    hL[:,:,1,:] = hM[:,:,5,:]
    hL[:,:,1,1] = -hz*Z
    hR[:,:,:,1] = hM[:,:,:,1]

    hM = permutedims(hM, (3,2,4,1)) #[m1,ket,m2,bra]
    hL = permutedims(hL, (3,2,4,1)) #[m1,ket,m2,bra]
    hR = permutedims(hR, (3,2,4,1)) #[m1,ket,m2,bra]

    hM = convert(MPOTensor{T}, hM)
    hL = convert(MPOTensor{T}, hL)
    hR = convert(MPOTensor{T}, hR)

    (hL, hM, hR)
end

function ANNNI_PBC_MPO_split(::Type{T}; hz::Real=1.0, delta1::Real=0.0, delta2::Real=0.0, lambda::Real=1.0)::MPO_PBC_uniform_split{T} where {T}
    #Turn off hz for boundary terms, since they're already in the OBC part.
    hL, hM, hR = ANNNI_OBC_MPO(T, hz=0.0, delta1=delta1, delta2=delta2, lambda=lambda)

    #Pick out only the needed terms from the OBC Hamiltonian. Reduces the max. bond dimension to 3.
    hL = hL[:,:,4:5,:]
    hM1 = hM[4:5,:,2:4,:]
    hM2 = hM[2:4,:,1:2,:]
    hR = hR[1:2,:,:,:]

    h_B = MPOTensor{T}[hL, hM1, hM2, hR]

    (ANNNI_OBC_MPO(T, hz=hz, delta1=delta1, delta2=delta2, lambda=lambda), h_B)
end

function ANNNI_Hn_MPO_split(::Type{T}, n::Integer, N::Integer; hz::Real=1.0, delta1::Real=0.0, delta2::Real=0.0, lambda::Real=1.0) where {T}
    (hL, hM, hR), (hb1, hb2, hb3, hb4) = ANNNI_PBC_MPO_split(T, hz=hz, delta1=delta1, delta2=delta2, lambda=lambda)

    Z = [1.0 0.0; 0.0 -1.0]

    hL[1,:,1,:] *= cis(n*2π/N)
    hL[1,:,2,:] *= cis(n*1.5*2π/N)
    hL[1,:,3,:] = -delta2 * cis(n*(1.5)*2π/N) * Z - hz * cis(n*2*2π/N) * I
    hL[1,:,4,:] *= cis(n*2*2π/N)

    get_hM = (j::Integer)->begin
        hM_j = copy(hM)
        hM_j[5,:,2,:] *= cis(n*(j+0.5)*2π/N)
        hM_j[5,:,3,:] = -delta2 * cis(n*(j+0.5)*2π/N) * Z - hz * cis(n*(j+1)*2π/N) * I
        hM_j[5,:,4,:] *= cis(n*(j+1)*2π/N)
        hM_j
    end

    Hn_OBC = MPOTensor{T}[hL, (get_hM(n) for n in 2:N-1)..., hR]

    #Boundary MPO tensor for site N-1
    hb1[1,:,1,:] *= cis(n*N*2π/N)

    #Boundary tensor for site N
    hb2[2,:,1,:] *= cis(n*0.5*2π/N)
    hb2[2,:,2,:] = -delta2 * cis(n*0.5*2π/N) * Z
    hb2[2,:,3,:] *= cis(n*1*2π/N)

    Hn_b = MPOTensor{T}[hb1, hb2, hb3, hb4]

    n, (Hn_OBC, Hn_b)
end

#------------------------------

function OBF_local_MPO(::Type{T}; lambda1::Real=1.0, lambda2::Real=1.0,
                            lambdaC::Real=0.0)::MPO_open{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]
    Y = [0.0im -1.0im; 1.0im 0.0im]

    Tin = (lambdaC == 0.0 ? Float64 : ComplexF64)
    d = (lambdaC == 0.0 ? 2 : 3)

    h1 = zeros(Tin, 2,2,1,3)
    h2 = zeros(Tin, 2,2,3,d)
    h3 = zeros(Tin, 2,2,d,1)

    h1[:,:,1,1] = lambda2*Z
    h1[:,:,1,2] = lambda2*X - lambda1*E
    h1[:,:,1,3] = E

    h2[:,:,1,1] = Z
    h2[:,:,2,2] = Z
    h2[:,:,3,1] = -lambda1*E

    h3[:,:,1,1] = X
    h3[:,:,2,1] = Z

    if lambdaC != 0.0
        h2[:,:,3,2] = lambdaC * Y
        h2[:,:,3,3] = -lambdaC * Z

        h3[:,:,3,1] = Y
    end

    h1 = permutedims(h1, (3,2,4,1)) #[m1,ket,m2,bra]
    h2 = permutedims(h2, (3,2,4,1)) #[m1,ket,m2,bra]
    h3 = permutedims(h3, (3,2,4,1)) #[m1,ket,m2,bra]

    h1 = convert(MPOTensor{T}, h1)
    h2 = convert(MPOTensor{T}, h2)
    h3 = convert(MPOTensor{T}, h3)

    MPOTensor{T}[h1, h2, h3]
end

function OBF_OBC_MPO(::Type{T}; lambda1::Real=1.0, lambda2::Real=1.0,
                        lambdaC::Real=0.0)::MPO_open_uniform{T} where {T}
    E = Matrix{Float64}(I,2,2)
    X = [0.0 1.0; 1.0 0.0]
    Z = [1.0 0.0; 0.0 -1.0]
    Y = [0.0im -1.0im; 1.0im 0.0im]

    Tin = (lambdaC == 0.0 ? Float64 : ComplexF64)
    hM = zeros(Tin, 2,2,6,6)

    hM[:,:,1,1] = E
    hM[:,:,2,1] = X
    hM[:,:,3,1] = Z
    hM[:,:,4,2] = lambda2*Z #Z
    hM[:,:,5,3] = Z
    hM[:,:,6,1] = -lambda1*X
    hM[:,:,6,3] = -lambda1*Z
    hM[:,:,6,4] = Z #lambda2*Z
    hM[:,:,6,5] = lambda2*X
    hM[:,:,6,6] = E

    if lambdaC != 0.0
        hM[:,:,4,1] = -lambdaC*Y
        hM[:,:,6,3] += lambdaC*Y
    end

    hL = zeros(Tin, 2,2,1,6)
    hR = zeros(Tin, 2,2,6,1)

    hL[:,:,1,:] = hM[:,:,6,:]
    hR[:,:,:,1] = hM[:,:,:,1]

    hM = permutedims(hM, (3,2,4,1)) #[m1,ket,m2,bra]
    hL = permutedims(hL, (3,2,4,1)) #[m1,ket,m2,bra]
    hR = permutedims(hR, (3,2,4,1)) #[m1,ket,m2,bra]

    hM = convert(MPOTensor{T}, hM)
    hL = convert(MPOTensor{T}, hL)
    hR = convert(MPOTensor{T}, hR)

    (hL, hM, hR)
end

function OBF_PBC_MPO_split(::Type{T}; lambda1::Real=1.0, lambda2::Real=1.0,
                                lambdaC::Real=0.0)::MPO_PBC_uniform_split{T} where {T}
    hL, hM, hR = OBF_OBC_MPO(T, lambda1=lambda1, lambda2=lambda2, lambdaC=lambdaC)

    #Pick out only the needed terms from the OBC Hamiltonian. Reduces the max. bond dimension to 3.
    hL = hL[:,:,4:6,:]
    hM1 = hM[4:6,:,2:5,:]
    hM2 = hM[2:5,:,1:3,:]
    hR = hR[1:3,:,:,:]

    h_B = MPOTensor{T}[hL, hM1, hM2, hR]

    (OBF_OBC_MPO(T, lambda1=lambda1, lambda2=lambda2, lambdaC=lambdaC), h_B)
end

function OBF_Hn_MPO_split(::Type{T}, n::Integer, N::Integer; lambda1::Real=1.0, lambda2::Real=1.0,
    ZZXoff::Number=5/4, XZZoff::Number=3/4) where {T}
    (hL, hM, hR), (hb1, hb2, hb3, hb4) = OBF_PBC_MPO_split(T, lambda1=lambda1, lambda2=lambda2)

    Z = [1.0 0.0; 0.0 -1.0]

    hL[1,:,1,:] *= cis(n*2π/N)
    hL[1,:,3,:] *= cis(n*1.5*2π/N)

    #hL[1,:,4,:] *= cis(n*(1 + ZZXoff)*2π/N) #ZZX
    hL[1,:,5,:] *= cis(n*(1 + XZZoff)*2π/N) #XZZ

    get_hM = (j::Integer)->begin
        hM_j = copy(hM)
        hM_j[6,:,1,:] *= cis(n*j*2π/N)
        hM_j[6,:,3,:] *= cis(n*(j+0.5)*2π/N)
        #hM_j[6,:,4,:] *= cis(n*(j+ZZXoff)*2π/N) #ZZX
        hM_j[4,:,2,:] *= cis(n*(j-1+ZZXoff)*2π/N) #ZZX
        hM_j[6,:,5,:] *= cis(n*(j+XZZoff)*2π/N) #XZZ
        hM_j
    end

    Hn_OBC = MPOTensor{T}[hL, (get_hM(n) for n in 2:N-1)..., hR]

    #Boundary tensor for site N-1
    #hb1[1,:,1,:] *= cis(n*(N-1 + ZZXoff)*2π/N) #ZXX
    hb1[1,:,2,:] *= cis(n*(N-1 + XZZoff)*2π/N) #XXZ (N-1)

    #Boundary tensor for site N
    hb2[3,:,2,:] *= cis(n*0.5*2π/N)
    #hb2[3,:,3,:] *= cis(n*ZZXoff*2π/N) #ZZX
    hb2[1,:,1,:] *= cis(n*(N-1 + ZZXoff)*2π/N) #ZZX (N-1)
    hb2[3,:,4,:] *= cis(n*XZZoff*2π/N) #XXZ (N)

    #Boundary tensor for site 1
    hb3[3,:,2,:] *= cis(n*(ZZXoff)*2π/N) #ZZX (N)

    Hn_b = MPOTensor{T}[hb1, hb2, hb3, hb4]

    n, (Hn_OBC, Hn_b)
end

#------------------------------

function weylops(p::Integer)
    om = cis(2π / p)
    U = diagm(ComplexF64[om^j for j in 0:p-1])
    V = diagm(ones(p - 1), 1)
    V[end, 1] = 1
    U, V, om
end

function potts3_OBC_MPO(::Type{T}; h::Real=1.0) where {T}
    E = Matrix{ComplexF64}(I,3,3)
    U, V, om = weylops(3)

    hM = zeros(ComplexF64, 3,3,4,4)

    hM[:,:,1,1] = E
    hM[:,:,2,1] = U
    hM[:,:,3,1] = U'
    hM[:,:,4,1] = -h/2 * (V+V')
    hM[:,:,4,2] = -0.5*U'
    hM[:,:,4,3] = -0.5*U
    hM[:,:,4,4] = E

    hL = zeros(ComplexF64, 3,3,1,4)
    hR = zeros(ComplexF64, 3,3,4,1)

    hL[:,:,1,:] = hM[:,:,4,:]
    hR[:,:,:,1] = hM[:,:,:,1]

    hM = permutedims(hM, (3,2,4,1)) #[m1,ket,m2,bra]
    hL = permutedims(hL, (3,2,4,1)) #[m1,ket,m2,bra]
    hR = permutedims(hR, (3,2,4,1)) #[m1,ket,m2,bra]

    hM = convert(MPOTensor{T}, hM)
    hL = convert(MPOTensor{T}, hL)
    hR = convert(MPOTensor{T}, hR)

    (hL, hM, hR)
end

function potts3_local_MPO(::Type{T}; h::Real=1.0) where {T}
    hL, hM, hR = potts3_OBC_MPO(T, h=h)

    MPOTensor{T}[hL[:,:,1:3,:], hR[1:3,:,:,:]]
end

function potts3_PBC_MPO_split(::Type{T}; h::Real=1.0)::MPO_PBC_uniform_split{T} where {T}
    hL, hM, hR = potts3_OBC_MPO(T, h=h)

    h_B = MPOTensor{T}[hL[:,:,2:3,:], hR[2:3,:,:,:]]

    ((hL, hM, hR), h_B)
end

function potts3_Hn_MPO_split(::Type{T}, n::Integer, N::Integer; h::Real=1.0) where {T}
    (hL, hM, hR), (hb1, hb2) = potts3_PBC_MPO_split(T, h=h)

    hL[1,:,1,:] *= cis(n*2π/N)
    hL[1,:,2,:] *= cis(n*1.5*2π/N)
    hL[1,:,3,:] *= cis(n*1.5*2π/N)

    hR[4,:,1,:] *= cis(n*N*2π/N)

    get_hM = (j::Integer)->begin
        hM_j = copy(hM)
        hM_j[4,:,1,:] *= cis(n*j*2π/N)
        hM_j[4,:,2,:] *= cis(n*(j+0.5)*2π/N)
        hM_j[4,:,3,:] *= cis(n*(j+0.5)*2π/N)
        hM_j
    end

    Hn_OBC = MPOTensor{T}[hL, (get_hM(n) for n in 2:N-1)..., hR]

    hb1 *= cis(n*0.5*2π/N)

    Hn_b = MPOTensor{T}[hb1, hb2]

    n, (Hn_OBC, Hn_b)
end


"""
    struct GeneralSpace{k} <: ElementarySpace{k}

A GeneralSpace is finite-dimensional space over an arbitrary field `F` without
additional structure. It is thus characterized by its dimension, and whether or
not it is the dual and/or conjugate space. For a real field `F`, the space and
its conjugate are the same.
"""
struct GeneralSpace{k} <: ElementarySpace{k}
    d::Int
    dual::Bool
    conj::Bool
    GeneralSpace{k}(d::Int; dual::Bool = false, conj::Bool = false) where {k} =
        (d >= 0 ? new{k}(d, dual, conj) : throw(ArgumentError("Dimension of a vector space should be bigger than zero")))
end

dim(V::GeneralSpace) = V.d
Base.indices(V::GeneralSpace) = Base.OneTo(dim(V))

dual(V::GeneralSpace{k}) where {k} = GeneralSpace{k}(V.d, !V.dual, V.conj)
Base.conj(V::GeneralSpace{k}) where {k} = GeneralSpace{k}(V.d, V.dual, !V.conj)

function Base.show(io::IO, V::GeneralSpace{k}) where {k}
    if V.conj
        print(io, "conj(")
    end
    print(io, "GeneralSpace{", k, "}(", V.d, ")")
    if V.dual
        print(io, "'")
    end
    if V.conj
        print(io, ")")
    end
end


using FixedPointAcceleration
func(x) = cos.(x)
Inputs = 1.1
# We should not have any slowdown from starting in simple for two. Then SEA for 3. Then Simple. Then VEA until the end.
# The reason is that SEA/VEA does nothing of interest until 7 by default anyway.
fp_chain      = fixed_point(func, Inputs; Algorithm = :Simple, MaxIter = Integer(2))
fp_chain      = fixed_point(func, fp_chain; Algorithm = :SEA, MaxIter = Integer(3))
fp_chain      = fixed_point(func, fp_chain; Algorithm = :Simple, MaxIter = Integer(1))
fp_chain      = fixed_point(func, fp_chain; Algorithm = :VEA, MaxIter = Integer(100))

fp_nochain = fixed_point(func, Inputs; Algorithm = :VEA, MaxIter = Integer(100))
fp_chain.Iterations_ == fp_nochain.Iterations_
all(abs.(fp_nochain.Inputs_ .- fp_chain.Inputs_) .< 1e-14)

func(x) = [0.5*sqrt(abs(x[1] + x[2])), 1.5*x[1] + 0.5*x[2]]
Inputs = [1.1,2.2]

fp_chain      = fixed_point(func, Inputs; Algorithm = :Simple, MaxIter = Integer(2))
fp_chain      = fixed_point(func, fp_chain; Algorithm = :MPE, MaxIter = Integer(3))
fp_chain      = fixed_point(func, fp_chain; Algorithm = :Simple, MaxIter = Integer(1))
fp_chain      = fixed_point(func, fp_chain; Algorithm = :RRE, MaxIter = Integer(100))

fp_nochain = fixed_point(func, Inputs; Algorithm = :RRE, MaxIter = Integer(100))
fp_chain.Iterations_ == fp_nochain.Iterations_
all(abs.(fp_nochain.Inputs_ .- fp_chain.Inputs_) .< 1e-14)


function pointstoforces(
    edges::Array{GeometricalPredicates.Line2D{GeometricalPredicates.Point2D},1},
    scaler::Scaler,
    fscale::Float64,
    /pfix,
    fh::Function,
)::Dict{Point2D,Array{Float64,1}}
    bars = Array{Point2D,1}()
    barvec = Array{Array{Float64,1},1}()
    pointstofvec = Dict{Point2D,Array{Float64,1}}()
    for edge in edges
        b = unscaledpoint2d(getb(edge), scaler)
        a = unscaledpoint2d(geta(edge), scaler)
        push!(bars, barcenter(a, b))
        push!(barvec, barvector(a, b))
        push!(pointstofvec, geta(edge) => [0, 0])
        push!(pointstofvec, getb(edge) => [0, 0])
    end
    fvec = forcevectors(bars, barvec, fscale, fh)
    return pointstoforces(edges, pointstofvec, fvec, pfix)
end

function barcenter(a::Point2D, b::Point2D)::Point2D
    return Point(getx(a) + ((getx(b) - getx(a)) / 2), gety(a) + ((gety(b) - gety(a)) / 2))
end

function barvector(a::Point2D, b::Point2D)::Array{Float64, 1}
    return [getx(b) - getx(a), gety(b) - gety(a)]
end

function forcevectors(
    bars::Array{Point2D,1},
    barvec::Array{Array{Float64,1},1},
    fscale::Float64,
    fh::Function,
)
    L = [sqrt(sum(vsum .^ 2)) for vsum in barvec]
    hbars = [fh(getx(p), gety(p)) for p in bars]
    L0 = hbars * fscale * sqrt(sum(L .^ 2) / sum(hbars .^ 2))
    fvec = maximum.(L0 - L) ./ L .* barvec
end

function pointstoforces(
    edges::Array{GeometricalPredicates.Line2D{GeometricalPredicates.Point2D},1},
    pointstofvec::Dict{Point2D,Array{Float64,1}},
    fvec,
    pfix
)::Dict{Point2D,Array{Float64,1}}
    iterator = 1
    for edge in edges
        a = geta(edge)
        b = getb(edge)
        push!(pointstofvec, a => pointstofvec[a] + (-fvec[iterator]))
        push!(pointstofvec, b => pointstofvec[b] + (fvec[iterator]))
        iterator += 1
    end
    for p in eachrow(pfix)
        p = Point(p[1], p[2])
        delete!(pointstofvec, p)
        push!(pointstofvec, p => [0, 0])
    end
    return pointstofvec
end


using LinearAlgebra

useVerbose  = [true,true,false]

macro verbose(level::Integer, string::String)
    @assert(0 < level < 4)
    if level > 0; padding = ""  ; end
    if level > 1; padding = "--"; end
    if level > 2; padding = "--"; end
    string = padding * string
    if useVerbose[level]
        return :( println($string) )
    end
end

function get_matrix_and_copy(in)
    return in, deepcopy(in), matrix_double_t(in)
end

function skew(u)
    S = zeros(3,3)
    S[1,2] = -u[3];
    S[1,3] = +u[2];
    S[2,1] = +u[3];
    S[2,3] = -u[1];
    S[3,1] = -u[2];
    S[3,2] = +u[1];
    return S;
end

function SO3_hat(u)
    return skew(u);
end

function SO3_vee(gfrak)
    out = zeros(3,1);
    out[1] = gfrak[3,2];
    out[2] = gfrak[1,3];
    out[3] = gfrak[2,1];
    return out;
end

function SU2_hat(u)
    L1 = [ 0.0   im;  im  0.0];
    L2 = [ 0.0 -1.0; 1.0  0.0];
    L3 = [  im  0.0; 0.0  -im];
    gfrak = L1*u[1] + L2*u[2] + L3*u[3];
    return gfrak
end

function SU2_vee(gfrak)
    out = [ imag(gfrak[1,2] + gfrak[2,1])/2.0;
            real(gfrak[2,1] - gfrak[1,2])/2.0;
            imag(gfrak[1,1] - gfrak[2,2])/2.0];
    return out
end

function quat_2_SU2(qX::Array{Float64, 2})
    X = zeros(2,2) + zeros(2,2)*im;
    X[1,1] = +qX[1]+qX[4]*im
    X[1,2] = -qX[3]+qX[2]*im
    X[2,1] = +qX[3]+qX[2]*im
    X[2,2] = +qX[1]-qX[4]*im
    return X
end

function SU2_2_quat(X::Array{Complex{Float64}, 2})
    q = zeros(4,1);
    q[1,1] = real(X[1,1])
    q[2,1] = imag(X[2,1])
    q[3,1] = real(X[2,1])
    q[4,1] = imag(X[1,1])
    return q
end

function quat_2_SO3(q::Array{Float64, 2})
    a, b, c, d = q[1,1], q[2,1], q[3,1], q[4,1];
    R = zeros(3,3);
    R[1,1] = a^2+b^2-c^2-d^2;
    R[1,2] = 2*(b*c-a*d);
    R[1,3] = 2*(b*d+a*c);
    R[2,1] = 2*(b*c+a*d);
    R[2,2] = a^2-b^2+c^2-d^2;
    R[2,3] = 2*(c*d-a*b);
    R[3,1] = 2*(b*d-a*c);
    R[3,2] = 2*(c*d+a*b);
    R[3,3] = a^2-b^2-c^2+d^2;
    return R
end

function quaternion_left_product(q)
    qim = q[2:4]
    qre = q[1]
    QL = zeros(4,4)
    QL[1,2:4]   = -qim;
    QL[2:4,1]   = +qim;
    QL[2:4,2:4] = skew(qim);
    QL = QL + I*qre;
    return QL
end

function quaternion_right_product(q)
    qim = q[2:4]
    qre = q[1]
    QR = zeros(4,4);
    QR[1,2:4]   = -qim;
    QR[2:4,1]   = +qim;
    QR[2:4,2:4] = -skew(qim);
    QR = QR + I*qre;
    return QR
end

function attitude_dynamics(q, w, J, tau; delta = zeros(3,))
    we      = zeros(4,1);
    we[2:4] = w;
    dq      = quaternion_left_product(q) * we / 2.0;
    dw      = (J) \ (-skew(w)*(J*w) + tau + delta);
    return dq, dw
end

function ntuple_2_mat(tup, Nrow, Ncol)
    mat = zeros(Nrow, Ncol)
    for i = 1:Nrow
        for j = 1:Ncol
            mat[i,j] = tup[i][j];
        end
    end
    return mat;
end

function rand_PSD(N)
    J  = rand(N,N); J  = J + J'; J  = J + I*maximum(abs.(eigvals(J)));
    return J;
end

function maximal_feasible_kc_SU2(kX, kw, J)
    ev   = eigvals(J);
    maxJ = maximum(ev);
    minJ = minimum(ev);
    bounds = [4.0 * kw, 2.0 * sqrt(kw * minJ), 4.0 * kw * kX * minJ * minJ / (maxJ * kw * kw + minJ * minJ * kX)]
    return minimum(bounds);
end

function maximal_feasible_kc_SO3(kR, kw, J)
    ev   = eigvals(J);
    maxJ = maximum(ev);
    minJ = minimum(ev);
    bounds = [kw, sqrt(kR * minJ), 4.0 * kw * kR * minJ * minJ / (maxJ * kw * kw + 4.0 * minJ * minJ * kR)]
    return minimum(bounds);
end

function allFieldsEqual(
    C1::con_state_qw_fsf_t,
    C2::con_state_qw_fsf_t,
    checkStatic::Bool
)
    @test isapprox(C1.gain_kR, C2.gain_kR);
    @test isapprox(C1.gain_kc, C2.gain_kc);
    @test isapprox(C1.gain_kw, C2.gain_kw);
    @test isapprox(C1.param_a, C2.param_a);
    @test isapprox(C1.param_b, C2.param_b);
    @test isapprox(C1.param_c, C2.param_c);
    @test isapprox(C1.param_d, C2.param_d);
    @test isapprox(C1.thrust, C2.thrust);
    for i = 1:3
        for j = 1:3
            @test isapprox(C1.inertia[i][j], C2.inertia[i][j]);
        end
    end
    if checkStatic
        for i = 1:3
            @test isapprox(C1.torques[i],  C2.torques[i] );
        end
        @test isapprox(C1.dist_Psi,       C2.dist_Psi );
        @test isapprox(C1.dist_Gamma,     C2.dist_Gamma );
        @test isapprox(C1.dist_lyapunov,  C2.dist_lyapunov );
    end
end

function allFieldsEqual(
    R1::ref_state_qw_t,
    R2::ref_state_qw_t,
)
    for i = 1:4
        @test  isapprox(R1.quaternion[i], R2.quaternion[i]);
    end
    for i = 1:3
        @test  isapprox(R1.attrates[i], R2.attrates[i]);
        @test  isapprox(R1.attaccel[i], R2.attaccel[i] );
    end
end

function allFieldsEqual(
    S1::dyn_state_qw_t,
    S2::dyn_state_qw_t,
)
    for i = 1:4
        @test  isapprox(S1.quaternion[i], S2.quaternion[i]);
    end
    for i = 1:3
        @test  isapprox(S1.attrates[i], S2.attrates[i]);
    end
end

function get_M1M2W_SO3(C, phi)
    # Extract parameters
    kR, kc, kw  = C.gain_kR, C.gain_kc, C.gain_kw;
    J           = ntuple_2_mat(C.inertia, 3, 3);
    minJ, maxJ  = minimum(eigvals(J)), maximum(eigvals(J));

    # Evaluate matrices
    M1 = [[+kR, -kc] [-kc, +minJ]] / 2.0;
    M2 = [[2*kR/(2 - phi), kc] [kc, maxJ]] / 2.0;
    W  = [[+kR*kc/maxJ, -kc*kw/(2 * minJ)] [-kc*kw/(2 * minJ), kw - kc]];
    return M1, M2, W
end

function get_M1M2W_SU2(C, phi)
    # Extract parameters
    kX, kc, kw  = C.gain_kR, C.gain_kc, C.gain_kw;
    J           = ntuple_2_mat(C.inertia, 3, 3);
    minJ, maxJ  = minimum(eigvals(J)), maximum(eigvals(J));

    # Evaluate matrices
    M1 = [[+4.0*kX, -kc] [-kc, +minJ]] / 2.0;
    M2 = [[+8.0*kX/(2.0 - phi), kc] [kc, maxJ]]/2.0;
    W  = [[+kX*kc/maxJ, -kc*kw/(2 * minJ)] [-kc*kw/(2 * minJ), kw - kc/4.0]];
    return M1, M2, W
end

function numerical_differentiation(in, time)
    NT  = size(in, 2);
    out = zeros(size(in));
    for t = 2:NT;
        dt = time[t] - time[t - 1];
        out[:,t] = (in[:,t] - in[:,t-1])/dt;
    end
    return out
end

function maximal_feasible_eps_SO3(
    C::con_state_qw_fsf_t,
    R0::Array{T,2},
    Rr0::Array{T,2},
    w0::Array{T,2},
    wr0::Array{T,2}
) where {T<:Number}

    # Extract parameters
    kR, kc, kw  = C.gain_kR, C.gain_kc, C.gain_kw;
    J           = ntuple_2_mat(C.inertia, 3, 3);

    # Compute maximum permissible phi
    Psi         = tr(I-Rr0'*R0)/2.0;
    eR0         = SO3_vee(Rr0'*R0 - R0'*Rr0) / 2.0;
    ew0         = w0 -  R0'*Rr0*wr0;
    attpart     = kR * Psi;
    crosspart   = kc * ew0'*eR0;
    ratepart    = ew0'*J*ew0/2.0;
    Lyap0       = attpart + crosspart[1,1] + ratepart[1,1];
    phi         = Lyap0/kR;

    M1, M2, W   = get_M1M2W_SO3(C, phi)

    M1max, M1min = maximum(eigvals(M1)), minimum(eigvals(M1));
    M2max, M2min = maximum(eigvals(M2)), minimum(eigvals(M2));
    Wmax,  Wmin  = maximum(eigvals(W)),  minimum(eigvals(W));

    mindecayrate = Wmin/M2max;
    maxeps       = ((M1min*Wmin) / M2max)*phi*(2.0 - phi);
    uniboundgain = (M2max / (M1min*Wmin));

    return maxeps, mindecayrate, uniboundgain, phi, M1, M2, W
end


using Singular
using SignatureGB
R, (x1,x2,x3,x4,x5,x6,x7) = Singular.PolynomialRing(Fp(101), ["x1", "x2", "x3", "x4", "x5", "x6", "x7"])
I = spoly{n_Zp}[x1 + x2 + x3 + x4 + x5 + x6 + x7, x1*x2 + x2*x3 + x3*x4 + x4*x5 + x5*x6 + x1*x7 + x6*x7, x1*x2*x3 + x2*x3*x4 + x3*x4*x5 + x4*x5*x6 + x1*x2*x7 + x1*x6*x7 + x5*x6*x7, x1*x2*x3*x4 + x2*x3*x4*x5 + x3*x4*x5*x6 + x1*x2*x3*x7 + x1*x2*x6*x7 + x1*x5*x6*x7 + x4*x5*x6*x7, x1*x2*x3*x4*x5 + x2*x3*x4*x5*x6 + x1*x2*x3*x4*x7 + x1*x2*x3*x6*x7 + x1*x2*x5*x6*x7 + x1*x4*x5*x6*x7 + x3*x4*x5*x6*x7, x1*x2*x3*x4*x5*x6 + x1*x2*x3*x4*x5*x7 + x1*x2*x3*x4*x6*x7 + x1*x2*x3*x5*x6*x7 + x1*x2*x4*x5*x6*x7 + x1*x3*x4*x5*x6*x7 + x2*x3*x4*x5*x6*x7, x1*x2*x3*x4*x5*x6*x7 - 1]
comp_ideal = gens(R)
SignatureGB.f5(comp_ideal)
gb = SignatureGB.f5(I, verbose = true)
println(SignatureGB.is_gb(gb))


@testset "Results" begin
    # Inline definition of the Sphere function
    f_sphere(x) = sum(x.^2)

    RANDOM_SEED = 6918234
    rng = MersenneTwister(RANDOM_SEED)

    @testset "Benchmarks" begin
        total_iterations = 20000
        val = PSO(
            Easom(),
            Population(35, 2, -100.0, 100.0, rng),
            total_iterations,
            rng
        )
        ground_truth = @SVector [π, π]
        @test isapprox(val.x, ground_truth, atol = 1e-8)
        @test isapprox(val.min, -1.0)
        @test val.impl ≡ "PSO"
        @test val.iterations ≡ total_iterations
    end

    @testset "Functions" begin
        total_iterations = 20000
        val = PSO(
            f_sphere,
            Population(35, 3, -100.0, 100.0, rng),
            total_iterations,
            rng
        )
        @test isapprox(val.x, zeros(3), atol=eps())
        @test isapprox(val.min, 0.0, atol=eps())
        @test val.impl ≡ "PSO"
        @test val.iterations ≡ total_iterations
        optim_res = OptimizationResults(val.x,
                f_sphere(val.x),
                "PSO",
                val.iterations)
        @show optim_res
    end
end


using SNOPT7
using SparseArrays

function objfun(mode_::Ptr{Cint}, n_::Ptr{Cint}, x_::Ptr{Float64},
                f_::Ptr{Float64}, g_::Ptr{Float64},
                status::Ptr{Cint})
    mode = unsafe_load(mode_)
    n    = unsafe_load(n_)

    x    = unsafe_wrap(Array, x_, Int(n))

    if mode == 0 || mode == 2
#        obj = (x[1] - 2.0)^2 + x[2]^2
        obj = x[1] + x[2]
        unsafe_store!(f_,obj)
    end

    if mode == 1 || mode == 2
        g = unsafe_wrap(Array,g_, Int(n))
#        g[1] = 2.0*(x[1] - 2.0)
#        g[2] = 2.0*x[2]
        g[1] = 1.0
        g[2] = 1.0
    end
    return
end

function confun(mode_::Ptr{Cint},
                nncon_::Ptr{Cint}, nnjac_::Ptr{Cint}, negcon_::Ptr{Cint},
                x_::Ptr{Float64}, c_::Ptr{Float64}, J_::Ptr{Float64},
                status::Ptr{Cint})
    mode   = unsafe_load(mode_)
    nncon  = unsafe_load(nncon_)
    nnjac  = unsafe_load(nnjac_)
    negcon = unsafe_load(negcon_)

    x    = unsafe_wrap(Array, x_, Int(nnjac))
    if mode == 0 || mode == 2
        c = unsafe_wrap(Array, c_, Int(nncon))
        c[1] = x[1]^3 - x[2] #(1.0 - x[1])^3  - x[2]

    end

    if mode == 1 || mode == 2
        J = unsafe_wrap(Array,J_, Int(negcon))
        J[1] = 3.0*x[1]^2 #-3.0*(1-x[1])^2
        J[2] = -1.0

    end
    return

end

function eval_f(x)
    return (x[1] - 2.0)^2 + x[2]^2
end

function eval_grad_f(x, grad)
    grad[1] = 2.0*(x[1] - 2.0)
    grad[2] = 2.0*x[2]
end

function eval_g(x,c)
    c[1] = (1.0 - x[1])^3  - x[2]
end

function eval_jac_g(x,J)
    J[1] = -3.0*(1-x[1])^2
    J[2] = -1.0
end

inf  = 1.0e20
n    = 2
m    = 1
iObj = 0

hs    = zeros(Int,n+m)

#bl = [  0., 0., 0. ]
#bu = [  inf, inf, inf ]
#x  = [ -2. , -2., 0. ]
#x  = [ 1. , 0., 0. ]

bl = [ -inf, 1., 0.0 ]
bu = [  inf, inf, inf ]
x  = [ 0. , 3., 1. ]

nnCon = 1
nnJac = 2
nnObj = 2

J = [ -100 -1.0 ]
J = sparse(J)

prob = initialize("hs13.out", "screen")
info = readOptions(prob, "hs13.spc")
info = snopt(prob, "cold", "hs13",
             m, n, nnCon, nnObj, nnJac, 0., iObj,
             #eval_f, eval_grad_f, eval_g, eval_jac_g,
             confun, objfun,
             J, bl, bu, hs, x)

println(x)
println(prob.x)
println(prob.obj_val)
println(prob.status)


##
using CSV, DataFrames, DataFramesMeta, StatsPlots, Dates, DelimitedFiles
cd(@__DIR__)
# download and unzip the data file, from
# https://realized.oxford-man.ox.ac.uk/images/oxfordmanrealizedvolatilityindices.zip

## read the data into a DataFrame
data = CSV.read("oxfordmanrealizedvolatilityindices.csv",DataFrame)

## select the desired columns
data = data[:,[:Column1, :Symbol, :close_price, :rv10, :bv]]
# set desired names
rename!(data, :Column1 => :Date)
rename!(data, :rv10 => :rv)
# select desired index
sp500 = @subset(data, :Symbol .== ".SPX")

## drop times from date by taking the leading 10 characters (yyyy-mm-dd)
sp500.Date = [sp500.Date[i][1:10] for i = 1:size(sp500,1)]

## compute percentage returns
sp500.rets = vcat(0.0,[100.0 *(log(sp500.close_price[i]) - log(sp500.close_price[i-1])) for i=2:size(sp500,1)])
# scale volatility measures to be compatible with percentage returns (log price mult. by 100)
sp500.rv = 10000.0 .* sp500.rv
sp500.bv = 10000.0 .* sp500.bv

## select the date range
sp500 = @subset(sp500, :Date .>= "2013-12-17") # these dates give 1000 obs.
sp500 = @subset(sp500, :Date .<= "2017-12-05")

## write out variables to plain text file
data = sp500[:,[:Date, :rets, :rv, :bv]]
#CSV.write("sp500.csv", data) # write as CSV

##
# some plots
plot(sp500.Date, sp500.rets, tickfontsize=5, legend=false)

##
plot(sp500.Date, [sp500.rv, sp500.bv], tickfontsize=5, label=["rv" "bv"])

##
# the next plot shows that volatility is higher when
# returns are in the tails of distribution, and that
# the overall correlation is negative
marginalkde(sp500.rets, sp500.bv)