### A Pluto.jl notebook ###
# v0.19.0

using Markdown
using InteractiveUtils

# ╔═╡ 856f336c-e232-4f1f-b1ac-759b4558acd1
using Manifolds, Manopt, Random, LinearAlgebra

# ╔═╡ b0769dfa-28cf-440e-9ba2-1ef488f171a9
using FiniteDiff, ReverseDiff

# ╔═╡ 0213d26a-18ac-11ec-03fd-ada5992bcea8
md"""
# Using (Euclidean) AD in Manopt.jl
"""

# ╔═╡ f3bc91ee-5871-4cba-ac89-190deb71ad0f
md"""
Since [Manifolds.jl](https://juliamanifolds.github.io/Manifolds.jl/latest/) 0.7 the support of automatic differentiation support has been extended.

This tutorial explains how to use Euclidean tools to derive a gradient for a real-valued function ``F\colon \mathcal M → ℝ``. We will consider two methods: an intrinsic variant and a variant employing the embedding. These gradients can then be used within any gradient based optimisation algorithm in [Manopt.jl](https://manoptjl.org).

While by default we use [FiniteDifferences.jl](https://juliadiff.org/FiniteDifferences.jl/latest/), you can also use [FiniteDiff.jl](https://github.com/JuliaDiff/FiniteDiff.jl), [ForwardDiff.jl](https://juliadiff.org/ForwardDiff.jl/stable/), [ReverseDiff.jl](https://juliadiff.org/ReverseDiff.jl/), or  [Zygote.jl](https://fluxml.ai/Zygote.jl/).
"""

# ╔═╡ d9be6c2f-65fd-4685-9005-da22bf985e28
md"""
In this Notebook we will take a look at a few possibilities to approximate or derive the gradient of a function ``f:\mathcal M \to ℝ`` on a Riemannian manifold, without computing it yourself. There is mainly two different philosophies:

1. Working _instrinsically_, i.e. stay on the manifold and in the tangent spaces. Here, we will consider approximating the gradient by forward differences.

2. Working in an embedding – there we can use all tools from functions on Euclidean spaces – finite differences or automatic differenciation – and then compute the corresponding Riemannian gradient from there.

Let's first load all packages we need.
"""

# ╔═╡ 18d7459f-eed6-489b-a096-ac77ccd781af
md"""
## 1. (Intrinsic) Forward Differences

A first idea is to generalise (multivariate) finite differences to Riemannian manifolds. Let ``X_1,\ldots,X_d ∈ T_p\mathcal M`` denote an orthonormal basis of the tangent space ``T_p\mathcal M`` at the point ``p∈\mathcal M`` on the Riemannian manifold.

We can generalise the notion of a directional derivative, i.e. for the “direction” ``Y∈T_p\mathcal M`` let ``c\colon [-ε,ε]``, ``ε>0``, be a curve with ``c(0) = p``, ``\dot c(0) = Y`` and we obtain

```math
	Df(p)[Y] = \frac{\mathrm{d}}{\mathrm{d}t} f(c(t)) = \lim_{h \to 0} \frac{1}{h}(f(\exp_p(hY))-f(p))
```

We can approximate ``Df(p)[X]`` by a finite difference scheme for an ``h>0`` as

```math
DF(p)[Y] ≈ G_h(Y) := \frac{1}{h}(f(\exp_p(hY))-f(p))
```

Furthermore the gradient ``\operatorname{grad}f`` is the Riesz representer of the differential, ie.

```math
	Df(p)[Y] = g_p(\operatorname{grad}f(p), Y),\qquad \text{ for all } Y ∈ T_p\mathcal M
```

and since it is a tangent vector, we can write it in terms of a basis as

```math
	\operatorname{grad}f(p) = \sum_{i=1}^{d} g_p(\operatorname{grad}f(p),X_i)X_i
	= \sum_{i=1}^{d} Df(p)[X_i]X_i
```

and perform the approximation from above to obtain
```math
	\operatorname{grad}f(p) ≈ \sum_{i=1}^{d} G_h(X_i)X_i
```
for some suitable step size ``h``.This comes at the cost of ``d+1`` function evaluations and ``d`` exponential maps.
"""

# ╔═╡ a3df142e-94df-48d2-be08-d1f1f3854c76
md"""
This is the first variant we can use. An advantage is, that it is _intrinsic_ in the sense that it does not require any embedding of the manifold.
"""

# ╔═╡ 9a030ac6-1f44-4fa6-8bc9-1c0278e97fe2
md""" ### An Example: The Rayleigh Quotient

The Rayleigh quotient is concerned with finding Eigenvalues (and Eigenvectors) of a symmetric matrix $A\in ℝ^{(n+1)×(n+1)}$. The optimisation problem reads

```math
F\colon ℝ^{n+1} \to ℝ,\quad F(\mathbf x) = \frac{\mathbf x^\mathrm{T}A\mathbf x}{\mathbf x^\mathrm{T}\mathbf x}
```

Minimizing this function yields the smallest eigenvalue ``\lambda_1`` as a value and the corresponding minimizer ``\mathbf x^*`` is a corresponding eigenvector.

Since the length of an eigenvector is irrelevant, there is an ambiguity in the cost function. It can be better phrased on the sphere ``𝕊^n`` of unit vectors in ``\mathbb R^{n+1}``, i.e.

```math
\operatorname*{arg\,min}_{p \in 𝕊^n} f(p) = \operatorname*{arg\,min}_{p \in 𝕊^n} p^\mathrm{T}Ap
```

We can compute the Riemannian gradient exactly as

```math
\operatorname{grad} f(p) = 2(Ap - pp^\mathrm{T}Ap)
```

so we can compare it to the approximation by finite differences.
"""

# ╔═╡ 19747159-d383-4547-9315-0ed2494904a6
begin
    Random.seed!(42)
    n = 200
    A = randn(n + 1, n + 1)
    A = Symmetric(A)
    M = Sphere(n)
    nothing
end

# ╔═╡ 41c204dd-6e4e-4a70-8f06-209a469e0680
f1(p) = p' * A'p

# ╔═╡ 2e33de5e-ffaa-422a-91d9-61f588ed1211
gradf1(p) = 2 * (A * p - p * p' * A * p)

# ╔═╡ bbd9a010-1981-45b3-bf7d-c04bcd2c2128
md"""Manifolds provides a finite difference scheme in Tangent spaces, that you can introduce to use an existing framework (if the wrapper is implemented) form Euclidean space. Here we use `FiniteDiff.jl`."""

# ╔═╡ 08456b40-74ec-4319-93e7-130b5cf70ac3
r_backend = Manifolds.TangentDiffBackend(Manifolds.FiniteDiffBackend())

# ╔═╡ 12327b62-7e79-4381-b6a7-f85b08a8251b
gradf1_FD(p) = Manifolds.gradient(M, f1, p, r_backend)

# ╔═╡ 07f9a630-e53d-45ea-b109-3d4de190723d
begin
    p = zeros(n + 1)
    p[1] = 1.0
    X1 = gradf1(p)
    X2 = gradf1_FD(p)
    norm(M, p, X1 - X2)
end

# ╔═╡ 8e5f677d-dafa-49b9-b678-3f129be31dcf
md"We obtain quite a good approximation of the gradient."

# ╔═╡ 77769eab-54dd-41dc-8125-0382e5ef0bf1
md"""
## 2. Conversion of an Euclidean Gradient in the Embedding to a Riemannian Gradient of an (not necessarily isometrically) embedded Manifold

Let ``\tilde f\colon\mathbb R^m \to \mathbb R`` be a function in the embedding of an ``n``-dimensional manifold ``\mathcal M \subset \mathbb R^m`` and ``f\colon \mathcal M \to \mathbb R`` denote the restriction of ``\tilde f`` to the manifold ``\mathcal M``.

Since we can use the push forward of the embedding to also embed the tangent space ``T_p\mathcal M``, ``p\in \mathcal M``, we can similarly obtain the differential ``Df(p)\colon T_p\mathcal M \to \mathbb R`` by restricting the differential ``D\tilde f(p)`` to the tangent space.

If both ``T_p\mathcal M`` and ``T_p\mathcal R^m`` have the same inner product, or in other words the manifold is isometrically embedded in ``R^m`` (like for example the sphere ``\mathbb S^n\subset\mathbb R^{m+1}`` then this restriction of the differential directly translates to a projection of the gradient, i.e.

```math
\operatorname{grad}f(p) = \operatorname{Proj}_{T_p\mathcal M}(\operatorname{grad} \tilde f(p))
```

More generally we might have to take a change of the metric into account, i.e.

```math
\langle  \operatorname{Proj}_{T_p\mathcal M}(\operatorname{grad} \tilde f(p)), X \rangle
= Df(p)[X] = g_p(\operatorname{grad}f(p), X)
```

or in words: we have to change the Riesz representer of the (restricted/projected) differential of ``f`` (``\tilde f``) to the one with respect to the Riemannian metric. This is done using [`change_representer`](https://juliamanifolds.github.io/Manifolds.jl/latest/manifolds/metric.html#Manifolds.change_representer-Tuple{AbstractManifold,%20AbstractMetric,%20Any,%20Any}).
"""

# ╔═╡ 57cda07f-e432-46af-b771-5e5a3067feac
md"""
### A continued Example
We continue with the Rayleigh Quotient from before, now just starting with the defintion of the Euclidean case in the embedding, the function ``F``.
"""

# ╔═╡ c3f3aeba-2849-4715-94e2-0c44613a2ce9
F(x) = x' * A * x / (x' * x);

# ╔═╡ 786fce04-53ef-448d-9657-31208b35fb7e
md"The cost function is the same by restriction"

# ╔═╡ c1341fef-adec-4574-a642-a1a8a9c1fee5
f2(M, p) = F(p);

# ╔═╡ 0818a62f-1bef-44f7-a33f-1ab0054e853c
md"The gradient is now computed combining our gradient scheme with ReverseDiff."

# ╔═╡ 89cd6b4b-f9ef-47ac-afd3-cf9aacf43256
function grad_f2_AD(M, p)
    return Manifolds.gradient(
        M, F, p, RiemannianProjectionBackend(Manifolds.ReverseDiffBackend())
    )
end

# ╔═╡ 7c5a8a17-6f63-4587-a94a-6936bdd3cec6
X3 = grad_f2_AD(M, p)

# ╔═╡ b3e7f57f-d87a-47c5-b8ad-48b6d205fa73
norm(M, p, X1 - X3)

# ╔═╡ 893db402-283f-4e3e-8bf7-c6f22e485efb
md"""
### An Example for a nonisometrically embedded Manifold

on the manifold ``\mathcal P(3)`` of symmetric positive definite matrices.
"""

# ╔═╡ 8494a0d6-dbf2-4eb0-a555-f00e446fbe38
md"""
The following function computes (half) the distance squared (with respect to the linear affine metric) on the manifold ``\mathcal P(3)`` to the identity, i.e. $I_3$. denoting the unit matrix we consider the function

```math
	G(q) = \frac{1}{2}d^2_{\mathcal P(3)}(q,I_3) = \lVert \operatorname{Log}(q) \rVert_F^2,
```
where $\operatorname{Log}$ denotes the matrix logarithm and ``\lVert \cdot \rVert_F`` is the Frobenius norm.
This can be computed for symmetric positive definite matrices by summing the squares of the ``\log``arithms of the eigenvalues of ``q`` and divide by two:
"""

# ╔═╡ c93eb2da-89df-4751-b086-62be604d41e6
G(q) = sum(log.(eigvals(Symmetric(q))) .^ 2) / 2

# ╔═╡ e2bf6f55-7235-4d75-8bee-a325434e32ad
md"""
We can also interpret this as a function on the space of matrices and apply the Euclidean finite differences machinery; in this way we can easily derive the Euclidean gradient. But when computing the Riemannian gradient, we have to change the representer (see again [`change_representer`](https://juliamanifolds.github.io/Manifolds.jl/latest/manifolds/metric.html#Manifolds.change_representer-Tuple{AbstractManifold,%20AbstractMetric,%20Any,%20Any})) after projecting onto the tangent space ``T_p\mathcal P(n)`` at ``p``.

Let's first define a point and the manifold ``N=\mathcal P(3)``.
"""

# ╔═╡ 153378ca-703d-4a84-bc63-22347399a160
rotM(α) = [1.0 0.0 0.0; 0.0 cos(α) sin(α); 0.0 -sin(α) cos(α)]

# ╔═╡ 699f0177-2c5b-434b-9eca-b6fc573e497f
q = rotM(π / 6) * [1.0 0.0 0.0; 0.0 2.0 0.0; 0.0 0.0 3.0] * transpose(rotM(π / 6))

# ╔═╡ 9b52d05c-4cba-4922-85a4-1a2a3c74823e
N = SymmetricPositiveDefinite(3)

# ╔═╡ cb3fe7aa-1262-48f2-9ebc-3e959c72a33e
is_point(N, q)

# ╔═╡ 13712c64-48fd-4f2a-9ee4-1949e51d316f
md"""We could first just compute the gradient using `FiniteDiff.jl`, but this yields the Euclidean gradient:"""

# ╔═╡ 64beb3dd-9507-4792-be02-ae1405704690
FiniteDiff.finite_difference_gradient(G, q)

# ╔═╡ 2be4f9e8-0331-44ac-839f-7bb71d9edef9
md"""Instead, we use the [`RiemannianProjectedBackend`](https://juliamanifolds.github.io/Manifolds.jl/latest/features/differentiation.html#Manifolds.RiemannianProjectionBackend) of `Manifolds.jl`, which in this case internally uses `FiniteDiff.jl` to compute a Euclidean gradient but then uses the conversion explained above to derive the Riemannian gradient.

We define this here again as a function `grad_G_FD` that could be used in the `Manopt.jl` framework within a gradient based optimisation.
"""

# ╔═╡ 6f1d748f-27ce-496b-8561-f16972da50cc
function grad_G_FD(N, q)
    return Manifolds.gradient(
        N, G, q, RiemannianProjectionBackend(Manifolds.FiniteDiffBackend())
    )
end

# ╔═╡ 7dd656ea-08de-4172-8a92-87ad2228ce69
G1 = grad_G_FD(N, q)

# ╔═╡ 219573d2-283f-456c-a5c3-fadd734fc157
md"""
Now, we can agaon compare this to the (known) solution of the gradient, namely the gradient of (a half) the distance suqared, i.e. ``G(q) = \frac{1}{2}d^2_{\mathcal P(3)}(q,I_3)`` is given by ``\operatorname{grad} G(q) = -\operatorname{log}_q I_3``, where ``\operatorname{log}`` is the [logarithmic map](https://juliamanifolds.github.io/Manifolds.jl/latest/manifolds/symmetricpositivedefinite.html#Base.log-Tuple{SymmetricPositiveDefinite,%20Vararg{Any,%20N}%20where%20N}) on the manifold.
"""

# ╔═╡ e28a2752-877c-4ab4-a253-8d26fa9a73c2
G2 = -log(N, q, Matrix{Float64}(I, 3, 3))

# ╔═╡ 25c65878-1be6-4fec-b65e-9c1741320a41
md"""Both terms agree up to ``1.2×10^{-10}``:"""

# ╔═╡ 9a66d4f3-508d-4285-9a93-df1323575202
norm(G1 - G2)

# ╔═╡ c07fb3d0-d12f-44d7-bcab-7a0d39e6af8d
isapprox(M, q, G1, G2; atol=2 * 1e-10)

# ╔═╡ f47c70b6-ca05-498f-9e10-58c3839ca427
md""" In this case we can not use `ReverseDiff.jl`, since it can not handle the `eigvals!` function that is called internally."""

# ╔═╡ 32d8d025-3993-4d31-9eea-3463e0af1c12
md"""
## Summary

This tutorial illustrates how to use tools from Euclidean spaces, finite differences or automatic differentiation, to compute gradients on Riemannian manifolds. The scheme allows to use _any_ differentiation framework within the embedding to derive a Riemannian gradient.
"""

# ╔═╡ 00000000-0000-0000-0000-000000000001
PLUTO_PROJECT_TOML_CONTENTS = """
[deps]
FiniteDiff = "6a86dc24-6348-571c-b903-95158fe2bd41"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Manifolds = "1cead3c2-87b3-11e9-0ccd-23c62b72b94e"
Manopt = "0fc0a36d-df90-57f3-8f93-d78a9fc72bb5"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
ReverseDiff = "37e2e3b7-166d-5795-8a7a-e32c996b4267"

[compat]
FiniteDiff = "~2.8.1"
Manifolds = "~0.7.1"
Manopt = "~0.3.14"
ReverseDiff = "~1.10.0"
"""

# ╔═╡ 00000000-0000-0000-0000-000000000002
PLUTO_MANIFEST_TOML_CONTENTS = """
# This file is machine-generated - editing it directly is not advised

julia_version = "1.7.2"
manifest_format = "2.0"

[[deps.AbstractFFTs]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "485ee0867925449198280d4af84bdb46a2a404d0"
uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
version = "1.0.1"

[[deps.ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[deps.ArnoldiMethod]]
deps = ["LinearAlgebra", "Random", "StaticArrays"]
git-tree-sha1 = "62e51b39331de8911e4a7ff6f5aaf38a5f4cc0ae"
uuid = "ec485272-7323-5ecc-a04f-4719b315124d"
version = "0.2.0"

[[deps.ArrayInterface]]
deps = ["Compat", "IfElse", "LinearAlgebra", "Requires", "SparseArrays", "Static"]
git-tree-sha1 = "265b06e2b1f6a216e0e8f183d28e4d354eab3220"
uuid = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9"
version = "3.2.1"

[[deps.Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[deps.Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[deps.ChainRulesCore]]
deps = ["Compat", "LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "f885e7e7c124f8c92650d61b9477b9ac2ee607dd"
uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4"
version = "1.11.1"

[[deps.ChangesOfVariables]]
deps = ["LinearAlgebra", "Test"]
git-tree-sha1 = "9a1d594397670492219635b35a3d830b04730d62"
uuid = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0"
version = "0.1.1"

[[deps.ColorSchemes]]
deps = ["ColorTypes", "Colors", "FixedPointNumbers", "Random"]
git-tree-sha1 = "a851fec56cb73cfdf43762999ec72eff5b86882a"
uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4"
version = "3.15.0"

[[deps.ColorTypes]]
deps = ["FixedPointNumbers", "Random"]
git-tree-sha1 = "024fe24d83e4a5bf5fc80501a314ce0d1aa35597"
uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
version = "0.11.0"

[[deps.Colors]]
deps = ["ColorTypes", "FixedPointNumbers", "Reexport"]
git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40"
uuid = "5ae59095-9a9b-59fe-a467-6f913c188581"
version = "0.12.8"

[[deps.CommonSubexpressions]]
deps = ["MacroTools", "Test"]
git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7"
uuid = "bbf7d656-a473-5ed7-a52c-81e309532950"
version = "0.3.0"

[[deps.Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "dce3e3fea680869eaa0b774b2e8343e9ff442313"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.40.0"

[[deps.CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[deps.CovarianceEstimation]]
deps = ["LinearAlgebra", "Statistics", "StatsBase"]
git-tree-sha1 = "bc3930158d2be029e90b7c40d1371c4f54fa04db"
uuid = "587fd27a-f159-11e8-2dae-1979310e6154"
version = "0.2.6"

[[deps.DataAPI]]
git-tree-sha1 = "cc70b17275652eb47bc9e5f81635981f13cea5c8"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.9.0"

[[deps.DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "7d9d316f04214f7efdbb6398d545446e246eff02"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.10"

[[deps.Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[deps.DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[deps.DensityInterface]]
deps = ["InverseFunctions", "Test"]
git-tree-sha1 = "80c3e8639e3353e5d2912fb3a1916b8455e2494b"
uuid = "b429d917-457f-4dbc-8f4c-0cc954292b1d"
version = "0.4.0"

[[deps.DiffResults]]
deps = ["StaticArrays"]
git-tree-sha1 = "c18e98cba888c6c25d1c3b048e4b3380ca956805"
uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5"
version = "1.0.3"

[[deps.DiffRules]]
deps = ["LogExpFunctions", "NaNMath", "Random", "SpecialFunctions"]
git-tree-sha1 = "d8f468c5cd4d94e86816603f7d18ece910b4aaf1"
uuid = "b552c78f-8df3-52c6-915a-8e097449b14b"
version = "1.5.0"

[[deps.Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[deps.Distributions]]
deps = ["ChainRulesCore", "DensityInterface", "FillArrays", "LinearAlgebra", "PDMats", "Printf", "QuadGK", "Random", "SparseArrays", "SpecialFunctions", "Statistics", "StatsBase", "StatsFuns", "Test"]
git-tree-sha1 = "7f3bec11f4bcd01bc1f507ebce5eadf1b0a78f47"
uuid = "31c24e10-a181-5473-b8eb-7969acd0382f"
version = "0.25.34"

[[deps.DocStringExtensions]]
deps = ["LibGit2"]
git-tree-sha1 = "b19534d1895d702889b219c382a6e18010797f0b"
uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
version = "0.8.6"

[[deps.Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[deps.Einsum]]
deps = ["Compat"]
git-tree-sha1 = "4a6b3eee0161c89700b6c1949feae8b851da5494"
uuid = "b7d42ee7-0b51-5a75-98ca-779d3107e4c0"
version = "0.4.1"

[[deps.FillArrays]]
deps = ["LinearAlgebra", "Random", "SparseArrays", "Statistics"]
git-tree-sha1 = "8756f9935b7ccc9064c6eef0bff0ad643df733a3"
uuid = "1a297f60-69ca-5386-bcde-b61e274b549b"
version = "0.12.7"

[[deps.FiniteDiff]]
deps = ["ArrayInterface", "LinearAlgebra", "Requires", "SparseArrays", "StaticArrays"]
git-tree-sha1 = "8b3c09b56acaf3c0e581c66638b85c8650ee9dca"
uuid = "6a86dc24-6348-571c-b903-95158fe2bd41"
version = "2.8.1"

[[deps.FixedPointNumbers]]
deps = ["Statistics"]
git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc"
uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
version = "0.8.4"

[[deps.ForwardDiff]]
deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "LinearAlgebra", "LogExpFunctions", "NaNMath", "Preferences", "Printf", "Random", "SpecialFunctions", "StaticArrays"]
git-tree-sha1 = "6406b5112809c08b1baa5703ad274e1dded0652f"
uuid = "f6369f11-7733-5829-9624-2563aa707210"
version = "0.10.23"

[[deps.FunctionWrappers]]
git-tree-sha1 = "241552bc2209f0fa068b6415b1942cc0aa486bcc"
uuid = "069b7b12-0de2-55c6-9aab-29f3d0a68a2e"
version = "1.1.2"

[[deps.Graphs]]
deps = ["ArnoldiMethod", "DataStructures", "Distributed", "Inflate", "LinearAlgebra", "Random", "SharedArrays", "SimpleTraits", "SparseArrays", "Statistics"]
git-tree-sha1 = "92243c07e786ea3458532e199eb3feee0e7e08eb"
uuid = "86223c79-3864-5bf0-83f7-82e725a168b6"
version = "1.4.1"

[[deps.HybridArrays]]
deps = ["LinearAlgebra", "Requires", "StaticArrays"]
git-tree-sha1 = "c1d5b1dcdf2140644e1c6beb9ca09fbed601c241"
uuid = "1baab800-613f-4b0a-84e4-9cd3431bfbb9"
version = "0.4.9"

[[deps.IfElse]]
git-tree-sha1 = "debdd00ffef04665ccbb3e150747a77560e8fad1"
uuid = "615f187c-cbe4-4ef1-ba3b-2fcf58d6d173"
version = "0.1.1"

[[deps.Inflate]]
git-tree-sha1 = "f5fc07d4e706b84f72d54eedcc1c13d92fb0871c"
uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9"
version = "0.1.2"

[[deps.InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[deps.InverseFunctions]]
deps = ["Test"]
git-tree-sha1 = "a7254c0acd8e62f1ac75ad24d5db43f5f19f3c65"
uuid = "3587e190-3f89-42d0-90ee-14403ec27112"
version = "0.1.2"

[[deps.IrrationalConstants]]
git-tree-sha1 = "7fd44fd4ff43fc60815f8e764c0f352b83c49151"
uuid = "92d709cd-6900-40b7-9082-c6be49f344b6"
version = "0.1.1"

[[deps.JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[deps.Kronecker]]
deps = ["LinearAlgebra", "NamedDims", "SparseArrays", "StatsBase"]
git-tree-sha1 = "a51f46415c844dee694cb8b20a3fcbe6dba342c2"
uuid = "2c470bb0-bcc8-11e8-3dad-c9649493f05e"
version = "0.5.1"

[[deps.LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[deps.LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[deps.LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[deps.LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[deps.Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[deps.LinearAlgebra]]
deps = ["Libdl", "libblastrampoline_jll"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[deps.LogExpFunctions]]
deps = ["ChainRulesCore", "ChangesOfVariables", "DocStringExtensions", "InverseFunctions", "IrrationalConstants", "LinearAlgebra"]
git-tree-sha1 = "be9eef9f9d78cecb6f262f3c10da151a6c5ab827"
uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688"
version = "0.3.5"

[[deps.Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[deps.MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "3d3e902b31198a27340d0bf00d6ac452866021cf"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.9"

[[deps.Manifolds]]
deps = ["Colors", "Distributions", "Einsum", "Graphs", "HybridArrays", "Kronecker", "LinearAlgebra", "ManifoldsBase", "Markdown", "Random", "RecipesBase", "RecursiveArrayTools", "Requires", "SimpleWeightedGraphs", "SpecialFunctions", "StaticArrays", "Statistics", "StatsBase"]
git-tree-sha1 = "3811935fd2549b0f5f9b365d6e7173fcdb98ec9a"
uuid = "1cead3c2-87b3-11e9-0ccd-23c62b72b94e"
version = "0.7.1"

[[deps.ManifoldsBase]]
deps = ["LinearAlgebra", "Markdown"]
git-tree-sha1 = "77a5949567437d185ee929c405e3c6c0768118ea"
uuid = "3362f125-f0bb-47a3-aa74-596ffd7ef2fb"
version = "0.12.9"

[[deps.Manopt]]
deps = ["ColorSchemes", "ColorTypes", "Colors", "DataStructures", "Dates", "LinearAlgebra", "ManifoldsBase", "Markdown", "Random", "Requires", "SparseArrays", "StaticArrays", "Statistics", "Test"]
git-tree-sha1 = "e87c818e19f79444fff1c77880cdb43b51974dfe"
uuid = "0fc0a36d-df90-57f3-8f93-d78a9fc72bb5"
version = "0.3.14"

[[deps.Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[deps.MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[deps.Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "bf210ce90b6c9eed32d25dbcae1ebc565df2687f"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.0.2"

[[deps.Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[deps.MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[deps.NaNMath]]
git-tree-sha1 = "bfe47e760d60b82b66b61d2d44128b62e3a369fb"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "0.3.5"

[[deps.NamedDims]]
deps = ["AbstractFFTs", "ChainRulesCore", "CovarianceEstimation", "LinearAlgebra", "Pkg", "Requires", "Statistics"]
git-tree-sha1 = "4f34e3ff2fa7f2a1c03fb2c4fdd637380d760bbc"
uuid = "356022a1-0364-5f58-8944-0da4b18d706f"
version = "0.2.42"

[[deps.NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[deps.OpenBLAS_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"]
uuid = "4536629a-c528-5b80-bd46-f80d51c5b363"

[[deps.OpenLibm_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "05823500-19ac-5b8b-9628-191a04bc5112"

[[deps.OpenSpecFun_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1"
uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e"
version = "0.5.5+0"

[[deps.OrderedCollections]]
git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.1"

[[deps.PDMats]]
deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"]
git-tree-sha1 = "ee26b350276c51697c9c2d88a072b339f9f03d73"
uuid = "90014a1f-27ba-587c-ab20-58faa44d9150"
version = "0.11.5"

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[deps.Preferences]]
deps = ["TOML"]
git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.2"

[[deps.Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[deps.QuadGK]]
deps = ["DataStructures", "LinearAlgebra"]
git-tree-sha1 = "78aadffb3efd2155af139781b8a8df1ef279ea39"
uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc"
version = "2.4.2"

[[deps.REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[deps.Random]]
deps = ["SHA", "Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[deps.RecipesBase]]
git-tree-sha1 = "6bf3f380ff52ce0832ddd3a2a7b9538ed1bcca7d"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.2.1"

[[deps.RecursiveArrayTools]]
deps = ["ArrayInterface", "ChainRulesCore", "DocStringExtensions", "FillArrays", "LinearAlgebra", "RecipesBase", "Requires", "StaticArrays", "Statistics", "ZygoteRules"]
git-tree-sha1 = "c944fa4adbb47be43376359811c0a14757bdc8a8"
uuid = "731186ca-8d62-57ce-b412-fbd966d074cd"
version = "2.20.0"

[[deps.Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[deps.Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.1.3"

[[deps.ReverseDiff]]
deps = ["ChainRulesCore", "DiffResults", "DiffRules", "ForwardDiff", "FunctionWrappers", "LinearAlgebra", "MacroTools", "NaNMath", "Random", "SpecialFunctions", "StaticArrays", "Statistics"]
git-tree-sha1 = "14b09b7d28378e6e8be334edc0ade781b5c65e85"
uuid = "37e2e3b7-166d-5795-8a7a-e32c996b4267"
version = "1.10.0"

[[deps.Rmath]]
deps = ["Random", "Rmath_jll"]
git-tree-sha1 = "bf3188feca147ce108c76ad82c2792c57abe7b1f"
uuid = "79098fc4-a85e-5d69-aa6a-4863f24498fa"
version = "0.7.0"

[[deps.Rmath_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "68db32dff12bb6127bac73c209881191bf0efbb7"
uuid = "f50d1b31-88e8-58de-be2c-1cc44531875f"
version = "0.3.0+0"

[[deps.SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[deps.Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[deps.SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[deps.SimpleTraits]]
deps = ["InteractiveUtils", "MacroTools"]
git-tree-sha1 = "5d7e3f4e11935503d3ecaf7186eac40602e7d231"
uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d"
version = "0.9.4"

[[deps.SimpleWeightedGraphs]]
deps = ["Graphs", "LinearAlgebra", "Markdown", "SparseArrays", "Test"]
git-tree-sha1 = "a6f404cc44d3d3b28c793ec0eb59af709d827e4e"
uuid = "47aef6b3-ad0c-573a-a1e2-d07658019622"
version = "1.2.1"

[[deps.Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[deps.SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.0.1"

[[deps.SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[deps.SpecialFunctions]]
deps = ["ChainRulesCore", "IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"]
git-tree-sha1 = "f0bccf98e16759818ffc5d97ac3ebf87eb950150"
uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
version = "1.8.1"

[[deps.Static]]
deps = ["IfElse"]
git-tree-sha1 = "e7bc80dc93f50857a5d1e3c8121495852f407e6a"
uuid = "aedffcd0-7271-4cad-89d0-dc628f76c6d3"
version = "0.4.0"

[[deps.StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "3c76dde64d03699e074ac02eb2e8ba8254d428da"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.2.13"

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[deps.StatsAPI]]
git-tree-sha1 = "0f2aa8e32d511f758a2ce49208181f7733a0936a"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.1.0"

[[deps.StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "2bb0cb32026a66037360606510fca5984ccc6b75"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.13"

[[deps.StatsFuns]]
deps = ["ChainRulesCore", "InverseFunctions", "IrrationalConstants", "LogExpFunctions", "Reexport", "Rmath", "SpecialFunctions"]
git-tree-sha1 = "bedb3e17cc1d94ce0e6e66d3afa47157978ba404"
uuid = "4c63d2b9-4356-54db-8cca-17b64c39e42c"
version = "0.9.14"

[[deps.SuiteSparse]]
deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"]
uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9"

[[deps.TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[deps.Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[deps.Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[deps.UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[deps.Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[deps.Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[deps.ZygoteRules]]
deps = ["MacroTools"]
git-tree-sha1 = "8c1a8e4dfacb1fd631745552c8db35d0deb09ea0"
uuid = "700de1a5-db45-46bc-99cf-38207098b444"
version = "0.2.2"

[[deps.libblastrampoline_jll]]
deps = ["Artifacts", "Libdl", "OpenBLAS_jll"]
uuid = "8e850b90-86db-534c-a0d3-1478176c7d93"

[[deps.nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[deps.p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"
"""

# ╔═╡ Cell order:
# ╟─0213d26a-18ac-11ec-03fd-ada5992bcea8
# ╟─f3bc91ee-5871-4cba-ac89-190deb71ad0f
# ╟─d9be6c2f-65fd-4685-9005-da22bf985e28
# ╠═856f336c-e232-4f1f-b1ac-759b4558acd1
# ╠═b0769dfa-28cf-440e-9ba2-1ef488f171a9
# ╟─18d7459f-eed6-489b-a096-ac77ccd781af
# ╟─a3df142e-94df-48d2-be08-d1f1f3854c76
# ╟─9a030ac6-1f44-4fa6-8bc9-1c0278e97fe2
# ╠═19747159-d383-4547-9315-0ed2494904a6
# ╠═41c204dd-6e4e-4a70-8f06-209a469e0680
# ╠═2e33de5e-ffaa-422a-91d9-61f588ed1211
# ╟─bbd9a010-1981-45b3-bf7d-c04bcd2c2128
# ╠═08456b40-74ec-4319-93e7-130b5cf70ac3
# ╠═12327b62-7e79-4381-b6a7-f85b08a8251b
# ╠═07f9a630-e53d-45ea-b109-3d4de190723d
# ╟─8e5f677d-dafa-49b9-b678-3f129be31dcf
# ╟─77769eab-54dd-41dc-8125-0382e5ef0bf1
# ╟─57cda07f-e432-46af-b771-5e5a3067feac
# ╠═c3f3aeba-2849-4715-94e2-0c44613a2ce9
# ╟─786fce04-53ef-448d-9657-31208b35fb7e
# ╠═c1341fef-adec-4574-a642-a1a8a9c1fee5
# ╟─0818a62f-1bef-44f7-a33f-1ab0054e853c
# ╠═89cd6b4b-f9ef-47ac-afd3-cf9aacf43256
# ╠═7c5a8a17-6f63-4587-a94a-6936bdd3cec6
# ╠═b3e7f57f-d87a-47c5-b8ad-48b6d205fa73
# ╟─893db402-283f-4e3e-8bf7-c6f22e485efb
# ╟─8494a0d6-dbf2-4eb0-a555-f00e446fbe38
# ╠═c93eb2da-89df-4751-b086-62be604d41e6
# ╟─e2bf6f55-7235-4d75-8bee-a325434e32ad
# ╠═153378ca-703d-4a84-bc63-22347399a160
# ╠═699f0177-2c5b-434b-9eca-b6fc573e497f
# ╠═9b52d05c-4cba-4922-85a4-1a2a3c74823e
# ╠═cb3fe7aa-1262-48f2-9ebc-3e959c72a33e
# ╟─13712c64-48fd-4f2a-9ee4-1949e51d316f
# ╠═64beb3dd-9507-4792-be02-ae1405704690
# ╟─2be4f9e8-0331-44ac-839f-7bb71d9edef9
# ╠═6f1d748f-27ce-496b-8561-f16972da50cc
# ╠═7dd656ea-08de-4172-8a92-87ad2228ce69
# ╟─219573d2-283f-456c-a5c3-fadd734fc157
# ╠═e28a2752-877c-4ab4-a253-8d26fa9a73c2
# ╟─25c65878-1be6-4fec-b65e-9c1741320a41
# ╠═9a66d4f3-508d-4285-9a93-df1323575202
# ╠═c07fb3d0-d12f-44d7-bcab-7a0d39e6af8d
# ╟─f47c70b6-ca05-498f-9e10-58c3839ca427
# ╟─32d8d025-3993-4d31-9eea-3463e0af1c12
# ╟─00000000-0000-0000-0000-000000000001
# ╟─00000000-0000-0000-0000-000000000002


#=
   From BLOG example/mixture-of-gaussian.blog
   """
   A model for mixture of Gaussian distribution.
   The latent variable \verb|z| determines the mixture component of \verb|x|.
   The program asks the possible value of \verb|z| given the evidence of \verb|x| being 0.2.

   @author leili
   @since 2014-06-16
   """

  Summary Statistics
  parameters      mean       std   naive_se      mcse         ess      rhat   ess_per_sec
      Symbol   Float64   Float64    Float64   Float64     Float64   Float64       Float64

           z    0.5481    0.4977     0.0050    0.0047   8954.7436    1.0004     1230.7234
       xpost    0.0560    1.1241     0.0112    0.0108   8908.5957    0.9999     1224.3809

   Cf ~/webppl/mixture_of_gaussian.wppl

=#

using Turing, StatsPlots

include("jl_utils.jl")

@model function mixture_of_gaussian(x=0.2)
    z ~ flip(0.5)
    x ~ z == true ? Normal(0.5, 1.0) : Normal(-0.5, 1.0)

    # Posterior prediction of x
    xpost ~ z == true ? Normal(0.5, 1.0) : Normal(-0.5, 1.0)
end

x=0.2
model = mixture_of_gaussian(x)

# chns = sample(model, Prior(), 10_000)
# chns = sample(model, MH(), 100_000)
chns = sample(model, PG(15), 10_000)
# chns = sample(model, SMC(), 10_000)
# chns = sample(model, IS(), 10_000)

display(chns)
# display(plot(chns))


# -*- coding: utf-8 -*-
# ---
# jupyter:
#   jupytext:
#     formats: ipynb,jl:hydrogen
#     text_representation:
#       extension: .jl
#       format_name: hydrogen
#       format_version: '1.3'
#       jupytext_version: 1.11.2
#   kernelspec:
#     display_name: Julia 1.6.2
#     language: julia
#     name: julia-1.6
# ---

# %% [markdown]
# https://twitter.com/mkashi/status/1434847564256268291

# %%
ENV["LINES"] = 1000

# %%
function msb64(x)
    ε = eps(Float64)
    q = (1/ε + 1) * x
    m = q - (1 - ε/2) * q
    Int64(m)
end

@show eps(Float64) == 1/2^52

[(x = 2^k; (k, log2(msb64(x - 2)), log2(msb64(x - 1)), log2(msb64(x)))) for k in 1:54]

# %%
using Quadmath

function msb128(x)
    ε = eps(Float128)
    q = (1/ε + 1) * x
    m = q - (1 - ε/2) * q
    Int128(m)
end

@show eps(Float128) == 1/Int128(2)^112

[(x = Int128(2)^k; (k, log2(msb128(x - 2)), log2(msb128(x - 1)), log2(msb128(x)))) for k in 1:114]

# %%
function msb(x)
    ε = eps(float(typeof(x)))
    q = (1/ε + 1) * x
    m = q - (1 - ε/2) * q
    oftype(x, m)
end

# %%
@show precision(Float64)
@show eps(Float64) == 1/2^(precision(Float64)-1)
[(x = 2^k; (k, log2(msb(x - 2)), log2(msb(x - 1)), log2(msb(x)))) for k in 1:precision(Float64)+1]

# %%
@show precision(BigFloat)
@show eps(BigFloat) == 1/big(2)^(precision(BigFloat)-1)
[(x = big(2)^k; (k, log2(msb(x - 2)), log2(msb(x - 1)), log2(msb(x)))) for k in 1:precision(BigFloat)+1]

# %%
@code_typed msb(12345)

# %%
@code_native debuginfo=:none msb(12345)

# %%


# Brown and Dennis functions
#
#   Source: Problem 16 in
#   J.J. Moré, B.S. Garbow and K.E. Hillstrom,
#   "Testing Unconstrained Optimization Software",
#   ACM Transactions on Mathematical Software, vol. 7(1), pp. 17-41, 1981
#
#   classification SUR2-AN-4-0

function brownden()

  nlp = Model()

  @variable(nlp, x[1:4])
  setvalue(x, [25.0; 5.0; -5.0; -1.0])

  @NLobjective(
    nlp,
    Min,
    sum( ((x[1] + x[2] * i/5 - exp(i/5))^2 + (x[3] + x[4]*sin(i/5) -
      cos(i/5))^2)^2 for i = 1:20)
  )

  return nlp
end


using Distributions

function random_spatial_network(N::Integer; # number of nodes
    d::Integer = 2, # dimension
    Pk = Poisson(10), # Expected degree distribution
    Pf = Normal(0.0, 0.03) # Distance kernel
    )
    ks = rand(Pk, N)
    xs = rand(N,d)
    dists = Array{Float64,2}(undef, N, N)
    for i in 1:N, j in i:N
        dists[i,j] = norm(xs[i,:]-xs[j,:])
        dists[j,i] = dists[i,j]
    end
    f = x -> pdf(Pf, x)
    return random_spatial_network(dists, ks, f, 1), xs
end

function random_spatial_network(distances::AbstractMatrix, ks, f, ρ)
    N = size(distances, 1)
    k = mean(ks)
    ps = (ks * ks') .* f.(distances) / (ρ * k)
    return edge_probability_graph(ps)
end

function edge_probability_graph(ps::AbstractMatrix)
    N = size(ps, 1)
    edges = [(i,j) for i in 1:N-1, j in (i+1):N if rand() < ps[i,j]]
    return SimpleGraphFromIterator(edges)
end


module CGPrism3D
using LinearAlgebra
#using StatsBase

# Define chop function
export numchop
numchop(num::Real)=(abs(num) >= 1000*eps() ? num : zero(Real))

function numchop(num::Complex)
    numchop(imag(num)) == 0 ? numchop(real(num)) : Complex(numchop(real(num)), numchop(imag(num)))
end

# Define k-level of SU(2)_k
const K = 3 # K-level
const x = K+1 #number of spins
const y = K/2 # maximum spin

#Define all functions needed for G-symbol
# Define delta{ijk} -> coupling rules
function delta(i::Float64,j::Float64,k::Float64)
    sol = 0
    if i <=(j+k) && j<=(i+k) && k<=(i+j) && i+j+k <= K && 2*(i+j+k)%2 ==0
        sol = 1
    end
    return sol
end

#Define quantum numbers qn (this is real)
function qn(n::Float64)
    sol = (exp(pi*n*im/(K+2)) - exp(-pi*n*im/(K+2))) / (exp(pi*im/(K+2)) - exp(-pi*im/(K+2)))
    return real(sol)
end

#Define qn factorial
function qnfact(n::Float64)
    sol = 1
    for i in 1:n
        sol *= qn(i)
    end
    return sol
end

#Define square root of quantum dimension
function visqrt(i::Float64)
    sol = ((-1+0im)^i )*sqrt(qn(2*i+1))
    return sol
end

#Define triangle equality
function trian(i::Float64,j::Float64,k::Float64)
    sol = 0
    if delta(i,j,k) == 1
        sol = delta(i,j,k)*sqrt(qnfact(i+j-k)*qnfact(i-j+k)*qnfact(-i+j+k)/qnfact(i+j+k+1))
    end
    return sol
end

# Define Racah-Wigner six-j symbol
function RacWig6j(i::Float64,j::Float64,m::Float64,k::Float64,l::Float64,n::Float64)
    a = i+j+m; b = i+l+n; c = k+j+n; d = k+l+m;  e = i+j+k+l; f = i+k+m+n; g = j+l+m+n
    sol = 0
    if delta(i,j,m) != 0 && delta(i,l,n) != 0 && delta(k,j,n) != 0 && delta(k,l,m) != 0
        sumz = 0
        for z in max(a,b,c,d):min(e,f,g)
            sumz += (-1)^z *qnfact(z+1)/
                ((qnfact(e-z)*qnfact(f-z)*qnfact(g-z))* (qnfact(z-a)*qnfact(z-b)*qnfact(z-c)*qnfact(z-d)))
        end
        sol = trian(i,j,m)*trian(i,l,n)*trian(k,j,n)*trian(k,l,m)*sumz
    end
    return sol
end

#Define F-symbol
function Fsymb(i::Float64,j::Float64,m::Float64,k::Float64,l::Float64,n::Float64)
    sol = 0
    if delta(i,j,m) != 0 && delta(i,l,n) != 0 && delta(k,j,n) != 0 && delta(k,l,m) != 0
        sol = (-1+0im)^(i+j+k+l)*sqrt(qn(2*m+1)*qn(2*n+1)) * RacWig6j(i,j,m,k,l,n)
    end
    return sol
end

#Define G-symbol
function Gsymb(i::Float64,j::Float64,m::Float64,k::Float64,l::Float64,n::Float64)
    sol = 0
    if delta(i,j,m) != 0 && delta(i,l,n) != 0 && delta(k,j,n) != 0 && delta(k,l,m) != 0
        sol = Fsymb(i,j,m,k,l,n) /(visqrt(m)*visqrt(n))
    end
    return sol
end

export Tetra6j, delta, visqrt
#Define G-symbol times dimensions
function Tetra6j(i::Float64,j::Float64,m::Float64,k::Float64,l::Float64,n::Float64)
    sol = 0
    if delta(i,j,m) != 0 && delta(i,l,n) != 0 && delta(k,j,n) != 0 && delta(k,l,m) != 0
        dims = prod(visqrt.([i,j,m,k,l,n]))
        sol =  numchop(dims*Gsymb(i,j,m,k,l,n))
    end
    return sol
end

#Define G-symbol 2
function Gsymb2(i::Float64,j::Float64,m::Float64,k::Float64,l::Float64,n::Float64)
    sol = 0
    if delta(i,j,m) != 0 && delta(i,l,n) != 0 && delta(k,j,n) != 0 && delta(k,l,m) != 0
        sol = (-1+0im)^(i+j+k+l+m+n)*RacWig6j(i,j,m,k,l,n)
    end
    return sol
end

## Define Prisms
#Prism A as gluing together three tetrahedra ;  Prism B changing diagonals using F-symbol
function prismA(jd1::Float64,jd2::Float64,je1::Float64,jb1::Float64,jg::Float64,ja1::Float64,jc::Float64,
                jb2::Float64,jf1::Float64,ja2::Float64,jf2::Float64,je2::Float64)
    sol = 0
    if delta(jd1,jd2,je1) != 0 && delta(jd1,jb1,jg) != 0 && delta(ja1,jd2,jg) != 0 && delta(ja1,jb1,je1) != 0 && delta(jd1,jc,jb2) != 0 && delta(jf1,jd2,jb2) != 0 && delta(jf1,jc,je1) != 0 && delta(jd1,ja2,jf2) != 0 && delta(jc,je2,jf2) != 0 && delta(jb2,je2,ja2) != 0
        dims = visqrt(ja1)*visqrt(jb1)*visqrt(jg)*visqrt(jf1)*visqrt(jc)*visqrt(jb2)*visqrt(je2)*visqrt(ja2)*visqrt(jf2)*visqrt(jd1)*visqrt(jd2)*visqrt(je1)
        sol =  dims*Gsymb(jd1,jd2,je1,ja1,jb1,jg) * Gsymb(jd1,jd2,je1,jf1,jc,jb2) * Gsymb(jd1,jc,jb2,je2,ja2,jf2)
    end
    return sol
end

function prismB(jd2p::Float64,jd1p::Float64,je2::Float64,jb2::Float64,jg::Float64,ja2::Float64,jc::Float64,
                jb1::Float64,jf2::Float64,ja1::Float64,jf1::Float64,je1::Float64)
    sol = 0
    if delta(jd1p,jd2p,je2) != 0 && delta(jd2p,jb2,jg) != 0 && delta(ja2,jd1p,jg) != 0 && delta(ja2,jb2,je2) != 0 && delta(jd2p,jc,jb1) != 0 && delta(jf2,jd1p,jb1) != 0 && delta(jf2,jc,je2) != 0 && delta(jd2p,ja1,jf1) != 0 && delta(jc,je1,jf1) != 0 && delta(jb1,je1,ja1) != 0
        for jd1 in 0.:0.5:y
            if delta(jb1,jg,jd1) != 0 && delta(ja2,jf2,jd1) != 0
                for jd2 in 0.:0.5:y
                    if delta(ja1,jg,jd2) != 0 && delta(jb2,jf1,jd2) != 0
                        sol += numchop(Fsymb(ja1,jg,jd2,jb2,jf1,jd2p)*Fsymb(jb1,jg,jd1,ja2,jf2,jd1p)*prismA(jd1,jd2,je1,jb1,jg,ja1,jc,jb2,jf1,ja2,jf2,je2))
                    end
                end
            end
        end
    end
    return sol
end

# This is the relabelling vertices. Does it work in general after one step in algorithm?
function prismB2(jd2::Float64,jd1::Float64,je2::Float64,jb2::Float64,jg::Float64,ja2::Float64,jc::Float64,
                jb1::Float64,jf2::Float64,ja1::Float64,jf1::Float64,je1::Float64)
    return prismA(jd1,jd2,je1,jb1,jg,ja1,jc,jb2,jf1,ja2,jf2,je2)
end

export dataPrsmA, dataPrsmB, dataPrsmB2, dataTet, dataTetF

# get all non-zero amplitudes and spins
function dataPrsmA()
    ampsInfo = Array{Float64,1}[]
    amps = Float64[]
    for jd1 in 0.:0.5:y, jd2 in 0.:0.5:y, je1 in 0.:0.5:y
        if delta(jd1,jd2,je1) != 0
            for jb1 in 0.:0.5:y, jg in 0.:0.5:y
                if delta(jd1,jb1,jg) != 0
                    for ja1 in 0.:0.5:y
                        if delta(ja1,jd2,jg) != 0 && delta(ja1,jb1,je1) != 0
                            for jc in 0.:0.5:y, jb2 in 0.:0.5:y
                                if delta(jd1,jc,jb2) != 0
                                    for jf1 in 0.:0.5:y
                                        if delta(jf1,jd2,jb2) != 0 && delta(jf1,jc,je1) != 0
                                            for ja2 in 0.:0.5:y, jf2 in 0.:0.5:y
                                                if delta(jd1,ja2,jf2) != 0
                                                    for je2 in 0.:0.5:y
                                                        if delta(jc,je2,jf2) != 0 && delta(jb2,je2,ja2) != 0 #
                                                            sol = numchop(prismA(jd1,jd2,je1,jb1,jg,ja1,jc,jb2,jf1,ja2,jf2,je2))
                                                            if sol != 0
                                                                indx = [jb1,jg,ja1,jd1,jd2,je1,jc,jb2,jf1,ja2,jf2,je2]
                                                                push!(ampsInfo,indx)
                                                                push!(amps,sol)
                                                            end
                                                        end
                                                    end
                                                end
                                            end
                                        end
                                    end
                                end
                            end
                        end
                    end
                end
            end
        end
    end
    return ampsInfo, amps
end

function dataPrsmB()
    ampsInfo = Array{Float64,1}[]
    amps = Float64[]
    for jd2 in 0.:0.5:y, jd1 in 0.:0.5:y, je2 in 0.:0.5:y
        if delta(jd1,jd2,je2) != 0
            for jb2 in 0.:0.5:y, jg in 0.:0.5:y
                if delta(jd2,jb2,jg) != 0
                    for ja2 in 0.:0.5:y
                        if delta(ja2,jd1,jg) != 0 && delta(ja2,jb2,je2) != 0
                            for jc in 0.:0.5:y, jb1 in 0.:0.5:y
                                if delta(jd2,jc,jb1) != 0
                                    for jf2 in 0.:0.5:y
                                        if delta(jf2,jd1,jb1) != 0 && delta(jf2,jc,je2) != 0
                                            for ja1 in 0.:0.5:y, jf1 in 0.:0.5:y
                                                if delta(jd2,ja1,jf1) != 0
                                                    for je1 in 0.:0.5:y
                                                        if delta(jc,je1,jf1) != 0 && delta(jb1,je1,ja1) != 0 #
                                                            sol = numchop(prismB(jd2,jd1,je2,jb2,jg,ja2,jc,jb1,jf2,ja1,jf1,je1))
                                                            if sol != 0
                                                                indx = [jb2,jg,ja2,jd2,jd1,je2,jc,jb1,jf2,ja1,jf1,je1]
                                                                push!(ampsInfo,indx)
                                                                push!(amps,sol)
                                                            end
                                                        end
                                                    end
                                                end
                                            end
                                        end
                                    end
                                end
                            end
                        end
                    end
                end
            end
        end
    end
    return ampsInfo, amps
end

function dataPrsmB2()
    ampsInfo = Array{Float64,1}[]
    amps = Float64[]
    for jd2 in 0.:0.5:y, jd1 in 0.:0.5:y, je2 in 0.:0.5:y
        if delta(jd2,jd1,je2) != 0
            for jb2 in 0.:0.5:y, jg in 0.:0.5:y
                if delta(jd2,jb2,jg) != 0
                    for ja2 in 0.:0.5:y
                        if delta(ja2,jd1,jg) != 0 && delta(ja2,jb2,je2) != 0
                            for jc in 0.:0.5:y, jb1 in 0.:0.5:y
                                if delta(jd2,jc,jb1) != 0
                                    for jf2 in 0.:0.5:y
                                        if delta(jf2,jd1,jb1) != 0 && delta(jf2,jc,je2) != 0
                                            for ja1 in 0.:0.5:y, jf1 in 0.:0.5:y
                                                if delta(jd2,ja1,jf1) != 0
                                                    for je1 in 0.:0.5:y
                                                        if delta(jc,je1,jf1) != 0 && delta(jb1,je1,ja1) != 0 #
                                                            sol = numchop(prismB2(jd2,jd1,je2,jb2,jg,ja2,jc,jb1,jf2,ja1,jf1,je1))
                                                            if sol != 0
                                                                indx = [jb2,jg,ja2,jd2,jd1,je2,jc,jb1,jf2,ja1,jf1,je1]
                                                                push!(ampsInfo,indx)
                                                                push!(amps,sol)
                                                            end
                                                        end
                                                    end
                                                end
                                            end
                                        end
                                    end
                                end
                            end
                        end
                    end
                end
            end
        end
    end
    Indxspins = sort(ampsInfo)
    ampvals = amps[sortperm(ampsInfo)]
    return Indxspins, ampvals
end

function dataTet()
    t6j = []
    t6s = []
    for ja in 0.:0.5:y, jb in 0.:0.5:y, jc in 0.:0.5:y
        if delta(ja,jb,jc) != 0
            for jd in 0.:0.5:y, je in 0.:0.5:y
                if delta(jc,jd,je) != 0
                    for jf in 0.:0.5:y
                        if delta(ja,je,jf) != 0 && delta(jb,jd,jf) != 0
            #dims = prod(visqrt.([ja,jb,jc,jd,je,jf]))
                            sol = numchop(Tetra6j(ja,jb,jc,jd,je,jf) )
                            if sol != 0
                                push!(t6s,sol)
                                push!(t6j,[ja,jb,jc,jd,je,jf])
                            end
                        end
                    end
                end
            end
        end
    end
    return t6j,t6s
end

function dataTetF()
    t6j = []
    t6s = []
    for ja in 0.:0.5:y, jb in 0.:0.5:y, jc in 0.:0.5:y
        if delta(ja,jb,jc) != 0
            for jd in 0.:0.5:y, je in 0.:0.5:y
                if delta(jc,jd,je) != 0
                    for jf in 0.:0.5:y
                        if delta(ja,je,jf) != 0 && delta(jb,jd,jf) != 0
            #dims = prod(visqrt.([ja,jb,jc,jd,je,jf]))
                            sol = numchop(Fsymb(ja,jb,jc,jd,je,jf) )
                            if sol != 0
                                push!(t6s,sol)
                                push!(t6j,[ja,jb,jc,jd,je,jf])
                            end
                        end
                    end
                end
            end
        end
    end
    return t6j,t6s
end

export tensorSplitSVD, fullSplitPrism3D, tensorGlue, tensorSum, tensorGluePrism3D, tensor22move, tensor31move

# a general function to split a tensor M^C_{AB} to U^C_{Ai} and  V^C_{iB} with C the shared face.. for particular values of C
# first reduce to matrix form and then perform svd to get U, s, and V
#This returns the svd of tensorM,i.e., U^C_{Ai}, s_i, and V^C_{iB}.
function tensorSplitSVD(tensorM,posnA,posnB,posnC,spinC)
    indx = tensorM[1]
    ampvals = tensorM[2]
    fc = findall(x-> x[posnC] == spinC, indx) #find spinsC in all spins length(posnC) = length(spinC)
    ampInfo = indx[fc] # this contains all spin configuration containing ampsC
    amps = ampvals[fc] # this contains the corresponding spin amplitude values for ampsC
    Ucol = unique(getindex.(ampInfo, [posnA]))
    Vrow = unique(getindex.(ampInfo, [posnB]))
    matM = zeros(length(Ucol),length(Vrow))
    if length(matM) != 0
        for i in 1:length(ampInfo)
            qu = findall(x-> x == ampInfo[i][posnA], Ucol)[1]# [1] since it returns an array [1]
            qv = findall(x-> x == ampInfo[i][posnB], Vrow)[1]
            matM[qu,qv] = amps[i]
        end
        U, s, V = svd(matM)
        return U,s,V
    else
        return 0,0,0
    end
end

#A general function to splits fully the tensor M^C_{AB} to tensors U^C_{Ai} and V^C_{iB}.
# It uses tensorSplitSVD,'fixes' the sign problem from svd, and normalize
#This is designed specifically for the prisms we are working with. It keeps only the first singular value (assuming a gemetric split)
# splits prism into tetrahedron and a  pyramid
function fullSplitPrism3D(dataM,posnA,posnB,posnC) # posnA,B,C must be vectors @assert length(dataM) = vcat(A,B,C)
    indx = dataM[1]
    ampvals = dataM[2]
    posAC = vcat(posnA,posnC) # assert posnA,posnC must be both vectors
    posCB = vcat(posnC,posnB) # assert posnB,posnC must be both vectors
    indxU = unique(getindex.(indx, [posAC]))
    indxV = unique(getindex.(indx, [posCB]))
    ampsC = unique(getindex.(indx, [posnC])) # get all unique spins C (jd1,jd2,je1)
    ampsU = [] # this will store all amplitudes for U^C_{A}
    ampsV = [] # this will store all amplitudes for V^C_{B}
    #indxUV = [] # if we want this will store all spins in the right order
    for i in 1:length(ampsC) # loop over the unique spins
        jd1 = ampsC[i][1]; jd2 = ampsC[i][2]; je1 = ampsC[i][3]  #jd1,jd2,je1 in spin C form a triangle
        if delta(jd1,jd2,je1) != 0
            spinC = [jd1,jd2,je1]
            U, s, V = tensorSplitSVD(dataM,posnA,posnB,posnC,spinC)
            #truncate and use only first singular value
            U1 = U[:,1]
            V1 = V[:,1]
            s1 = s[1]
            valU = U1*sqrt(s1)*sqrt(prod(visqrt.(ampsC[i]))) # multiply by leftover dimension factors
            valV = V1*sqrt(s1)*sqrt(prod(visqrt.(ampsC[i])))
            # valU,valV are all either real or purely imaginary
            valU = real(valU) - imag(valU) # take -ve of imaginary part if it gives only imag
            valV = real(valV) + imag(valV)
            ja1 = indxU[1][1]; jb1 = indxU[1][2]; jg = indxU[1][3] # pick the first spins in U
            #Fixing sign problem -- keep the same sign for U and Tetra6j symbol
            if sign(valU[1]) == sign(Tetra6j(jd1,jd2,je1,jb1,jg,ja1) )
                push!(ampsU, valU)#/valU[1])
                push!(ampsV, valV)#/valV[1])
            else
                push!(ampsU, -valU)#/valU[1])
                push!(ampsV, -valV)#/valV[1])
            end
        end
    end
    solU = collect(Iterators.flatten(ampsU))
    solV = collect(Iterators.flatten(ampsV))
    ansU = solU/solU[1] # normalization condition
    ansV = solV/solV[1]
    return (indxU,ansU),(indxV,ansV) #,indxUV
end

#its better if TA the 'smallest' tensor, posnA and posnB should be of the same length
function tensorGlue(tensorB,tensorA,posnB,posnA)# glue two tensors TA,TB along posnA from TA and posnB from TB
    indxA = tensorA[1] # spins of TA
    indxB = tensorB[1] # spins of TB
    ampsA = tensorA[2] # amplitudes of TA
    ampsB = tensorB[2] # amplitudes of TB
    lena = collect(1:length(indxA[1]))
    deleteat!(lena,sort(posnA)) # assert needs posnA to be a vector
    #if length(posnA) > 1 deleteat!(lena,sort(posnA,dims=2) else  deleteat!(lena,sort(posnA)) end
    amps = []
    qq = []
    for i in 1:length(indxA)
        fc = findall(x-> x[posnB] == indxA[i][posnA], indxB) # needs length(posnA) = length(posnB)
        ans = @. ampsA[i]*ampsB[fc]
        push!(amps,ans)
        indxa = repeat([indxA[i][lena]],length(indxB[fc]))
        #for j in indxA[i][lena]
        indxa1 = vcat.(indxB[fc],indxa)
        push!(qq,indxa1)
        #push!(qq,[collect(Iterators.flatten([indxB[fc][j],indxA[i][lena]])) for j in 1:length(indxB[fc])])
    end
    sol = collect(Iterators.flatten(amps))
    indxsol = collect(Iterators.flatten(qq))
    return indxsol, sol
end
export tensorGlueN

function tensorGlueN(tensorB,tensorA,posnB,posnA)# glue two tensors TA,TB along posnA from TA and posnB from TB
    indxA = tensorA[1] # spins of TA
    indxB = tensorB[1] # spins of TB
    ampsA = tensorA[2] # amplitudes of TA
    ampsB = tensorB[2] # amplitudes of TB
    shrdA = getindex.(indxA,[posnA])
    shrdB = getindex.(indxB,[posnB])
    lena = collect(1:length(indxA[1]))
    deleteat!(lena,sort(posnA)) # assert needs posnA to be a vector
    #if length(posnA) > 1 deleteat!(lena,sort(posnA,dims=2) else  deleteat!(lena,sort(posnA)) end
    shrdAB = []
    qq = []
    amps = []
    sa = sortperm(shrdA)
    sb = sortperm(shrdB)
    shrdA = shrdA[sa]
    shrdB = shrdB[sb]
    sindxA = indxA[sa]
    sindxB = indxB[sb]
    sampsA = ampsA[sa]
    sampsB = ampsB[sb]
    shrdAu = unique(shrdA)
    shrdBu = unique(shrdB)
    shrd = shrdAu
    if shrdAu == shrdBu
        shrd = shrdAu
    else
        for i in shrdBu
            shrd = shrdAu
            if !(i in shrdAu)
                push!(shrd,i)
            end
        end
    end
    cshrdA = countmap(shrdA)
    cshrdB = countmap(shrdB)
    for i in shrd
        ca = count(x->x==i,shrdA)
        cb = count(x->x==i,shrdB)
        push!(shrdAB, (i, ca,cb ))
    end
    cnt = 1
    cntb = 1
    for i in shrdAB
        # Put condition for when one of a or b is zero
        for j in 1:i[2]
            ans = @. sampsA[j]*sampsB[1:i[3]]
            push!(amps,ans)
            indxa = repeat([sindxA[j][lena]],i[3])
            #for j in indxA[i][lena]
            indxa1 = vcat.(sindxB[1:i[3]],indxa)
            push!(qq,indxa1)
        end
        deleteat!(sindxA,1:i[2])
        deleteat!(sindxB,1:i[3])
        deleteat!(sampsA,1:i[2])
        deleteat!(sampsB,1:i[3])
        #cntb = cnt
    end
    indxsol = collect(Iterators.flatten(qq))
    sol = collect(Iterators.flatten(amps))
    return indxsol,sol
end

export visqrt, tensorGluePrism3D2, tensorGluePrism3DN

function tensorGluePrism3D(tensorB,tensorA,posnB,posnA)
    indx, amps = tensorGlue(tensorB,tensorA,posnB,posnA)
    face = getindex.(indx, [posnB])
    ans = complex(zeros(length(face)))
    for i in 1:length(face)
        ans[i] = prod(visqrt.(face[i]))
    end
    ampsN = @. amps / ans
    ampsN = real(ampsN)+imag(ampsN)
    return indx, ampsN
end

function tensorGluePrism3DN(tensorB,tensorA,posnB,posnA)
    indx, amps = tensorGlueN(tensorB,tensorA,posnB,posnA)
    face = getindex.(indx, [posnB])
    ans = complex(zeros(length(face)))
    for i in 1:length(face)
        ans[i] = prod(visqrt.(face[i]))
    end
    ampsN = @. amps / ans
    ampsN = real(ampsN)+imag(ampsN)
    return indx, ampsN
end

## This is to try to take care of bulk spins
function tensorGluePrism3D2(tensorB,tensorA,posnB,posnA)
    indx, amps = tensorGlue(tensorB,tensorA,posnB,posnA)
    face = getindex.(indx, [posnB])
    ans = complex(ones(length(face)))
    for i in 2:length(face)
        ans[i] = prod(visqrt.(face[i]))
    end
    ampsN = @. amps / ans
    ampsN = real(ampsN)+imag(ampsN)
    return indx, ampsN
end

function tensorSum(ampJ,posnN)
    indx = ampJ[1]
    amps = ampJ[2]
    lenS = collect(1:length(indx[1]))
    deleteat!(lenS,sort(posnN))
    #if length(posnN) > 1   deleteat!(lenS,sort(posnN,dims=2)) else  deleteat!(lenS,sort(posnN)) end
    indxef = getindex.(indx,[lenS]) # get all spins without N
    indxefu = unique(indxef)
    tt = sortperm(indxef) # sort them spins
    indxef = indxef[tt] # apply sort on spins
    amps = amps[tt]  # apply sort on amps
    #fc = findall(x-> x[lenS]==ans ,indx)
    qq = []
    indq = []
    fc = sortperm(indxef)
    indxN = indxef[fc]
    ampsN = amps[fc]

    for i in 2:length(indxN)
        if indxN[i] == indxN[i-1]
            ampsN[i] += ampsN[i-1]
        elseif numchop(ampsN[i-1]) != 0
            push!(indq,indxN[i-1])
            push!(qq,ampsN[i-1])
        end
    end
    push!(indq,indxN[end])
    push!(qq,ampsN[end])
    return indq, qq
end

function tensor22move(tensor,face)# place middle edge at last position # first 2 position form a triangle
    # assert face contains 5 elements
    indx = tensor[1] # spins of TB
    amps = tensor[2]
    flast = face[end]
    ff1 = collect(1:length(indx[1]))
    deleteat!(ff1,flast)
    indxf1 = getindex.(indx, [ff1])
    indxf1u = unique(indxf1)
    #indxf1uMod = unique(indxf1)
    indxlast = getindex.(indx, [flast])
    for i in 1:4
        if face[i]>flast
            face[i] = face[i]-1
        end
    end
    sol = []
    findx = []
    for i in 1:length(indxf1u)
        iMod = copy(indxf1u[i])
        j1 = iMod[face[1]]; j2 = iMod[face[2]]; j3 = iMod[face[3]]; j4 = iMod[face[4]]; #j5 = ampsN[i][face[5]]
    #    #ans = 0#amps[i]
    #    ind = zeros(length(indx[1]))
        for j in 0.:0.5:y
            #iMod = deepcopy(indxf1u)
            if delta(j1,j3,j) != 0 && delta(j2,j4,j) != 0
                fc = findall(x-> x == iMod,indxf1 )
                ans = 0
                for t in fc
                    ju = indxlast[t]
                    ans += numchop(amps[t] * Fsymb(j1,j3,j,j4,j2,ju)) #* ampls[t]
                end
                if numchop(ans) !=0
                    push!(sol,ans)
                    iMods = copy(indxf1u[i])
                    push!(findx,insert!(iMods,flast,j))
                end
    #            end
            end
        end
    end
    return findx,sol#indx,amps
end

function tensor31move(tensor,tet)
    # assert tet contains 6 elements
    indx = tensor[1] # spins of TB
    amps = tensor[2] # amps
    indxout = getindex.(indx, [tet[4:6]])
    ff1 = collect(1:length(indx[1]))
    deleteat!(ff1,sort(tet[4:6]))
    indxkeep = getindex.(indx, [ff1])
    indxkeepu = unique(getindex.(indx, [ff1]))
    sol = []
    indxn = []
    for i in 1:length(indxkeepu)
        ff = findfirst(x-> x == indxkeepu[i], indxkeep)
        j1 = indx[ff][tet[1]]; j2 = indx[ff][tet[2]]; j3 = indx[ff][tet[3]]
        j4 = indx[ff][tet[4]]; j5 = indx[ff][tet[5]]; j6 = indx[ff][tet[6]]
        dim = visqrt(j4)*visqrt(j5)/visqrt(j3)
        fac = numchop(dim*Fsymb(j1,j2,j3,j4,j5,j6))
        ans = numchop(amps[ff]/fac )
        push!(sol,ans)
        push!(indxn,indxkeepu[i])
        #global cou = 0
        #if sign(fac) == sign(amps[i] )
        #    ans = numchop(amps[i] / fac)
        #    push!(sol,ans)
        #else
        #    ans = numchop(amps[i] / fac)
        #    push!(sol,-ans)
        #    #cou += 1
        #end
    end
    return indxn,sol#indx,sol#,couff1
end

end # module


# # DCP analysis
using Convex
x = Variable();
y = Variable();
expr = quadoverlin(x - y, 1 - max(x, y));
println("expression curvature = ", vexity(expr));
println("expression sign = ", sign(expr));


#----------------- Get the A matrix
function getDivGrad(n1,n2,n3)

        # the Divergence
        D1 = kron(sparse(I,n3,n3),kron(sparse(I,n2,n2),ddx(n1)))
        D2 = kron(sparse(I,n3,n3),kron(ddx(n2),sparse(I,n1,n1)))
        D3 = kron(ddx(n3),sparse(I,n1*n2,n1*n2))
        # DIV from faces to cell-centers
        Div = [D1 D2 D3]

        return Div*Div';
end
#----------------- 1D finite difference on staggered grid
function ddx(n)
# generate 1D derivatives
	II, JJ, VV = SparseArrays.spdiagm_internal(0 => -ones(n), 1 => ones(n));
	return sparse(II, JJ, VV, n, n+1)

end


using Convex, GLPKMathProgInterface
include("antidiag.jl")

n = 8
x = Variable((n, n), :Bin)
# We can have at most one queen on any anti-diagonal
constr = Constraint[sum(antidiag(x, k)) <= 1 for k = -n+2:n-2]
# We can have at most one queen on any diagonal
constr += Constraint[sum(diag(x, k)) <= 1 for k = -n+2:n-2]
# We must have one queen per row and one queen per column
constr += Constraint[sum(x, 1) == 1, sum(x, 2) == 1]
p = satisfy(constr)
solve!(p, GLPKSolverMIP())

# Testing
for k = -n+2:n-2
	@assert evaluate(sum(antidiag(x, k))) <= 1
	@assert evaluate(sum(diag(x, k))) <= 1
end
@assert all(evaluate(sum(x, 1)) .== 1)
@assert all(evaluate(sum(x, 2)) .== 1)


mutable struct Controller3 <: Controller
    Δt::Real
    g::Real
    M::Real

    c::Real
    k::Real
    Fs::Real

    K::AbstractMatrix
    F::Function
    control!::Function

    function Controller3(F,Δt,g,M,m,l,c,k,Fs,Q,R)
        I=1/3*m*l^2
        v1=(M+m)/(I*(M+m)+(l^2*m*M))
        v2=(I+l^2*m)/(I*(M+m)+(l^2*m*M))

        A = [0 1 0 0
            0 -k*v2 ((l*m)^2*g*v2)/(I+m*l^2) (-l*m*c*v2)/(I+m*l^2)
            0 0 0 1
            0 -(l*m*k*v1)/(M+m) l*m*g*v1 -1*c*v1
        ]

        B = [
            0
            v2
            0
            l*m*v1/(M+m)
        ]

        P = ConstrainedControl.care(A,B,Q,R)
        K = R\B'*P

        new(Δt,g,M,c,k,Fs,K,F,control!)
    end
end

function control!(mechanism,controller::Controller3,kstep)
    Δt = controller.Δt
    g = controller.g
    M = controller.M
    c = controller.c
    k = controller.k
    Fs = controller.Fs
    K = controller.K
    F = controller.F
    cart = mechanism.bodies[1]
    pend = mechanism.bodies[2]

    x = cart.state.xc[2]
    v = cart.state.vc[2]
    q = pend.state.qc
    θ = (rotation_angle(q) * rotation_axis(q))[1]
    ω = pend.state.ωc[1]

    z = [x;v;θ;ω]

    Fcart = F(z,K,kstep*Δt)
    Fcart, Fpend = friction(Fcart, 0, v, ω, k, c, Fs)

    setForce!(mechanism, geteqconstraint(mechanism, 3), [Fcart])
    setForce!(mechanism, geteqconstraint(mechanism, 4), [Fpend])

    return
end

function run3!(str; vis::Bool=true)
    open("Files/E3.jl",create=true,write=true) do file
        write(file,"
        Δt = 0.004

        # Parameters from Table 1.1
        g = 9.81
        L1 = 0.28125
        l1 = L1/2
        l = l1 # (l1-0.135)/2 + l1
        M = 4.5
        m = 0.2585
        # I = 1/3*m*l^2
        c = 0.002
        k = 0.3
        Fs = 8

        # Joint axes
        ex = [1.0;0.0;0.0]
        ey = [0.0;1.0;0.0]

        p = [0.0;0.0;l1] # joint connection point

        # Desired orientation
        θ1 = 0

        # Links
        origin = Origin{Float64}()
        cart = Box(0.1, 0.5, 0.1, M)
        pend = Box(0.055, 0.055, L1, m)

        # Constraints
        joint1 = EqualityConstraint(Prismatic(origin, cart, ey))
        joint2 = EqualityConstraint(Revolute(cart, pend, ex; p2 = -p))

        links = [cart;pend]
        constraints = [joint1;joint2]

        mech = Mechanism(origin, links, constraints, Δt = Δt, g = -g)

        xinit = 0.0
        theta1init = pi
        Q = [
            1 0 0 0
            0 1 0 0
            0 0 1 0
            0 0 0 1
        ]
        R = 1
        F(z,K) = 0

        ### Begin Student Input

        ")
        write(file, str)
        write(file, "

        ### End Student Input

        setPosition!(origin,cart,Δx = [0;xinit;0])
        setPosition!(cart,pend,p2 = -p, Δq = UnitQuaternion(RotX(theta1init)))

        controller = Controller3(F,Δt,g,M,m,l,c,k,Fs,Q,R)

        steps = Base.OneTo(Int(10/Δt))
        storage = Storage{Float64}(steps,2)

        try
            simulate!(mech,storage,controller,record = true)
        catch
            println(\"Unstable behavior\")
        end")

        if vis
            write(file, "
            visualize(mech,storage)")
        end
    end

    try
        include("Files/E3.jl")
    catch
        println("Error. Bad code.")
    end

    return
end


using AQFED.Math
export bachelierFormula
function bachelierFormula(isCall::Bool, strike::Number, forward::Number, vol::Number, tte::Number, discountDf::Number)
    sign = 1
    if !isCall
        sign = -1
    end
    if vol == 0
        return max(sign * (forward - strike), 0)
    end
    sqrtvar = sqrt(tte) * vol
    d = sign * (forward - strike) / sqrtvar
    if forward == strike
        return sqrtvar * Math.OneOverSqrt2Pi
    end
    Nd = normcdf(d)
    nd = normpdf(d)
    return sign * (forward - strike) * Nd + sqrtvar * nd
end


function _differentiate(A::Union{Tensor{T, 0}, ConstantTensor{T, 0}}, y::VariableTensor{0}, difffn) where T
    return A.value == y || A.value == [y] ? ConstantTensor(1) : ZeroTensor()
end

function _differentiate(A::VariableTensor{0}, y::VariableTensor{0}, difffn)
    return A == y ? ConstantTensor(1) : ZeroTensor()
end

function _differentiate(Z::ZeroTensor{0}, y::VariableTensor{0}, difffn)
    return Z
end

function _differentiate(add::AddOperation{0}, y::VariableTensor{0}, difffn)
    return +([difffn(child) for child in add.children]...)
end

function _differentiate(op::OuterProductOperation{0}, y::VariableTensor{0}, difffn)
    A, B = op.children
    return A*difffn(B) + difffn(A)*B
end

function _differentiate(div::DivisionOperation{0}, y::VariableTensor, difffn)
    A, B = div.children
    return (difffn(A)*B - A*difffn(B))/(B^2)
end

function _differentiate(si::SineOperation{0}, y::VariableTensor{0}, difffn)
    return Base.cos(si.children[1]) * difffn(si.children[1])
end

function _differentiate(co::CosineOperation{0}, y::VariableTensor{0}, difffn)
    return - Base.sin(co.children[1]) * difffn(co.children[1])
end

function _differentiate(ta::TangentOperation{0}, y::VariableTensor{0}, difffn)
    return difffn(ta.children[1]) / (cos(ta.children[1])^2)
end

function _differentiate(asi::ArcsineOperation{0}, y::VariableTensor{0}, difffn)
    return difffn(asi.children[1]) / sqrt(Tensor(1) - asi.children[1]^2)
end

function _differentiate(aco::ArccosineOperation{0}, y::VariableTensor{0}, difffn)
    return - difffn(aco.children[1]) / sqrt(Tensor(1) - aco.children[1]^2)
end

function _differentiate(ata::ArctangentOperation{0}, y::VariableTensor{0}, difffn)
    return difffn(ata.children[1]) / (Tensor(1) + ata.children[1]^2)
end

function _differentiate(root::RootNode, y::VariableTensor{0}, diffn)
    return RootNode(diffn(root.children[1]))
end

"""
    differentiate(A::AbstractTensor{0}, x::VariableTensor{0})

Differentiates A w.r.t x.
"""
function differentiate(A::AbstractTensor{0}, x::VariableTensor{0})
    return traversal(A, x -> x, _differentiate, nothing, x, nothing, true)
end

function differentiate(y::Number, x::VariableTensor{0})
    return 0
end

function _divergence(A::Tensor{T, 1}, vars::Tuple{Vararg{VariableTensor{0}}}, divergencefn) where T
    return +([Tensor(differentiate(A.value[i], vars[i])) for i in 1:length(vars)]...)
end

function  _divergence(A::ConstantTensor{T, 1}, vars::Tuple{Vararg{VariableTensor{0}}}, divergencefn) where T
    return +([differentiate(A.value, var) for var in vars]...)
end

function _divergence(A::TerminalTensor{1}, vars::Tuple{Vararg{VariableTensor{0}}}, divergencefn)
    return ZeroTensor(A.shape...)
end

function _divergence(op::OuterProductOperation{1}, vars::Tuple{Vararg{VariableTensor{0}}}, divergencefn)
    x, A = op.children
    if length(A.shape) == 0
        x, A = A, x
    end
    if A isa AddOperation
        return divergencefn(+([x*child for child in A.children]...))
    elseif A isa Tensor{T,1} where T
        return divergencefn(Tensor([y*x for y in A.value], A.shape...))
    elseif A isa ConstantTensor{T, 1} where T
        return divergencefn(ConstantTensor(A.value * x, A.shape...))
    elseif A isa ZeroTensor{T, 1} where T
        return ZeroTensor()
    elseif A isa OuterProductOperation
        y, B = A.children
        if length(B.shape) == 0
            y, B = B, y
        end
        return divergencefn((x * y) * B)
    else
        throw(DomainError(op, "Cannot be differentiated further"))
    end
end

function _divergence(A::AddOperation{1}, vars::Tuple{Vararg{VariableTensor{0}}}, divergencefn)
    return +([divergencefn(child) for child in A.children]...)
end

function _divergence(A::RootNode, vars::Tuple{Vararg{VariableTensor{0}}}, divergencefn)
    return RootNode(divergencefn(A.children[1]))
end

"""
    divergence(A::AbstractTensor{1}, vars::Vararg{VariableTensor{0}})

Computes the divergence of a vector-shaped tensor A w.r.t vars.
"""
function divergence(A::AbstractTensor{1}, vars::Vararg{VariableTensor{0}})
    if dim(A.shape[1]) != length(vars)
        throw(DimensionMismatch(string(vars, " does not have enough variables to gradient ", A)))
    end
    return traversal(A, x -> x, _divergence, nothing, vars, nothing, true)
end


using GeohashHilbert
using Random
using Test
using CSV

const GHH = GeohashHilbert
const ALL_BITS_PER_CHAR = [2, 4, 6]

randlat() = -90 + 180 * rand()
randlon() = -180 + 360 * rand()
randprec() = rand(1:10)

# converting between int/string and back should reproduce starting values
function test_int_str_conversion()
    for _ in 1:200 # 200 independent tests
        x = rand(0:50_000)
        bpc = rand([2,4,6])
        base = 2^bpc
        chars_needed = ceil(Int, log(base, x + 1))
        nchar = rand(chars_needed : (chars_needed + 5))
        x_str = GHH.int_to_str(x, nchar, bpc)
        @test length(x_str) == nchar
        x_prime = GHH.str_to_int(x_str, bpc)
        @test x == x_prime
    end
    return nothing
end

# converting between xy and integer should be lossless
function test_int_xy_conversion()
    # many independent random tests of normal behavior
    for _ in 1:1000
        k = rand(2:20)
        n = 2^k
        int = rand(0:(n^2 - 1))
        x, y = GHH.int_to_xy(int, n)
        int_prime = GHH.xy_to_int(x, y, n)
        @test int_prime == int
    end
    # test illegal input
    @test_throws DomainError GHH.int_to_xy(-1, 8) # t too small
    @test_throws DomainError GHH.int_to_xy(100, 8) # t too large
    return nothing
end

# For lat/lons which are centers of geohash cells, encoding then decoding
# should match input coordinates.
# We can find lat/lons which are centers of cells by encoding and then decoding
# random points.
function test_cell_centers_encode_decode(bits_per_char = 2)
    for _ in 1:50
        lon, lat = randlon(), randlat()
        prec = randprec()
        geohash = GHH.encode(lon, lat, prec, bits_per_char)
        cell_center = GHH.decode(geohash, bits_per_char)
        geohash2 = GHH.encode(cell_center..., prec, bits_per_char)
        @test geohash == geohash2
        cell_center2 = GHH.decode(geohash2, bits_per_char)
        @test cell_center2 == cell_center
    end
    return nothing
end

# Make sure we match Python geohash hilbert on encoding, particularly on edges
# and corners of cells.
function test_match_python_encode(bits_per_char = 2)
    # python_hashes.csv has columns lon, lat, prec, bpc [bits per char], geohash
    typedict = Dict(
        :lon => Float64,
        :lat => Float64,
        :prec => Int,
        :bpc => Int,
        :geohash => String
    )
    python_hashes = CSV.File(joinpath(@__DIR__, "python_hashes.csv"); types = typedict)
    for row in python_hashes
        julia_hash = GHH.encode(row.lon, row.lat, row.prec, row.bpc)
        @test julia_hash == row.geohash
    end
    return nothing
end

# Illegal arguments should raise exceptions.
# This is not an exhaustive list of all kinds of illegal args.
function test_illegal_arguments()
    @test_throws Exception GHH.int_to_str(5, 5, 3) # 3 illegal bits per char
    @test_throws Exception GHH.str_to_int("hey there", 2) # illegal characters
    @test_throws Exception GHH.str_to_int("0123123", 3) # illegal bits per char
    @test_throws DomainError encode(-200, 0, 4, 4) # illegal longitude
    @test_throws DomainError encode(0, 100, 4, 4) # illegal latitude
    @test_throws DomainError encode(0, 0, 0, 4) # illegal precision
    @test_throws DomainError encode(0, 0, 11, 6) # too much precision
    @test_throws Exception encode(0, 0, 2, -1) # illegal bits per char
    @test_throws DomainError GHH.int_to_xy(16, 4) # first argument is too large
    @test_throws DomainError GHH.find_quadrant(5, GHH.ULeft) # illegal quadrant
    @test_throws DomainError GHH.xy_to_int(5, 1, 4) # x too large
    @test_throws DomainError GHH.xy_to_int(2, 0, 4) # y too small
end

println("Greetings tester; your test will now begin.")
Random.seed!(47)
test_int_str_conversion()
test_int_xy_conversion()
for bpc in ALL_BITS_PER_CHAR
    test_cell_centers_encode_decode(bpc)
end
test_match_python_encode()
test_illegal_arguments()
println("Great job, you passed!")


#    Problem : GROUP A
#    *********
#    This problem is quartic and has a banded Hessian with bandwidth = 9.
#
#    Origonal SIF Source:
#    Problem 61 in
#    A.R. Conn, N.I.M. Gould, M. Lescrenier and Ph.L. Toint,
#    "Performance of a multifrontal scheme for partially separable
#    optimization",
#    Report 88/4, Dept of Mathematics, FUNDP (Namur, B), 1988.
#
#    Implementation translated from Source:
#    http://eprints.tsu.ge/234/14/Tests%20collection-K-F.pdf
#
#    BDQRTIC.SIF classification SUR2-AN-V-0
#
#    Number of variables (variable)

f = (x) -> begin
    fx = 0.0
    xn_sqr = x[lastindex(x)]^2
    for i in firstindex(x):lastindex(x)-4
        α = 3.0 - 4.0x[i]
        γ = x[i]^2 + 2.0x[i+1]^2 + 3.0x[i+2]^2 + 4.0x[i+3]^2 + 5.0xn_sqr
        fx += α*γ + γ^2
    end
    return fx
end

g! = (g, x) -> begin
    xn = x[lastindex(x)]
    cxn_sqr = 5.0xn^2
    gn = 0.0
    for i in firstindex(x):lastindex(x)-4
        α = 3.0 - 4.0x[i]
        γ = x[i]^2 + 2.0x[i+1]^2 + 3.0x[i+2]^2 + 4.0x[i+3]^2 + cxn_sqr
        g[i] += -8.0α + 4.0γ*x[i];
        g[i+1] += 8.0γ*x[i+1];
        g[i+2] += 12.0γ*x[i+2];
        g[i+3] += 16.0γ*x[i+3];
        gn += 20.0γ*xn;
    end
    g[lastindex(x)] = gn
    return g
end

fg! = (g, x) -> begin
    fx = 0.0
    xn = x[lastindex(x)]
    cxn_sqr = xn^2
    gn = 0.0
    for i in firstindex(x):lastindex(x)-4
        α = 3.0 - 4.0x[i]
        γ = x[i]^2 + 2.0x[i+1]^2 + 3.0x[i+2]^2 + 4.0x[i+3]^2 + cxn_sqr
        fx += α*γ + γ^2
        g[i]   += -8.0α + 4.0γ*x[i];
        g[i+1] += 8.0γ*x[i+1];
        g[i+2] += 12.0γ*x[i+2];
        g[i+3] += 16.0γ*x[i+3];
        gn += 20.0γ*xn;
    end
    g[lastindex(x)] = gn
    return fx, g
end

init = (n::Int=5000) -> begin
	n < 5 && @warn("BDQRTIC: number of variables must be ≥ 5")
	n = max(n, 5)

    return n, ones(n)
end

@warn "TODO: Debug BDQRTIC"
#TestSet["BDQRTIC"] = UncProgram("BDQRTIC", f, g!, fg!, init)


# Benders Algorithm for Mixed Integer Linear Programming
# To Solve the News Boy Problem
# Edward J. Xu
# Feb 28th, 2019
using JuMP
using GLPKMathProgInterface
# Benders Decomposition algorithm
demand = [12, 14, 16, 18, 20, 22, 24, 26, 28, 30] # Demand of newspapers in each scenario
lengthS = length(demand)
prob = [0.05, 0.10, 0.10, 0.10, 0.15, 0.15, 0.10, 0.10, 0.10, 0.05 ] # probability of scenario
c = 20  # purchase price
p = 70  # selling price
h = 10  # scrap value
y = 20
let
    mas = Model(solver = GLPKSolverMIP())
    @variable(mas, q)
    @variable(mas, 0 <= y <= 40, Int)
    # add master-problem objective
    @objective(mas, Min, (c - h) * y + q)
    boundUp = Inf
    boundLow = - Inf
    epsilon = 0.01
    # definition of sub variables
    uBar = ones(10)
    # initial value of master variables
    yBar = 0
    #
    objMas = 0
    objSub = 0
    timesIteration = 1
    while (boundUp - boundLow > epsilon && timesIteration < 10)
        # sub-problem
        let
            sub = Model(solver = GLPKSolverLP())
            # create full sub-problem model, using the value of master variables yBar
            @variable(sub, u[1: 10] >= 0)
            @objective(sub, Max, - u)
            consX_1 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[1] * (h - p))
            consX_2 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[2] * (h - p))
            consX_3 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[3] * (h - p))
            consX_4 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[4] * (h - p))
            consX_5 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[5] * (h - p))
            consX_6 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[6] * (h - p))
            consX_7 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[7] * (h - p))
            consX_8 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[8] * (h - p))
            consX_9 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[9] * (h - p))
            consX_10 = @constraint(sub, - (u_1 + u_2 + u_3 + u_4 + u_5 + u_6 + u_7 + u_8 + u_9 + u_10) <= prob[10] * (h - p))
            solve(sub)
            objSub = getobjectivevalue(sub)
            u1Bar = getvalue(u_1)
            u2Bar = getvalue(u_2)
            u3Bar = getvalue(u_3)
            u4Bar = getvalue(u_4)
            u5Bar = getvalue(u_5)
            u6Bar = getvalue(u_6)
            u7Bar = getvalue(u_7)
            u8Bar = getvalue(u_8)
            u9Bar = getvalue(u_9)
            u10Bar = getvalue(u_10)
        end
        boundUp = min(boundUp, objSub + (c - h) * yBar)
        # master-problem only add the feasibility constraint
        @constraint(mas, sum((- demand .- y).* [u1Bar, u2Bar, u3Bar, u4Bar, u5Bar, u6Bar, u7Bar, u8Bar, u9Bar, u10Bar]) <= q)
        solve(mas)
        objMas = getobjectivevalue(mas)
        yBar = getvalue(y)
        boundLow = max(objMas, boundLow)
        println("timesIteration: $(timesIteration)\t boundUp: $(boundUp)\t boundLow: $(boundLow)")
        timesIteration += 1
    end
end
println("Correct Ending")


"""
    SumOfSquaresError(NeuralNetwork)

Sum of squares error loss function.

    Usage:
```julia-repl
SumOfSquaresError(NeuralNetwork)
```
Arguments:
- `NeuralNetwork` : The NeuralNetwork structure.
"""
function SumOfSquaresError(NeuralNetwork)
	loss = 0.0

	for i in 1:size(NeuralNetwork.y, 1)
		loss += (NeuralNetwork.y[i] - NeuralNetwork.output[i]) ^ 2.0
    end

    return loss
end


"""
素数判定
"""
function is_prime_simple(n::Int64)
    if n < 2
        return false
    elseif n == 2
        return true
    else
        for i in 2:n-1
            #println(i, "...")
            if n % i == 0
                return false
            end
            #println("わりきれない")
        end
    end
    return true
end

function do_2(n₀::Int64, n₁::Int64)
    ns = n₀:n₁
    prime_list = is_prime_simple.(ns)
    num = prime_list' * prime_list
    println(n₀, " から ", n₁ ," に素数は", num, " 個ある")
end

#is_prime_simple(999961)
#@time is_prime_simple(999961)  # 100万以下で最大の素数

#do_2(1, 10)
@time do_2(1, 100000)


using NGWP, LightGraphs, MTSG, Plots, LaTeXStrings

## Build Graph
N = 16; G = path_graph(N)
X = zeros(N,2); X[:, 1] = 1:N
L = Matrix(laplacian_matrix(G))
𝛌, 𝚽 = eigen(L); 𝚽 = 𝚽 .* sign.(𝚽[1,:])'
W = 1.0 * adjacency_matrix(G)

## preliminaries for PC-NGWP
Gstar_Sig = GraphSig(W)
G_Sig = GraphSig(W, xy = X)
GP_dual = partition_tree_fiedler(Gstar_Sig; swapRegion = false)
GP_primal = pairclustering(𝚽, GP_dual)

## Plots
function path_pc_plot!(W, X; ind = 1:size(X, 1), c = :teal)
    # plot(size = (500, 50), framestyle = :none, xlim = [1, N+1.5], ylim = [-0.2, 0.3])
    gplot!(W, X, width = 1, color = :blue, style = :solid)
    scatter_gplot!(X; ms = 6, c = :teal)
    annotate!(X[:, 1], X[:, 2] .- 0.2, [text(string(n), :top, 7) for n = 1:N])
    scatter_gplot!(X[ind, :]; ms = 6, c = c)
    plt = plot!(ratio = :auto)
    return plt
end


function demoBurglar()

    burglar,earthquake,alarm,radio=1,2,3,4 # Variable order is arbitary
    yes,no= 1,2 # define states, starting from 1

    # This defines the meaning of the variables and their states.
    # It is not necessary for the computation, but is useful for
    # printing the results in a meaningful form:
    DictVariable=Dict{Integer,DiscreteVariable}()
    DictVariable[burglar]=DiscreteVariable("burglar",["yes","no"])
    DictVariable[earthquake]=DiscreteVariable("earthquake",["yes","no"])
    DictVariable[alarm]=DiscreteVariable("alarm",["yes","no"])
    DictVariable[radio]=DiscreteVariable("radio",["yes","no"])

    # Define the discrete probability tables:
    pot=Array{PotArray}(4) # there are 4 discrete potentials (arrays)
    pot[burglar]=PotArray(burglar,[0.01 0.99])
    pot[earthquake]=PotArray(earthquake,[0.000001 1-0.000001])

    table=zeros(2,2)
    table[yes,yes]=1
    table[no,yes]=0
    table[yes,no]=0
    table[no,no]=1
    pot[radio]=PotArray([radio earthquake],table)

    table=zeros(2,2,2)
    table[yes,yes,yes]=0.9999
    table[yes,yes,no]=0.99
    table[yes,no,yes]=0.99
    table[yes,no,no]=0.0001
    table[no,:,:]=1-table[yes,:,:]
    pot[alarm]=PotArray([alarm burglar earthquake],table)

    jointpot=prod(pot)

    println("p(burglar|alarm=yes):")
    show(condpot(setpot(jointpot,alarm,yes),burglar),DictVariable)

    println("p(burglar|alarm=yes,radio=yes):")
    show(condpot(setpot(jointpot,[alarm radio],[yes yes]),burglar),DictVariable)

    L=dag(pot)
    #try
        # This is just for plotting the DAG:
        DictPlot=Dict{Integer,VariablePlot}()
        DictPlot[burglar]=VariablePlot("Burglar",x=0,y=1,nodesize=0.3)
        DictPlot[earthquake]=VariablePlot("Earthquake",x=3,y=1,nodesize=0.35)
        DictPlot[alarm]=VariablePlot("Alarm",x=0,y=0,nodesize=0.2)
        DictPlot[radio]=VariablePlot("Radio",x=3,y=0,nodesize=0.2)

        PlotGraph(DictPlot,L)

        #PlotGraph(PlaceVertices(DictPlot,L,scale=5),L,arrowsize=0.2) # automatic node placement

    #end

end


module BnpUtil

@doc raw"""
Stick-breaking function.

Arguments
=========
- `v`: A vector of length $K - 1$, where $K > 1$.

Return
======
- A simplex (w) of dimension $K$. Where ∑ₖ wₖ = 1, and each wₖ ≥ 0.
"""
function stickbreak(v)
    K = length(v) + 1
    cumprod_one_minus_v = cumprod(1 .- v)

    eta = [if k == 1
               v[1]
           elseif k == K
               cumprod_one_minus_v[K-1]
           else
               v[k] * cumprod_one_minus_v[k-1]
           end
           for k in 1:K]

    return eta
end

end  # End of module BnpUtil


@testset "Kronecker products" begin

    A = randn(4, 4)
    B = Array{Float64, 2}([1 2 3;
         4 5 6;
         7 -2 9])
    C = rand(5, 6)
    D = rand(4, 4)

    v = rand(12)

    K = A ⊗ B
    K3 = kronecker(A, B, C)

    X = kron(A, B)  # true result

    @testset "Types and basic properties" begin
        @test issquare(A)
        @test !issquare(C)

        @test getmatrices(A)[1] === A

        @test !issymmetric(A)

        @test issquare(K)
        @test !issymmetric(K)

        @test K ≈ X

        @test order(A) == 1
        @test order(K) == 2

        @test eltype(K) <: Float64

        @test K isa AbstractKroneckerProduct
        @test K isa AbstractKroneckerProduct{Float64}

        @test K isa AbstractMatrix{Float64}
    end

    @testset "Linear algebra" begin
        @test tr(K) ≈ tr(X)
        @test det(K) ≈ det(X)
        @test !isposdef(K)
        @test transpose(K) ≈ transpose(X)
        @test conj(K) ≈ conj(X)
        @test K' ≈ X'
        @test inv(K) ≈ inv(X)

        # test on pos def functions
        As = A' * A
        Bs = B * B'
        @test logdet(As ⊗ Bs) ≈ log(det(As ⊗ Bs)) ≈ log(det(kron(As, Bs)))
    end

    @testset "Mismatch errors" begin
        P, Q = rand(10, 4), rand(4, 5)
        Kns = P ⊗ Q
        @test_throws DimensionMismatch inv(Kns)
        @test_throws DimensionMismatch det(Kns)
        @test_throws DimensionMismatch Kns * [1, 2, 3]
    end

    @testset "Higher order" begin
        @test order(K ⊗ A) == 3
        @test order(K3) == 3
        @test collect(K3) ≈ kron(X, C)

    end

    @testset "Kronecker powers" begin
        Kpow = ⊗(A, 5)
        @test order(Kpow) == 5
        @test size(Kpow, 1) == 4^5
        @test Kpow[1,1] ≈ A[1,1]^5
    end

    @testset "kron" begin
        @test kron(A ⊗ B, C) ≈ kron(A, B, C)
        @test kron(A, B ⊗ C) ≈ kron(A, B, C)
        @test kron(A ⊗ B, C ⊗ D) ≈ kron(A, B, C, D)
        @test collect((A⊗B) ⊗ (C⊗D)) ≈ kron(A, B, C, D)
    end

    @testset "Mixed product" begin
        A = rand(5, 4)
        B = rand(2, 3)
        C = rand(4, 6)
        D = rand(3, 4)

        K1 = (A ⊗ B)
        K2 = (C ⊗ D)

        @test K1 * K2 ≈ collect(K1) * collect(K2)

        @test_throws DimensionMismatch (A ⊗ C) * (B ⊗ D)
        @test_throws DimensionMismatch (A ⊗ D) * (C ⊗ B)
    end

    @testset "Add to dense" begin
        @test K + X ≈ Matrix(K) + X
        @test X + K ≈ X + Matrix(K)
    end

    @testset "Scalar multiplication" begin
        @test 3.0K ≈ 3.0X
        @test K * 2 ≈ 2X
        @test π * K3 ≈ π * collect(K3)
        @test 3.0K isa AbstractKroneckerProduct
        @test K * 2 isa AbstractKroneckerProduct
        @test 2(K ⊗ K) isa AbstractKroneckerProduct
    end

    @testset "Inplace scalar multiplication" begin
        A = rand(2, 2)
        B = rand(3, 4)

        K = copy(A) ⊗ copy(B)
        lmul!(3, K)
        rmul!(K, 2)
        @test K.A ≈ 3A
        @test K.B ≈ 2B
        @test K ≈ 6kron(A, B)
    end

    @testset "Solving Linear Systems" begin
        n = 8
        m = 16
        # 1) testing square kronecker product with square A, B
        A = randn(n, n)
        B = randn(m, m)
        K = A ⊗ B
        x = randn(n * m)
        b = K*x
        @test K\b ≈ x
        @test (x'*K)/K ≈ x'

        # testing least squares solution
        function test_ls_solve(size_A, size_B)
            A = randn(size_A)
            B = randn(size_B)
            x = randn(size(A, 2) * size(B, 2))
            K = kronecker(A, B)
            b = K*x
            b .+= randn(size(b)) # this moves b out of range(K), necessitating least-squares
            xls = K\b
            return K'*(K*xls) ≈ K'b # test via normal equations
        end

        # 2) kronecker product is square, but A, B aren't
        size_A = (m, n)
        size_B = (n, m)
        @test test_ls_solve(size_A, size_B)

        # 3) kronecker product is not square, dimension of x is larger than b
        size_A = (n, n)
        size_B = (n, m)
        @test test_ls_solve(size_A, size_B)

        # 4) kronecker product is not square, dimension of x is smaller than b
        size_A = (m, n)
        size_B = (m, m)
        @test test_ls_solve(size_A, size_B)
    end
end


function compute(n::Int, digits::Vector{Int})::String
    k = length(digits)
    result = ""
    remain = n - 1
    for i ∈ 1:k
        order = factorial(k - i)
        j = remain ÷ order + 1
        result = string(result, digits[j])
        remain %= order
        deleteat!(digits, j)
    end
    return result
end


include("../src/driver.jl")
using PyPlot
using Printf
function plot_lyapunov_exponents()
	npts = 10
	w = rand(npts)
	les = rand(npts)
	η = [0.001,0.01,0.02,0.025,0.05]
	fig, ax = subplots()
	ax.xaxis.set_tick_params(labelsize=30)
	ax.yaxis.set_tick_params(labelsize=30)

	for (i, ηi) = enumerate(η)
		w .= rand(npts)
		@show i, ηi
		for j = 1:npts
			les[j] = lyap_exp(w[j], ηi)
		end
		ax.plot(w, les, label=latexstring(L"\eta = ",@sprintf("%0.3f", ηi)))
	end
end
function plot_bifurcation()
	npts = 50
	wstar = rand(npts)
	nSteps = 50000
	η = LinRange(0.0001, 0.1, 20)
	fig, ax = subplots()
	ax.xaxis.set_tick_params(labelsize=30)
	ax.yaxis.set_tick_params(labelsize=30)

	for (i, ηi) = enumerate(η)
		@show i, ηi
		wstar .= rand(npts)
		for j = 1:npts
			wstar[j] = run(wstar[j], ηi, nSteps)
		end
		ax.plot(ηi*ones(npts), wstar, "bo", ms=3.0)
	end
	ax.grid(true)
	ax.set_xlabel(L"$ \eta $", fontsize=30)
	ax.set_ylabel(L"$ w^* $", fontsize=30)
end


using DSGE
using HDF5

include("../util.jl")
path = dirname(@__FILE__)

# Test in generic case
# Global minimum at (a, a^2)
a = 1
b = 100
function rosenbrock(x::Vector)
    return (a-x[1])^2.0 + b*(x[2]-x[1]^2.0)^2.0
end

# this is the actual gradient of the Rosenbrock function
function rosenbrock_grad(x::Array)
    dr = similar(x)
    dr[1] = -2*(a-x[1]) - 2b*(2x[1])*(x[2]-x[1]^2.0)
    dr[2] = 2b*(x[2]-x[1]^2.0)
    badg = false
    return dr, badg
end

# A really bad guess
x_init = [10.0, -9.0]
x_expected = [a, a^2.0]

res_real_grad, _ = csminwel(rosenbrock, rosenbrock_grad, x_init)
@test_approx_eq x_expected res_real_grad.minimum
res_numeric_grad, _ = csminwel(rosenbrock, x_init)
@test_approx_eq_eps x_expected res_numeric_grad.minimum 1e-8

nothing


module ExNonpar
using SDE
using SDE.Schauder
require(Pkg.dir("SDE","src", "SDEPlot.jl"))
using SDEPlot

bex1(s,x) = 4(sin(2pi*x) - x)
bex2(s,x) = -4min(x+.5, 0) - 4max(x-.5, 0)
si(s,x) = 1.
range(a) = (minimum(a), maximum(a))

function example(L, logN,T, f, B; VERBOSE = 0)

	global y, t
	srand(42)
	beta = 0.5
	a = -2
	b = 2
	K = 0
	if (B=="B1"); K = 1;end # Schauder basis with constant
	if (B=="B2"); K = 2;end # Schauder basis plus two affine component
				# otherwise Schauder basis

	VERBOSE > 0 && println("L $L, beta $beta")
	N = 10^logN
	dt = T/N
	P = Diffusion{1}(f, si, ())
    tt = linspace(0., T, N+1)
    W = sample(tt, Wiener())
    u = 0.5(b+a)
    y = euler(u, W, P).yy

	if (B == "B1")
		truec = fe_transfB1(x-> f(0., x), a,b, L)
	elseif (B == "B2")
		truec = fe_transfB2(x-> f(0., x), a,b, L)
	else
		truec = fe_transf(x-> f(0., x), a,b, L)

	end
    VERBOSE > 0 && println("true drift f.e. coefficients",truec')

	println("range y $(range(y))")

	post = bayes_drift(y, dt, a, b, L, 10ones(K), beta , B)

	pl = visualize_posterior(post,x -> f(0.,x), 1.96)

	(post, pl)
end
println("Try: post = ExNonpar.example(5,5, 100, ExNonpar.bex1, \"B2\"); Winston.display(post[2])")

end


println("Starting precompilation")
using OrdinaryDiffEq
println("Starting tests")
using Flux, DiffEqFlux
using Test
using Distributions
using Distances
using Zygote
using DistributionsAD
using LinearAlgebra

##callback to be used by all tests
function cb(p,l)
    @show l
    false
end

###
#test for default base distribution and deterministic trace CNF
###

nn = Chain(Dense(1, 1, tanh))
data_train = [Float32(rand(Beta(7,7))) for i in 1:100]
tspan = (0.0,10.0)
cnf_test = DeterministicCNF(nn,tspan,Tsit5())

function loss_adjoint(θ)
    logpx = [cnf_test(x,θ) for x in data_train]
    loss = -mean(logpx)
end

res = DiffEqFlux.sciml_train(loss_adjoint, 0.01.*cnf_test.p,
                                        ADAM(0.01), cb=cb,
                                        maxiters = 100)

θopt = res.minimizer
data_validate = [Float32(rand(Beta(7,7))) for i in 1:100]
actual_pdf = [pdf(Beta(7,7),r) for r in data_validate]
#use direct trace calculation for predictions
learned_pdf = [exp(cnf_test(r,θopt)) for r in data_validate]

@test totalvariation(learned_pdf, actual_pdf)/100 < 0.25

###
#test for alternative base distribution and deterministic trace CNF
###

nn = Chain(Dense(1, 3, tanh), Dense(3, 1, tanh))
data_train = [Float32(rand(Normal(6.0,0.7))) for i in 1:100]
tspan = (0.0,10.0)
cnf_test = DeterministicCNF(nn,tspan,Tsit5(),basedist=MvNormal([0.0],[2.0]))

function loss_adjoint(θ)
    logpx = [cnf_test(x,θ) for x in data_train]
    loss = -mean(logpx)
end

res = DiffEqFlux.sciml_train(loss_adjoint, 0.01.*cnf_test.p,
                                          ADAM(0.01), cb = cb,
                                          maxiters = 300)

θopt = res.minimizer
data_validate = [Float32(rand(Normal(6.0,0.7))) for i in 1:100]
actual_pdf = [pdf(Normal(6.0,0.7),r) for r in data_validate]
learned_pdf = [exp(cnf_test(r, θopt)) for r in data_validate]

@test totalvariation(learned_pdf, actual_pdf)/100 < 0.25

###
#test for multivariate distribution and deterministic trace CNF
###

nn = Chain(Dense(2, 2, tanh))
μ = ones(2)
Σ = 7*I + zeros(2,2)
mv_normal = MvNormal(μ, Σ)
data_train = [Float32.(rand(mv_normal)) for i in 1:100]
tspan = (0.0,10.0)
cnf_test = DeterministicCNF(nn,tspan,Tsit5())

function loss_adjoint(θ)
    logpx = [cnf_test(x,θ) for x in data_train]
    loss = -mean(logpx)
end

res = DiffEqFlux.sciml_train(loss_adjoint, cnf_test.p,
                                          ADAM(0.1), cb = cb,
                                          maxiters = 300)

θopt = res.minimizer
data_validate = [Float32.(rand(mv_normal)) for i in 1:100]
actual_pdf = [pdf(mv_normal,r) for r in data_validate]
learned_pdf = [exp(cnf_test(r, θopt)) for r in data_validate]

@test totalvariation(learned_pdf, actual_pdf)/100 < 0.25

###
#test for default multivariate distribution and FFJORD with regularizers
###

nn = Chain(Dense(1, 1, tanh))
data_train = [Float32(rand(Beta(7,7))) for i in 1:100]
tspan = (0.0,10.0)
ffjord_test = FFJORD(nn,tspan,Tsit5())

function loss_adjoint(θ)
    logpx = [-ffjord_test(x,θ,true)[1] + 0.1*(ffjord_test(x,θ,true)[2]+ffjord_test(x,θ,true)[3]) for x in data_train]
    loss = mean(logpx)
end
res = DiffEqFlux.sciml_train(loss_adjoint, 0.01.*ffjord_test.p,
                                        ADAM(0.01), cb=cb,
                                        maxiters = 100)

θopt = res.minimizer
data_validate = [Float32(rand(Beta(7,7))) for i in 1:100]
actual_pdf = [pdf(Beta(7,7),r) for r in data_validate]
#use direct trace calculation for predictions
learned_pdf = [exp(ffjord_test(r,θopt,false,false)[1]) for r in data_validate]

@test totalvariation(learned_pdf, actual_pdf)/100 < 0.25