### A Pluto.jl notebook ###
# v0.12.21

using Markdown
using InteractiveUtils

# This Pluto notebook uses @bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of @bind gives bound variables a default value (instead of an error).
macro bind(def, element)
    quote
        local el = $(esc(element))
        global $(esc(def)) = Core.applicable(Base.get, el) ? Base.get(el) : missing
        el
    end
end

# ╔═╡ 89fcfe1c-913a-11eb-1b04-0d87fd8dbcbe
#Set up the working environment
begin
	using Pkg
	Pkg.activate("..")
	using Plots, PlutoUI,LaTeXStrings
end

# ╔═╡ 9f7c1628-9139-11eb-0d69-f910a995eca7
md"""## Differential Equations

When modelling growth, we rely on **differential equations** that describe the change in some variable $y$ with respect to some other variable $x$.

$$\frac{dy}{dx} = f(y)$$

Here, $\frac{dy}{dx}$ represents the **rate of change in $y$ with respect to $x$**, and $f(y)$ is some function that depends on $y$.

Solving differential equations to obtain a function that expresses $y$ in terms of $x$ can be relatively straight-forward, but this is generally not the case. So why bother? It turns out that in the natural world for many processes *it tends to be much easier to describe what the change in some quantity depends on*, rather than coming up with an *explicit* formula how $y$ changes as $x$ varies proves difficult.

To make the mathematical language a bit less abstract, let us rewrite the equation above, so that the the variables more directly correspond to what we are talking about when modelling growth:

$$(1)\frac{dP}{dt} = f(P)$$

Here, $\frac{dP}{dt}$ denotes the growth rate of some population $P$ over time, which depends on the population size $P$ in a yet-to-be specified manner $f(P)$."""

# ╔═╡ babc9748-9139-11eb-0da1-0d942d591481
md"""## The simplest possible case

To familiarise ourselves with growth models, let's look at a very simple model, while ignoring the fact that the model does not make sense when modelling biological populations.

$$(2)\frac{dP}{dt} = r$$

Here, our growth rate $\frac{dP}{dt}$  is constant and equal to a growth rate constant $r$.

### Rate Plots
A useful and simple trick to understand how systems described by differential equations behave involves looking at the **rate plot**, where we plot how $\frac{dP}{dt}$  changes as the population size $P$ varies.

> **Exercise:** Sketch a rate plot for the system described by equation (2)."""

# ╔═╡ ee2bb580-9145-11eb-2bc2-dbd9aae68be9
md"""
Since the rate of growth is constant,  changes by the same amount in the same timespan, regardless of how much time has already passed. Thus the graph of  versus  is a straight line that intersects the y-Axis at  and grows with a slope of .
"""

# ╔═╡ 001fc24a-9146-11eb-39b9-955dac6bcf58
md"""
### Solving a simple differential equation
Nice - We've analysed a simple differential equation using a rate plot and deduced an approximate plot of $P$ over time from it. Now on to the somewhat harder part - solving the differential equation mathematically...

In general, differential equations can be solved by separating the variables ($P$ and $t$ in our case) and then integrating both sides with respect to the variables. If you're unfamiliar with integration, but are familiar with differentiation, as taught in highschool, it should suffice to say that integration is essentially differentiation in reverse.

Upon separating our variables in equation (2), we get

$$(3) \int dP = r \int dt $$

Doing differentiation in reverse for each side, we find

$$(4) P + C_1 = rt + C_2 $$

We find the derivative of a variable by lowering its exponent by 1.

$$(5) f(x) = x^n, f'(x) = x^{n-1}$$

Hence, integrating the expression above (which can be viewed as containing $P^0=1$ and $t^0=1$, respectively) with respect to some variable amounts to raising the exponents of our "invisible" variables by 1. You may wonder where the two constants come from. They need to be included, because constants disappear in differentiation and this needs to be accounted for by creating them in the course of integration. However, we don't need to worry too much about the constants for now. In fact, they behave a bit like plasticine and can be liberally lumped together for now, as we only need to determine the constant we're left with once we're done with our arithmetic.

By absorbing $C_1$ into $C_2$, we get a new constant $C = C_1 - C_2$, and equation (4) rearranges to

$$(6) P(t) = rt + C $$

Voilá, here's an equation for a straight line!

To figure out the value of our constant $C$, we need one more trick: We set $P(0)=P_0$ and $t=0$, and we get

$$(7) P_0 = C$$

Hence our final equation is

$$(8) P(t) = rt + P_0$$
"""

# ╔═╡ ad08de50-913d-11eb-3f87-fb27a282bdb3
md"""### Interactive Simulation
*Change the sliders to adjust the corresponding values*:
- growth rate constant ($r$) $(@bind r_0 Slider(-0.5:0.01:0.5, default=0.01, show_value=true))
- initial population size($P_0$) $(@bind P0_0 Slider(0:5:100, default=20, show_value=true))
"""

# ╔═╡ f58d273e-9139-11eb-0fe0-b11e90166682
begin
	# Our differential equation
	dPdt_0(P, r) = r
	# A vector holding different population size values
	population_0 = 0:0.5:100
	# A vector holding our growth rate values for different population sizes
	growthrate_0 = [dPdt_0(p,r_0) for p in population_0]

	# The solution to our differential equation
	P_0(t, r, P0) = P0 + r*t
	# A vector holding values for different timepoints
	time_0 = 0:0.5:50
	# A vector holding our simulated population sizes
	sim_pop_0 = [P_0(t,r_0,P0_0) for t in time_0]

	rateplot_0 = plot(population_0, growthrate_0,
				xlims = (0,100),
				ylims = (-1.2, 1.2),
				title = "Rateplot",
				label = "",
				xlabel = "P", ylabel = L"$\frac{dP}{dt}$")
	simplot_0 = plot(time_0, sim_pop_0,
				color = "red",
				ylims = (0,100),
				title = "Simulation",
				label = "",
				xlabel = "t", ylabel = "P")
	plot(rateplot_0, simplot_0, layout = (1, 2),fontfamily="Computer Modern")
end

# ╔═╡ aa53e7e0-913d-11eb-32ab-4fc1ce2b72ab
md"""
## 2. A more realistic case

The growth of most biological populations will at the very least depend on the number of individuals already present. After all, having twice as many bacteria that could potentially divide, we'd expect the growth rate to increase accordingly. To capture this relationship, we can write

$$(9) \frac{dP}{dt} = rP$$

> **Exercise:** Sketch a rate plot for the system described by equation (9).

From our rate plot, we can see that in this case, the growth rate increases linearly as our population grows.

> **Exercise:** Try to sketch a plot that shows how $P$ changes over time $t$, starting from an arbitrarily chosen initial population size $P_0$.

Maybe you were able to figure out that since the growth rate increases as $P$ grows, the growth of the population should accelerate in exponential manner. To get an explicit formula for $P(t)$, and double-check our intuition, let us use the methods we've learned above.

Separation of variables in equation (9) yields

$$(10) \int \frac{1}{P} dP = r \int dt$$

Integrating equation 10 (and maybe consulting a trusty resource for common antiderivatives to find $\int \frac{1}{x} dx = ln(x) + C$), we get

$$(11) ln(P) + C_1 = rt + C_2$$

Again, we can absorn $C_1$ and $C_2$ into a new constant $C = C_2 - C_1$ to obtain

$$(12) ln(P) = rt + C $$

Now, to extract our population size $P$ from the somewhat unyieldy $ln(P)$, we make use of the following identities $e^{ln(x)} = x$, $e^{a+b} = e^a \cdot e^b$ and obtain:

$$(13) P(t) = e^{rt + C} = e^C \cdot e^{rt}$$

Exploiting the plasticine-like qualities of our integration constant and settin $e^C = A$, we clean up equation (13) and get

$$(14) P(t) = A \cdot e^{rt}$$

To figure out the value of $A$, we employ the same trick as above and set $t = 0$ and $P(0) = P_0$.

$$(15) P_0 = A \cdot e^{r \cdot 0} = A$$

Hence our solution to equation (9) is

$$(16) P(t) = P_0 \cdot e^{rt}$$

This equation ought to look familiar. It is the equation for exponential growth (and decay, if $r<0$) and abounds in science (e.g. radioactive decay). Yet, to model populations in biology, it is of very limited use, as it suggests a world of unlimited resources. Eventually, any population is going to encounter limiting factors - it may run out of space or food, or may find itself fall prey to predators or parasites.
"""

# ╔═╡ 008851e2-9147-11eb-3b1c-0ff07fd4e84f
md"""### Interactive Simulation
*Change the sliders to adjust the corresponding values*:
- growth rate constant ($r$) $(@bind r_1 Slider(-0.25:0.01:0.25, default=0.01, show_value=true))
- initial population size($P_0$) $(@bind P0_1 Slider(0:5:100, default=20, show_value=true))
"""

# ╔═╡ 20245898-9147-11eb-16cb-7320a2ffbe0d
begin
	# Our differential equation
	dPdt_1(P, r) = r*P
	# A vector holding different population size values
	population_1 = 0:0.5:100
	# A vector holding our growth rate values for different population sizes
	growthrate_1 = [dPdt_1(p,r_1) for p in population_1]

	# The solution to our differential equation
	P_1(t, r, P0) = P0*exp(r*t)
	# A vector holding values for different timepoints
	time_1 = 0:0.5:50
	# A vector holding our simulated population sizes
	sim_pop_1 = [P_1(t,r_1,P0_1) for t in time_1]

	rateplot_1 = plot(population_1, growthrate_1,
				xlims = (0,100),
				ylims = (-1.2, 1.2),
				title = "Rateplot",
				label = "",
				xlabel = "P", ylabel = L"$\frac{dP}{dt}$")
	simplot_1 = plot(time_1, sim_pop_1,
				color = "red",
				ylims = (0,100),
				title = "Simulation",
				label = "",
				xlabel = "t", ylabel = "P")
	plot(rateplot_1, simplot_1, layout = (1, 2),fontfamily="Computer Modern")
end

# ╔═╡ 24f0552e-9148-11eb-1226-c326de038a70
md"""
## 3. The Logistic Model

Without being to explicit about the precise nature of our limiting factor, let us assume the population size be limited to a maximum number that can be supported in a given habitat. We call this the **carrying capacity, $K$**.

$$(17) \frac{dP}{dt} = rP(1-\frac{P}{K})$$

### Visual Analysis
If our population is close to $0$, it will won't "feel" any limiting effect and grow in a similar manner to the exponential model discussed above. As the population gets closer to the carrying capacity, the growth will begin to slow down.
"""

# ╔═╡ 3d7b3688-9148-11eb-3224-9b832c4692d9
md"""
*Change the sliders to adjust the corresponding values*:
- growth rate constant ($r$) $(@bind r_2 Slider(0:0.001:0.25, default=0.01, show_value=true))
- carrying capacity ($K$) $(@bind K_2 Slider(1:5:80, default=30, show_value=true))
- initial population size($P_0$) $(@bind P0_2 Slider(1:5:100, default=20, show_value=true))
"""

# ╔═╡ 5fbb5af8-9148-11eb-0ade-b3fd39af29c1
begin
	# Our differential equation
	dPdt_2(P, r, K) = r*P*(1- P/K)
	# A vector holding different population size values
	population_2 = 0:0.5:100
	# A vector holding our growth rate values for different population sizes
	growthrate_2 = [dPdt_2(p,r_2, K_2) for p in population_2]

	# The solution to our differential equation
	P_2(t, r, K,  P0) = K/((K-P0)/P0*exp(-r*K*t) + 1)
	# A vector holding values for different timepoints
	time_2 = 0:0.5:50
	# A vector holding our simulated population sizes
	sim_pop_2 = [P_2(t,r_2/K_2,K_2, P0_2) for t in time_2]

	rateplot_2 = plot(population_2, growthrate_2,
				xlims = (0,100),
				ylims = (-1.2, 2.5),
				title = "Rateplot",
				label = "",
				xlabel = "P", ylabel = L"$\frac{dP}{dt}$")
		plot!(rateplot_2, [K_2 for p in -2:3] , -2:3,
				color = "darkgrey",
				linestyle = :dash,
				label = "";
				annotatios = (K_2, 1, "P=K"))
		plot!(rateplot_2, population_2, [r_2 * p for p in population_2],
				color = "grey",
				linestyle = :dot,
				label = L"$\frac{dP}{dt} = rP$")

	simplot_2 = plot(time_2, sim_pop_2,
				color = "red",
				ylims = (0,100),
				title = "Simulation",
				label = "",
				xlabel = "t", ylabel = "P")
		plot!(simplot_2, time_2 , [K_2 for t in time_2],
				color = "darkgrey",
				linestyle = :dash,
				label = "",
				annotatios = ([40], [K_2], "P=K"))
	plot(rateplot_2, simplot_2, layout = (1, 2),fontfamily="Computer Modern")
end

# ╔═╡ 64b31f38-9149-11eb-11a0-718a61330e76
md"""
The corresponding rate plot looks like a prabola that crosses the x-axis at $P=0$ and $P=K$ and has a maximum at $P=\frac{K}{2}$. The x-crossings are important. Here, the growth rate is $0$, so they represent **equilibria**.  However, the two equilibria are not the same: Just to the right of $P=0$, the growth rate increases, so the equilibrium is **unstable**. The case for the equilibrium at $P=K$ is different: if $P<K$, the growth rate is positive, whereas the growth rate is negative if $P>K$. Consequently, the population will be attracted to $P=K$ from both sides. This is called a **stable equilibrium** - small fluctuations around $P=K$ will result in the population falling back to $P=K$.

Using the insight gained from the rate plot, we can infer the shape of the curve for $P(t)$. It will be *s-shaped* or *sigmoidal*, reflecting the change in growth rate around $P=\frac{K}{2}$, and level off around $P=K$.
"""

# ╔═╡ 6f8ed9ce-9149-11eb-2c08-1b1e1f3e3f5c
md"""
### Analytic solution
To obtain an explicit formula for $P(t)$, we need to marshall what we've learned so far, integrate, and brace ourselves for quite a bit of algebra...

To make our lives a bit easier, we rearrange the equation as follows

$$(18) \frac{dP}{dt} = r_{max}P(1-\frac{P}{K}) = r_{max}P(\frac{K}{K}-\frac{P}{K})= \frac{r_{max}}{K}P(K-P) = rP(K-P)$$

Next, we separate our variables:

$$(19) \int \frac{1}{P(K-P)} dP = r \int dt$$

To solve the integral on the left hand side, it helps to simplify the fraction using a *partial fraction decomposition*. Here we want to decompose our fraction $\frac{1}{P(K-P)}$ into two partial fractions of the form $\frac{A}{P} + \frac{B}{K-P}$, where $A$ and $B$ are constants that can be easily determined as follows:

$$\frac{A}{P} + \frac{B}{K-P} = \frac{1}{P(K-P)}$$

Multiply the two fractions on the left by $P$ and $(K-P)$ respectively to obtain a common denominator with the fraction on the right

$$\frac{A(K-P)}{P(K-P)} + \frac{BP}{P(K-P)} = \frac{1}{P(K-P)}$$
Multiply the whole expression by $P(K-P)$ and rearrange to find

$$AK + P(B-A) = 1 \rightarrow A,B = \frac{1}{K}$$

Hence,

$$\frac{1}{K} (\frac{1}{P} + \frac{1}{K-P}) = \frac{1}{P(K-P)}$$

Substituting the partial fraction decomposition into equation (19), we obtain
$$(20) \frac{1}{K} \int \frac{1}{P} + \frac{1}{K-P} dP = r \int dt$$

This way of rearranging our expression allows us to treat each fraction separately

$$(21) \frac{1}{K} (\int \frac{1}{P} dp + \int \frac{1}{K-P} dP) = r \int dt$$

###### Finding integrals by substitution
We already know how to integrate the first fraction - $\int \frac{1}{P} dP = ln(P) + C_1$, as well as the right hand side - $r \int dt = rt + C_2$. However, to integrate the second fraction, we need to use substitution.
Let

$$(22) u = K-P$$

It follows that

$$(23)\frac{du}{dP} = -1 \rightarrow  du = -dP \rightarrow dP = - du$$

We substitute equations (21) and (23) into $\int \frac{1}{K-P} dP$ and obtain$-\int \frac{1}{u} du$, which can be easily integrated and gives $-ln(u) +C =-ln(K-P) + C_3$. Plugging our solved integrals into equation (21) and absorbing our various constants into a new constant $B$, we get

$$(24) \frac{1}{K}[ln(P) - ln(K-P)] = rt + B$$

Using the identity $ln(a) - ln(b) = ln(\frac{a}{b})$, this rearranges to

$$(25) ln(\frac{P}{K-P}) = rKt + B$$

Applying the exponential function to both sides yields

$$(26) \frac{P}{K-P} = e^{rKt + B} = e^{rKt} \cdot e^{B}$$

Let $A = e^B$:

$$(27) \frac{P}{K-P} = Ae^{rKt}$$

###### Finding the value of the constant A

Let $P_0 = P(0)$ and $t=0$:

$$(28) \frac{P_0}{K-P_0} = A$$

Hence,

$$(29) \frac{P}{K-P} = \frac{P_0}{K-P_0}e^{rKt}$$

###### Solve expression for P

$$(30) P(K-P_0) = P_0(K-P)e^{rKt} = P_0Ke^{rKt} - P_0Pe^{rKt}$$

Next, we collect the terms that depend on $P$:

$$(31) P(K-P_0) + P_0Pe^{rKt} = P_0Ke^{rKt}$$

$$(32) P[(K-P_0) + P_0e^{rKt}] = P_0Ke^{rKt}$$

$$(33) P = \frac{P_0Ke^{rKt}}{(K-P_0) + P_0e^{rKt}}$$

This can be rewritten as

$$(34) P(t) = \frac{K}{\frac{K-P_0}{P_0}e^{-rKt} + 1}$$

## Conclusion
And that is that. Solving differential equations this way is possible for some, but not all. The algebra gets hairy quite quickly, particularly, when looking at systems of differential equations, where differential equations for two or more variables depend on each other. In such instances, we obtain timecourses by simulating these systems numerically. More on this soon...

"""

# ╔═╡ Cell order:
# ╟─89fcfe1c-913a-11eb-1b04-0d87fd8dbcbe
# ╟─9f7c1628-9139-11eb-0d69-f910a995eca7
# ╟─babc9748-9139-11eb-0da1-0d942d591481
# ╟─ee2bb580-9145-11eb-2bc2-dbd9aae68be9
# ╟─001fc24a-9146-11eb-39b9-955dac6bcf58
# ╟─ad08de50-913d-11eb-3f87-fb27a282bdb3
# ╟─f58d273e-9139-11eb-0fe0-b11e90166682
# ╟─aa53e7e0-913d-11eb-32ab-4fc1ce2b72ab
# ╟─008851e2-9147-11eb-3b1c-0ff07fd4e84f
# ╟─20245898-9147-11eb-16cb-7320a2ffbe0d
# ╟─24f0552e-9148-11eb-1226-c326de038a70
# ╟─3d7b3688-9148-11eb-3224-9b832c4692d9
# ╟─5fbb5af8-9148-11eb-0ade-b3fd39af29c1
# ╟─64b31f38-9149-11eb-11a0-718a61330e76
# ╟─6f8ed9ce-9149-11eb-2c08-1b1e1f3e3f5c


using SimilaritySearch, NeighborhoodApproximationIndex

function create_dataset(n, dim)
    [rand(Float32, dim) for i in 1:n]
end

function run(index, queries, k)
    @info ("========>", string(index), length(queries), k)
    @time for q in queries
        search(index, q, k)
    end
end

function main()
	dim = 8
	X = create_dataset(100_000, dim)
	queries = create_dataset(100, dim)
	dist = SqL2Distance()
    napp = DeloneInvIndex(dist, X; initial=:rand, maxiters=3, ksearch=1)
    seq = ExhaustiveSearch(dist, X)
    @show size(X), size(queries), dist
    k = 10
    run(seq, queries, k)
    run(napp, queries, k)
end

main()


using SparseArrays

function sparserecttest(lib::Polyhedra.Library)
    # Sparse H-representation, cartesian product
    # -1 ≤ x_1 ≤ 1
    # 2 ≤ x_2 ≤ 3
    h1 = HalfSpace(sparsevec([1], [ 1]),  1) ∩
         HalfSpace(sparsevec([1], [-1]),  1)
    h2 = HalfSpace(sparsevec([1], [ 1]),  3) ∩
         HalfSpace(sparsevec([1], [-1]), -2)
    h = h1 * h2
    v = convexhull([-1, 2],
                   [ 1, 2],
                   [-1, 3],
                   [ 1, 3])
    p = polyhedron(h, lib)
    @test !vrepiscomputed(p)
    inequality_fulltest(p, h)
    generator_fulltest(p, v)
    p1 = polyhedron(h1, lib)
    @test !vrepiscomputed(p1)
    p2 = polyhedron(h2, lib)
    @test !vrepiscomputed(p2)
    q = p1 * p2
    @test !vrepiscomputed(q)
    inequality_fulltest(q, h)
    generator_fulltest(q, v)
end


using Krylov
using LinearOperators
using MatrixMarket
# using ProfileView

mtx = "data/1138bus.mtx";
# mtx = "data/bcsstk09.mtx";
# mtx = "data/bcsstk18.mtx";

A = MatrixMarket.mmread(mtx);
n = size(A, 1);
b = ones(n); b_norm = norm(b);

# Define a linear operator with preallocation.
Ap = zeros(n);
op = LinearOperator(n, n, true, true, p -> A_mul_B!(1.0,  A, p, 0.0, Ap))

# Solve Ax=b.
(x, stats) = cg(op, b);
# @profile x = cg(A, b);
@time (x, stats) = cg(op, b);
show(stats);
r = b - A * x;
@printf("Relative residual: %8.1e\n", norm(r)/b_norm);

# ProfileView.view()


mutable struct PerfectBipartiteMatchingAlgosSolver <: PerfectBipartiteMatchingSolver
  # All other instances only work with complete bipartite graphs, do the same here.
  n_arms::Int

  function PerfectBipartiteMatchingAlgosSolver()
    new(-1)
  end
end

has_lp_formulation(::PerfectBipartiteMatchingAlgosSolver) = false
approximation_ratio(::PerfectBipartiteMatchingAlgosSolver) = 1.0
approximation_term(::PerfectBipartiteMatchingAlgosSolver) = 0.0
approximation_ratio_budgeted(::PerfectBipartiteMatchingAlgosSolver) = 0.5
approximation_term_budgeted(::PerfectBipartiteMatchingAlgosSolver) = 0.0
supports_solve_budgeted_linear(::PerfectBipartiteMatchingAlgosSolver) = true
supports_solve_all_budgeted_linear(::PerfectBipartiteMatchingAlgosSolver) = true

function build!(solver::PerfectBipartiteMatchingAlgosSolver, n_arms::Int)
  solver.n_arms = n_arms
  nothing
end

function solve_linear(solver::PerfectBipartiteMatchingAlgosSolver, rewards::Dict{Tuple{Int, Int}, Float64})
  g = complete_bipartite_graph(solver.n_arms, solver.n_arms)
  r = Dict{Edge{Int}, Float64}(Edge(k[1], solver.n_arms + k[2]) => v for (k, v) in rewards)
  i = BipartiteMatchingInstance(g, r)

  s = matching_hungarian(i).solution

  return [(src(e), dst(e) - solver.n_arms) for e in s]
end

function solve_budgeted_linear(solver::PerfectBipartiteMatchingAlgosSolver,
                               rewards::Dict{Tuple{Int, Int}, Float64},
                               weights::Dict{Tuple{Int, Int}, Int},
                               budget::Int)
  g = complete_bipartite_graph(solver.n_arms, solver.n_arms)
  r = Dict{Edge{Int}, Float64}(Edge(k[1], solver.n_arms + k[2]) => v for (k, v) in rewards)
  w = Dict{Edge{Int}, Int}(Edge(k[1], solver.n_arms + k[2]) => v for (k, v) in weights)
  i = BudgetedBipartiteMatchingInstance(g, r, w, budget)

  s = matching_hungarian_budgeted_lagrangian_approx_half(i).solution

  return [(src(e), dst(e) - solver.n_arms) for e in s]
end

function solve_all_budgeted_linear(solver::PerfectBipartiteMatchingAlgosSolver,
                                   rewards::Dict{Tuple{Int, Int}, Float64},
                                   weights::Dict{Tuple{Int, Int}, Int},
                                   max_budget::Int)
  g = complete_bipartite_graph(solver.n_arms, solver.n_arms)
  r = Dict{Edge{Int}, Float64}(Edge(k[1], solver.n_arms + k[2]) => v for (k, v) in rewards)
  w = Dict{Edge{Int}, Int}(Edge(k[1], solver.n_arms + k[2]) => v for (k, v) in weights)
  i = BudgetedBipartiteMatchingInstance(g, r, w, max_budget)

  s = matching_dp_budgeted(i).solution

  sol = Dict{Int, Vector{Edge{Int}}}()
  vl = last(instance.matching.vertex_left)
  vr = last(instance.matching.vertex_right)
  for budget in 0:max_budget
    sol[budget] = s.solutions[vl, vr, 0, budget]
  end
  return sol
end


using DynamicPolynomials
using MultivariateMoments
using SemialgebraicSets

const MM = MultivariateMoments

include("../src/BeurlingLasso.jl")
using .BeurlingLasso

@polyvar x[1:2]

# only valid for odd numbers
function doublefactorial(n::Integer)
    if n <= 0
        1
    else
        prod(2*k+1 for k=0:div(n-1, 2))
    end
end

nmoments = 15

φ = [sum(binomial(n, 2*k) * x[1]^(n-2*k) * x[2]^(2*k) * doublefactorial(2*k-1)
     for k=0:div(n, 2)) for n=0:nmoments-1]

params = [[-0.5, 0.5], [0.6, 0.3]]
weights = [0.7, 0.3]

natoms = length(params)

μ0 = AtomicMeasure(x, [WeightedDiracMeasure(params[i], weights[i]) for i=1:natoms])
y = [MM.expectation(μ0, φ[n]) for n=1:nmoments]

# λ = 0
#
# domain = @set x[1] >= -1 && x[1] <= 1 && x[2] >= -1 && x[2] <= 1
#
# prob1 = blasso(y, φ, λ, domain)
# solver1 = momsossolver(1e-3)
# solve!(prob1, solver1)

λ = 1e-5

φ_func(u, n) = φ[n](x=>u)
bounds = [[-1 1]; [-1 1]]

prob2 = blasso(y, φ_func, λ, bounds)
solver2 = sfwsolver(2, 1000)
solve!(prob2, solver2)


using QuantumOptics

b = SpinBasis(1//2)

sx = sigmax(b)
sy = sigmay(b)
sz = sigmaz(b)

N = 5

spinchain = b ⊗ b

for ispin in 3:N
    global spinchain
    spinchain = spinchain ⊗ b
end

XX = sx ⊗ sx
YY = sy ⊗ sy
ZZ = sz ⊗ sz

Jz=1

HIsing = embed(spinchain, [1,2] , Jz*ZZ)
for ispin in 2:N-1
    global HIsing
    HIsing += embed(spinchain, [ispin,ispin+1] , Jz*ZZ)
end

# make sure H is numerically Hermitian
HIsing = ( HIsing + dagger(HIsing) )/2

# find first 5 eigenvalues using sparse matrix
E5 = eigenenergies(HIsing, 5)
println(" ")
println("Lowest five eigenvalues")
println(E5)
println(" ")

# full spectrum
EE, UU = eigenstates(dense(HIsing))
println(" ")
println("All Eigenvalues")
println(EE)
println(" ")


using NeRCA
using Test

@testset "angle_between()" begin
    @test 0 == NeRCA.angle_between([1,0,0], [1,0,0])
    @test π/2 ≈ NeRCA.angle_between([1,0,0], [0,1,0])
    @test π/2 ≈ NeRCA.angle_between([1,0,0], [0,0,1])
    @test π ≈ NeRCA.angle_between([1,0,0], [-1,0,0])
end

@testset "azimuth()" begin
    @test π/2 == NeRCA.azimuth(Direction(0,1,0))
    @test 0 ≈ NeRCA.azimuth(Direction(1,0,0))
    @test -π/2 ≈ NeRCA.azimuth(Direction(0,-1,0))
    @test π ≈ NeRCA.azimuth(Direction(-1,0,0))
end

@testset "zenith()" begin
    @test 0.0 == NeRCA.zenith(Direction(0,0,-1))
    @test π/2 == NeRCA.zenith(Direction(0,1,0))
    @test π/2 == NeRCA.zenith(Direction(1,1,0))
    @test π ≈ NeRCA.zenith(Direction(0,0,1))
end


function FreeConvectionNDE(NN, ds; iterations=nothing)
    weights, reconstruct = Flux.destructure(NN)

    T = ds["T"]
    wT = ds["wT"]
    FT = eltype(T)
    Nz = size(T, 3)
    Nt = size(T, 4)
    zc = znodes(T)
    zf = znodes(wT)
    times = T.times

    H = abs(zf[1]) # Domain depth/height
    τ = times[end]  # Simulation length

    Δẑ = diff(zc)[1] / H  # Non-dimensional grid spacing
    Dzᶜ = Dᶜ(Nz, Δẑ) # Differentiation matrix operator
    Dzᶜ = convert(Array{FT}, Dzᶜ)

    if isnothing(iterations)
        iterations = 1:length(times)
    end

    """
    Non-dimensional PDE is

        ∂T/∂t = - σ_wT/σ_T * τ/H * ∂/∂z(wT)
    """
    function ∂T∂t(T, p, t)
        weights = p[1:end-6]
        bottom_flux, top_flux, σ_T, σ_wT, H, τ = p[end-5:end]

        NN = reconstruct(weights)
        wT_interior = NN(T)
        wT = [bottom_flux; wT_interior; top_flux]
        ∂z_wT = Dzᶜ * σ_wT/σ_T * τ/H * wT
        return -∂z_wT
    end

    tspan = FT.( (0.0, maximum(iterations) / Nt) )
    saveat = range(tspan[1], tspan[2], length=length(iterations))

    # See: https://github.com/SciML/DiffEqFlux.jl/blob/449efcecfc11f1eab65d0e467cf57db9f5a5dbec/src/neural_de.jl#L66-L67
    # We set the initial condition to `nothing`. Then we will set it to some actual initial condition when calling `solve`.
    ff = ODEFunction{false}(∂T∂t, tgrad=DiffEqFlux.basic_tgrad)
    return ODEProblem{false}(ff, nothing, tspan, saveat=saveat)
end

function FreeConvectionNDEParameters(ds, T_scaling, wT_scaling)
    wT = ds["wT"]
    zf = znodes(wT)
    times = wT.times

    H = abs(zf[1]) # Domain depth/height
    τ = times[end]  # Simulation length

    bottom_flux = wT_scaling(interior(wT)[1, 1, 1, 1])
    top_flux = wT_scaling(ds.metadata["temperature_flux"])

    FT = eltype(wT)
    return FT.([bottom_flux, top_flux, T_scaling.σ, wT_scaling.σ, H, τ])
end


# slab ocean ODE
function ocn_rhs!(du, u, (parameters, F_accumulated), t)
    """
    Slab layer equation
        d(T_sfc)/dt = - (F_accumulated) / (h_ocn * ρ_ocn * c_ocn)
        where
            F_accumulated = F_integrated / Δt_coupler
    """
    @unpack ocn_h, ocn_ρ, ocn_c = parameters
    @unpack T_sfc = du

    @. T_sfc = (-F_accumulated) / (ocn_h * ocn_ρ * ocn_c)
end

# set up domain
function hspace_1D(xlim = (-π, π), npoly = 0, helem = 10)
    FT = Float64

    domain = Domains.IntervalDomain(Geometry.XPoint{FT}(xlim[1]) .. Geometry.XPoint{FT}(xlim[2]), periodic = true)
    mesh = Meshes.IntervalMesh(domain; nelems = helem)
    topology = Topologies.IntervalTopology(mesh)

    # Finite Volume Approximation: Gauss-Lobatto with 1pt per element
    quad = Spaces.Quadratures.GL{npoly + 1}()
    space = Spaces.SpectralElementSpace1D(topology, quad)

    return space
end

# init simulation
function ocn_init(; xmin = -1000, xmax = 1000, helem = 20, npoly = 0)

    # construct domain spaces - get only surface layer (NB: z should be zero, not z = first central height)
    space = hspace_1D((xmin, xmax), npoly, helem)
    coords = Fields.coordinate_field(space)
    domain = space

    # initial condition
    T_sfc = map(coords) do coord
        T_sfc = 267.0
    end

    # prognostic variable
    Y = Fields.FieldVector(T_sfc = T_sfc)

    return Y, domain
end


# Convert between different parameterizations.

module Transform

# TODO: don't import Model; transformations should operate on
# generic ParamSets
using ..Model
using ..SensitiveFloats
import ..Log

export DataTransform, ParamBounds, ParamBox, SimplexBox,
       get_mp_transform, enforce_bounds!

################################
# Elementary functions.

"""
Unconstrain x in the unit interval to lie in R.
"""
function inv_logit{NumType <: Number}(x::NumType)
    @assert(x >= 0)
    @assert(x <= 1)
    -log(1.0 / x - 1)
end

function inv_logit{NumType <: Number}(x::Array{NumType})
    @assert(all(x .>= 0))
    @assert(all(x .<= 1))
    -log(1.0 ./ x - 1)
end

"""
Convert x in R to lie in the unit interval.
"""
function logit{NumType <: Number}(x::NumType)
    1.0 / (1.0 + exp(-x))
end

function logit{NumType <: Number}(x::Array{NumType})
    1.0 ./ (1.0 + exp(-x))
end

"""
Convert an (n - 1)-vector of real numbers to an n-vector on the simplex, where
the last entry implicitly has the untransformed value 1.
"""
function constrain_to_simplex{NumType <: Number}(x::Vector{NumType})
    if any(x .== Inf)
        z = NumType[ x_entry .== Inf ? one(NumType) : zero(NumType) for x_entry in x]
        z ./ sum(z)
        push!(z, 0)
        return(z)
    else
        z = exp(x)
        z_sum = sum(z) + 1
        z ./= z_sum
        push!(z, 1 / z_sum)
        return(z)
    end
end

"""
Convert an n-vector on the simplex to an (n - 1)-vector in R^{n -1}.  Entries
are expressed relative to the last element.
"""
function unconstrain_simplex{NumType <: Number}(z::Vector{NumType})
    n = length(z)
    NumType[ log(z[i]) - log(z[n]) for i = 1:(n - 1)]
end

################################
# The transforms for Celeste.

immutable ParamBox
    lower_bound::Float64
    upper_bound::Float64
    scale::Float64

    function ParamBox(lower_bound, upper_bound, scale)
        @assert lower_bound > -Inf # Not supported
        @assert scale > 0.0
        @assert lower_bound < upper_bound
        new(lower_bound, upper_bound, scale)
    end
end

immutable SimplexBox
    lower_bound::Float64
    scale::Float64
    n::Int

    function SimplexBox(lower_bound, scale, n)
        @assert n >= 2
        @assert 0.0 <= lower_bound < 1 / n
        new(lower_bound, scale, n)
    end
end

# The vector of transform parameters for a Symbol.
typealias ParamBounds Dict{Symbol, Union{Vector{ParamBox}, Vector{SimplexBox}}}

###############################################
# Functions for a "free transform".

function unbox_parameter{NumType <: Number}(param::NumType, param_box::ParamBox)
    lower_bound = param_box.lower_bound
    upper_bound = param_box.upper_bound
    scale = param_box.scale

    positive_constraint = (upper_bound == Inf)

    # exp and the logit functions handle infinities correctly, so
    # parameters can equal the bounds.
    @assert(lower_bound .<= param .<= upper_bound,
                    string("unbox_parameter: param outside bounds: ",
                                 "$param ($lower_bound, $upper_bound)"))

    if positive_constraint
        return log(param - lower_bound) * scale
    else
        param_bounded = (param - lower_bound) / (upper_bound - lower_bound)
        return inv_logit(param_bounded) * scale
    end
end

function box_parameter{NumType <: Number}(
            free_param::NumType, param_box::ParamBox)
    lower_bound = param_box.lower_bound
    upper_bound = param_box.upper_bound
    scale = param_box.scale

    positive_constraint = (upper_bound == Inf)
    if positive_constraint
        return (exp(free_param / scale) + lower_bound)
    else
        return(
            logit(free_param / scale) * (upper_bound - lower_bound) + lower_bound)
    end
end

"""
Convert an unconstrained (n-1)-vector to a simplicial n-vector, z, such that
  - sum(z) = 1
  - z >= simplex_box.lower_bound
See notes for a derivation and reasoning.
"""
function simplexify_parameter{NumType <: Number}(
            free_param::Vector{NumType}, simplex_box::SimplexBox)
    n = simplex_box.n
    lower_bound = simplex_box.lower_bound
    scale = simplex_box.scale

    @assert length(free_param) == (n - 1)

    # Broadcasting doesn't work with DualNumbers and Floats. :(
    # z_sim is on an unconstrained simplex.
    z_sim = constrain_to_simplex(NumType[ p / scale for p in free_param ])
    param = NumType[ (1 - n * lower_bound) * p + lower_bound for p in z_sim ]

    param
end

"""
Invert the transformation simplexify_parameter() by converting an n-vector
on a simplex to R^{n - 1}.
"""
function unsimplexify_parameter{NumType <: Number}(
            param::Vector{NumType}, simplex_box::SimplexBox)
    n = simplex_box.n
    lower_bound = simplex_box.lower_bound
    scale = simplex_box.scale

    @assert length(param) == n
    @assert all(param .>= lower_bound)
    @assert(abs(sum(param) - 1) < 1e-14, abs(sum(param) - 1))

    # z_sim is on an unconstrained simplex.
    # Broadcasting doesn't work with DualNumbers and Floats. :(
    z_sim = NumType[ (p - lower_bound) / (1 - n * lower_bound) for p in param ]
    free_param = NumType[ p * scale for p in unconstrain_simplex(z_sim) ]

    free_param
end

##################
# Derivatives

"""
Return derivatives of an unscaled transform from free parameters to a simplex.

Args:
  - z_sim: A vector in a simplex (NB: the function returns derivatives of the
           function f(unconstrained) = simplex, but it the answer is expressed
           in terms of the output of the function.)

Returns:
  - jacobian: n by (n -1) matrix of derivatives of the simplex output (in rows)
              wrt the free parameters (in columns)
  - hessian_vec: An n-length vector of the hessian of each simplex output
                 parameter with respect to the (n-1) free input parameters.
"""
function simplex_derivatives{NumType <: Number}(z_sim::Vector{NumType})
    n = length(z_sim)
    hessian_vec = Array(Array{NumType}, n)

    for i = 1:n
        hessian_vec[i] = Array(NumType, n - 1, n - 1)
        for j=1:(n - 1), k=1:(n - 1)
            if j != k
                if (j == i)
                    hessian_vec[i][j, k] = -z_sim[i] * z_sim[k] * (1 - 2 * z_sim[i])
                elseif (k == i)
                    hessian_vec[i][j, k] = -z_sim[i] * z_sim[j] * (1 - 2 * z_sim[i])
                else
                    hessian_vec[i][j, k] = 2 * z_sim[i] * z_sim[j] * z_sim[k]
                end
            else # j == k
                if i == j # All equal
                    hessian_vec[i][j, k] = z_sim[i] * (1 - z_sim[j]) * (1 - 2 * z_sim[k])
                else # j == k, but both are different from i
                    hessian_vec[i][j, k] = - z_sim[i] * z_sim[j] * (1 - 2 * z_sim[j])
                end
            end
        end
    end

    jacobian =
        NumType[ z_sim[i] * (i == j) - z_sim[i] * z_sim[j] for i=1:n, j=1:(n - 1) ]

    jacobian, hessian_vec
end

"""
Return the derivative and hessian of a simplex transform given the constrained
parameters.

Args:
  - param: The constrained parameter (NB: the derivatives are expressed
               as a function of the constrained parameterd despite being
                     the derivative of the function unconstrained -> constrained)
    - simplex_box: A box simplex constraint
"""
function box_simplex_derivatives{NumType <: Number}(
    param::Vector{NumType}, simplex_box::SimplexBox)
    lower_bound = simplex_box.lower_bound
  scale = simplex_box.scale
  n = simplex_box.n

  @assert length(param) == n

  # z_sim is on an unconstrained simplex.
  # Broadcasting doesn't work with DualNumbers and Floats. :(
  z_sim = NumType[ (p - lower_bound) / (1 - n * lower_bound) for p in param ]

    jacobian, hessian_vec = simplex_derivatives(z_sim)

    for i in 1:n
        hessian_vec[i] *= (scale ^ 2) * (1 - n * lower_bound)
    end
    jacobian *= scale * (1 - n * lower_bound)
  jacobian, hessian_vec
end

"""
Return the derivative and hessian of a box transform given the constrained
parameters.

Args:
  - param: The constrained parameter (NB: the derivatives are expressed
               as a function of the constrained parameterd despite being
                     the derivative of the function unconstrained -> constrained)
    - param_box: A box constraint
"""
function box_derivatives{NumType <: Number}(param::NumType, param_box::ParamBox)
    lower_bound = param_box.lower_bound
  upper_bound = param_box.upper_bound
  scale = param_box.scale

    if upper_bound == Inf
        centered_param = param - lower_bound
        return scale * centered_param, scale ^ 2 * centered_param
    else
        param_range = upper_bound - lower_bound
        centered_param = (param - lower_bound) / param_range
        derivative = param_range * centered_param * (1 - centered_param) / scale
        return derivative, derivative * (1 - 2 * centered_param) / scale
    end
end

"""
A datatype containing derivatives of the transform from free to constrained
parameters.

Members:
  dparam_dfree: The Jacobian of the transformation
                contrainted_param = f(free param)
  d2param_dfree2: A vector of hessians.  Each element is the Hessian of one
                  component of the aforementioned f()
"""
type TransformDerivatives{NumType <: Number}
    dparam_dfree::Matrix{NumType}
    d2param_dfree2::Array{Matrix{NumType}}
    Sa::Int

    # TODO: use sparse matrices?
    function TransformDerivatives(Sa::Int)
        dparam_dfree =
            zeros(NumType,
                        Sa * length(CanonicalParams), Sa * length(UnconstrainedParams))
        d2param_dfree2 = Array(Matrix{NumType}, Sa * length(CanonicalParams))
        for i in 1:(Sa * length(CanonicalParams))
            d2param_dfree2[i] =
                zeros(NumType,
                            Sa * length(UnconstrainedParams), Sa * length(UnconstrainedParams))
        end
        new(dparam_dfree, d2param_dfree2, Sa)
    end
end

"""
Populate a TransformDerivatives object in place.

Args:
  - bounds: A vector containing one ParamBounds for each active source in the
            same order as active_sources.
  - transform_derivatives: TransformDerivatives to be populated.

Returns:
  Update transform_derivatives in place.
"""
function get_transform_derivatives!{NumType <: Number}(
                            vp::VariationalParams{NumType},
                            active_sources::Vector{Int},
                            bounds::Vector{ParamBounds},
                            transform_derivatives::TransformDerivatives)

  @assert transform_derivatives.Sa == length(active_sources)

  for param in fieldnames(ids), sa = 1:length(active_sources)
    s = active_sources[sa]
      constraint_vec = bounds[sa][param]

      if isa(constraint_vec[1], ParamBox) # It is a box constraint
          @assert(length(constraint_vec) == length(getfield(ids_free, param)) ==
        length(getfield(ids, param)))

          # Get each components' derivatives one by one.
          for ind = 1:length(constraint_vec)
              @assert isa(constraint_vec[ind], ParamBox)
              vp_ind = getfield(ids, param)[ind]
              vp_free_ind = getfield(ids_free, param)[ind]

              jac, hess = box_derivatives(vp[s][vp_ind], constraint_vec[ind])

              vp_sf_ind = length(CanonicalParams) * (sa - 1) + vp_ind
              vp_free_sf_ind = length(UnconstrainedParams) * (sa - 1) + vp_free_ind

              transform_derivatives.dparam_dfree[vp_sf_ind, vp_free_sf_ind] = jac
              transform_derivatives.d2param_dfree2[
                  vp_sf_ind][vp_free_sf_ind, vp_free_sf_ind] = hess
          end
      else # It is a simplex constraint

          # If a param is not a box constraint, it must have all simplex constraints.
          @assert all([ isa(constraint, SimplexBox)  for constraint in constraint_vec])

          param_size = size(getfield(ids, param))
          @assert length(constraint_vec) == param_size[2]
          for col=1:(param_size[2])
              vp_free_ind = getfield(ids_free, param)[:, col]
              vp_ind = getfield(ids, param)[:, col]
              vp_sf_ind = length(CanonicalParams) * (sa - 1) + vp_ind
              vp_free_sf_ind = length(UnconstrainedParams) * (sa - 1) + vp_free_ind

              jac, hess = Transform.box_simplex_derivatives(
                  vp[s][vp_ind], constraint_vec[col])

              transform_derivatives.dparam_dfree[vp_sf_ind, vp_free_sf_ind] = jac
              for row in 1:(param_size[1])
                  transform_derivatives.d2param_dfree2[
                      vp_sf_ind[row]][vp_free_sf_ind, vp_free_sf_ind] = hess[row]
              end
          end
      end
  end
end

function get_transform_derivatives{NumType <: Number}(
                                vp::VariationalParams{NumType},
                                active_sources::Vector{Int},
                                bounds::Vector{ParamBounds})
  transform_derivatives =
    TransformDerivatives{Float64}(length(active_sources))
  get_transform_derivatives!(vp, active_sources, bounds, transform_derivatives)
  transform_derivatives
end

######################
# Functions to take actual parameter vectors.

"""
Convert between variational parameter vectors and unconstrained parameters.

Args:
  - vp: The vector of contrainted variationl parameters
  - vp_free: The vector of uncontrainted parameters for optimization
  - bounds: ParamBounds describing the transformation
  - to_unconstrained: If true, converts vp to vp_free.  If false, converts
                      vp_free to vp.

Returns:
  If to_unconstrained is true, updates vp_free in place.
  If to_unconstrained is false, updates vp in place.
"""
function perform_transform!{NumType <: Number}(
            vp::Vector{NumType}, vp_free::Vector{NumType}, bounds::ParamBounds,
            to_unconstrained::Bool)
    for (param, constraint_vec) in bounds
        is_box = isa(bounds[param], Array{ParamBox})
        if is_box
            # Apply a box constraint to each parameter.
            @assert(length(getfield(ids, param)) == length(getfield(ids_free, param)) ==
                length(bounds[param]))
            for ind in 1:length(getfield(ids, param))
                constraint = constraint_vec[ind]
                free_ind = getfield(ids_free, param)[ind]
                vp_ind = getfield(ids, param)[ind]
                to_unconstrained ?
                    vp_free[free_ind] = unbox_parameter(vp[vp_ind], constraint):
                    vp[vp_ind] = box_parameter(vp_free[free_ind], constraint)
            end
        else
            # Apply a simplex constraint to each parameter.
            @assert isa(bounds[param], Array{SimplexBox})
            param_size = size(getfield(ids, param))

            # Each column is a simplex and should have its own
            # simplicial constraints.
            @assert length(bounds[param]) == param_size[2]
            for col in 1:(param_size[2])
                free_ind = getfield(ids_free, param)[:, col]
                vp_ind = getfield(ids, param)[:, col]
                constraint = constraint_vec[col]
                to_unconstrained ?
                    vp_free[free_ind] = unsimplexify_parameter(vp[vp_ind], constraint):
                    vp[vp_ind] = simplexify_parameter(vp_free[free_ind], constraint)
            end
        end
    end
end

function free_to_vp!{NumType <: Number}(
        vp_free::Vector{NumType}, vp::Vector{NumType}, bounds::ParamBounds)
    perform_transform!(vp, vp_free, bounds, false)
end

function vp_to_free!{NumType <: Number}(
        vp::Vector{NumType}, vp_free::Vector{NumType}, bounds::ParamBounds)
    perform_transform!(vp, vp_free, bounds, true)
end

#####################
# Conversion to and from variational parameter vectors and arrays.

"""
Transform VariationalParams to an array.

Args:
  - vp = variational parameters
  - omitted_ids = ids in ParamIndex

There is probably no use for this function, since you'll only be passing
trasformations to the optimizer, but I'll include it for completeness.
"""
function free_vp_to_array{NumType <: Number}(vp::FreeVariationalParams{NumType},
                                             omitted_ids::Vector{Int})

    left_ids = setdiff(1:length(UnconstrainedParams), omitted_ids)
    new_P = length(left_ids)
    S = length(vp)
    x_new = zeros(NumType, new_P, S)

    for p1 in 1:length(left_ids), s=1:S
        p0 = left_ids[p1]
        x_new[p1, s] = vp[s][p0]
    end

    x_new
end

"""
Transform a parameter vector to variational parameters in place.

Args:
 - xs: A (param x sources) matrix created from free variational parameters.
 - vp_free: Free variational parameters.  Only the ids not in omitted_ids
            will be updated.
 - omitted_ids: Ids to omit (from ids_free)

Returns:
 - Update vp_free in place.
"""
function array_to_free_vp!{NumType <: Number}(
    xs::Matrix{NumType}, vp_free::FreeVariationalParams{NumType},
    omitted_ids::Vector{Int})

    left_ids = setdiff(1:length(UnconstrainedParams), omitted_ids)
    P = length(left_ids)
    S = length(vp_free)
    @assert size(xs) == (P, S)

    for s in 1:S, p1 in 1:P
        p0 = left_ids[p1]
        vp_free[s][p0] = xs[p1, s]
    end
end

#########################
# Define the exported variables.

"""
Functions to move between a single source's variational parameters and a
transformation of the data for optimization.

to_vp: A function that takes transformed parameters and returns
             variational parameters
from_vp: A function that takes variational parameters and returned
                 transformed parameters
to_vp!: A function that takes (transformed paramters, variational parameters)
                and updates the variational parameters in place
from_vp!: A function that takes (variational paramters, transformed parameters)
                    and updates the transformed parameters in place
...
transform_sensitive_float: A function that takes (sensitive float, model
    parameters) where the sensitive float contains partial derivatives with
    respect to the variational parameters and returns a sensitive float with total
    derivatives with respect to the transformed parameters.
bounds: The bounds for each parameter and each object in ElboArgs.
active_sources: The sources that are being optimized.    Only these sources'
    parameters are transformed into the parameter vector.
"""
type DataTransform
    to_vp::Function
    from_vp::Function
    to_vp!::Function
    from_vp!::Function
    vp_to_array::Function
    array_to_vp!::Function
    transform_sensitive_float::Function
    bounds::Vector{ParamBounds}
    active_sources::Vector{Int}
    active_S::Int
    S::Int
end

# TODO: Maybe this should be initialized with ElboArgs with optional
# custom bounds.    Or maybe it should be part of ElboArgs with one transform
# per celestial object rather than a single object containing an array of
# transforms.
function DataTransform(bounds::Vector{ParamBounds};
                       active_sources=collect(1:length(bounds)),
                       S=length(bounds))
    @assert length(bounds) == length(active_sources)
    @assert maximum(active_sources) <= S
    active_S = length(active_sources)

    function from_vp!{NumType <: Number}(
            vp::VariationalParams{NumType}, vp_free::VariationalParams{NumType})
        S = length(vp)
        @assert length(vp_free) == active_S
        for si=1:active_S
            s = active_sources[si]
            vp_to_free!(vp[s], vp_free[si], bounds[si])
        end
    end

    function from_vp{NumType <: Number}(vp::VariationalParams{NumType})
            vp_free = Array{NumType, 1}[
                zeros(NumType, length(ids_free)) for si = 1:active_S]
            from_vp!(vp, vp_free)
            vp_free
    end

    function to_vp!{NumType <: Number}(
            vp_free::FreeVariationalParams{NumType}, vp::VariationalParams{NumType})
        @assert length(vp_free) == active_S
        for si=1:active_S
            s = active_sources[si]
            free_to_vp!(vp_free[si], vp[s], bounds[si])
        end
    end

    function to_vp{NumType <: Number}(vp_free::FreeVariationalParams{NumType})
            @assert(active_S == S,
                    string("to_vp is not supported when active_sources is a ",
                           "strict subset of all sources."))
            vp = Array{NumType, 1}[
                zeros(NumType, length(CanonicalParams)) for s = 1:S]
            to_vp!(vp_free, vp)
            vp
    end

    function vp_to_array{NumType <: Number}(vp::VariationalParams{NumType},
                                            omitted_ids::Vector{Int})
                                            vp_trans = from_vp(vp)
        free_vp_to_array(vp_trans, omitted_ids)
    end

    function array_to_vp!{NumType <: Number}(xs::Matrix{NumType},
                                             vp::VariationalParams{NumType},
                                             omitted_ids::Vector{Int})
        # This needs to update vp in place so that variables in omitted_ids
        # stay at their original values.
        vp_trans = from_vp(vp)
        array_to_free_vp!(xs, vp_trans, omitted_ids)
        to_vp!(vp_trans, vp)
    end

    # Given a sensitive float with derivatives with respect to all the
    # constrained parameters, calculate derivatives with respect to
    # the unconstrained parameters.
    #
    # Note that all the other functions in DeterministicVI calculated derivatives with
    # respect to the constrained parameterization.
    function transform_sensitive_float{NumType <: Number}(
                                    sf::SensitiveFloat,
                                    vp::VariationalParams{NumType},
                                    active_sources::Vector{Int})
        @assert size(sf.d) == (length(CanonicalParams), length(active_sources))
        @assert length(active_sources) == active_S

        transform_derivatives = get_transform_derivatives(vp, active_sources, bounds)
        sf_free = zero_sensitive_float(UnconstrainedParams, NumType, active_S)

        sf_d_vec = sf.d[:]
        sf_free.v[1] = sf.v[1]
        sf_free.d =
        reshape(transform_derivatives.dparam_dfree' * sf_d_vec,
                        length(UnconstrainedParams), active_S)

        sf_free.h =
                transform_derivatives.dparam_dfree' *
        sf.h * transform_derivatives.dparam_dfree
        for ind in 1:length(sf_d_vec)
                sf_free.h += transform_derivatives.d2param_dfree2[ind] * sf_d_vec[ind]
        end
        # Ensure exact symmetry, which is necessary for some numerical linear
        # algebra routines.
        sf_free.h = 0.5 * (sf_free.h' + sf_free.h)
        sf_free
    end

    DataTransform(to_vp, from_vp, to_vp!, from_vp!, vp_to_array, array_to_vp!,
                  transform_sensitive_float, bounds, active_sources, active_S, S)
end

function get_mp_transform{NumType <: Number}(
                          vp::VariationalParams{NumType},
                          active_sources::Vector{Int};
                          loc_width::Float64=1.5e-3)
    bounds = Array(ParamBounds, length(active_sources))

    # Note that, for numerical reasons, the bounds must be on the scale
    # of reasonably meaningful changes.
    for si in 1:length(active_sources)
        s = active_sources[si]
        bounds[si] = ParamBounds()
        bounds[si][:u] = Array(ParamBox, 2)
        u = vp[s][ids.u]
        for axis in 1:2
            bounds[si][:u][axis] =
                ParamBox(u[axis] - loc_width, u[axis] + loc_width, 1.0)
        end
        bounds[si][:r1] = Array(ParamBox, Ia)
        bounds[si][:r2] = Array(ParamBox, Ia)
        for i in 1:Ia
            bounds[si][:r1][i] = ParamBox(-1.0, 10., 1.0)
            bounds[si][:r2][i] = ParamBox(1e-4, 0.1, 1.0)
        end
        bounds[si][:c1] = Array(ParamBox, 4 * Ia)
        bounds[si][:c2] = Array(ParamBox, 4 * Ia)
        for ind in 1:length(ids.c1)
            bounds[si][:c1][ind] = ParamBox(-10., 10., 1.0)
            bounds[si][:c2][ind] = ParamBox(1e-4, 1., 1.0)
        end
        bounds[si][:e_dev] = ParamBox[ ParamBox(1e-2, 1 - 1e-2, 1.0) ]
        bounds[si][:e_axis] = ParamBox[ ParamBox(1e-2, 1 - 1e-2, 1.0) ]
        bounds[si][:e_angle] = ParamBox[ ParamBox(-10.0, 10.0, 1.0) ]
        bounds[si][:e_scale] = ParamBox[ ParamBox(0.1, 70., 1.0) ]

        const simplex_min = 0.005
        bounds[si][:a] = Array(SimplexBox, 1)
        bounds[si][:a][1] = SimplexBox(simplex_min, 1.0, 2)

        bounds[si][:k] = Array(SimplexBox, D)
        for d in 1:D
            bounds[si][:k][d] = SimplexBox(simplex_min, 1.0, 2)
        end
    end

    DataTransform(bounds, active_sources=active_sources, S=length(vp))
end

"""
Put the variational parameters within the bounds of the transform.
Returns:
  Updates vp in place.
"""
function enforce_bounds!{NumType <: Number}(
                        vp::VariationalParams{NumType},
                        active_sources::Vector{Int},
                        transform::DataTransform)
    for sa=1:transform.active_S, (param, constraint_vec) in transform.bounds[sa]
        s = active_sources[sa]
        is_box = isa(constraint_vec, Array{ParamBox})
        if is_box
            # Box parameters.
            for ind in 1:length(getfield(ids, param))
                constraint = constraint_vec[ind]
                if !(constraint.lower_bound <=
                         vp[s][getfield(ids, param)[ind]] <=
                         constraint.upper_bound)
                    # Don't set the value to exactly the lower bound to avoid Inf
                    diff = constraint.upper_bound - constraint.lower_bound
                    epsilon = diff == Inf ? 1e-12: diff * 1e-12
                    vp[s][getfield(ids, param)[ind]] =
                        min(vp[s][getfield(ids, param)[ind]], constraint.upper_bound - epsilon)
                    vp[s][getfield(ids, param)[ind]] =
                        max(vp[s][getfield(ids, param)[ind]], constraint.lower_bound + epsilon)
                end
            end
        else
            # Simplex parameters.
            param_size = size(getfield(ids, param))
            for col in 1:param_size[2]
                constraint = constraint_vec[col]
                param_sum = zero(NumType)
                for row in 1:param_size[1]
                    if !(constraint.lower_bound <= vp[s][getfield(ids, param)[row, col]] <= 1.0)
                        # Don't set the value to exactly the lower bound to avoid Inf
                        epsilon = (1.0 - constraint.lower_bound) * 1e-12
                        vp[s][getfield(ids, param)[row, col]] =
                            min(vp[s][getfield(ids, param)[row, col]], 1.0 - epsilon)
                        vp[s][getfield(ids, param)[row, col]] =
                            max(vp[s][getfield(ids, param)[row, col]],
                                    constraint.lower_bound + epsilon)
                    end
                    param_sum += vp[s][getfield(ids, param)[row, col]]
                end
                if param_sum != 1.0
                    # Normalize in a way that maintains the lower bounds
                    rescale =
                      (1 - constraint.n * constraint.lower_bound) /
                      (param_sum - constraint.n * constraint.lower_bound)
                    vp[s][getfield(ids, param)[:, col]] =
                      constraint.lower_bound +
                      rescale * (vp[s][getfield(ids, param)[:, col]] -
                                 constraint.lower_bound)
                end
            end
        end
    end
end

end


using Random

Random.seed!(123)

# smooth case

f1 = SqrNormL2()
f2 = Translate(SqrNormL2(2.5), randn(10))
f = Sum(f1, f2)

predicates_test(f)

@test ProximalOperators.is_quadratic(f) == true
@test ProximalOperators.is_strongly_convex(f) == true
@test ProximalOperators.is_set(f) == false

xtest = randn(10)

result = f1(xtest) + f2(xtest)
@test f(xtest) ≈ result

grad1, val1 = gradient_test(f1, xtest)
grad2, val2 = gradient_test(f2, xtest)

gradsum, valsum = gradient_test(f, xtest)
@test gradsum ≈ grad1 + grad2

# nonsmooth case

g1 = NormL2()
g2 = Translate(SqrNormL2(2.5), randn(10))
g = Sum(g1, g2)

predicates_test(g)

@test ProximalOperators.is_smooth(g) == false
@test ProximalOperators.is_strongly_convex(g) == true
@test ProximalOperators.is_set(g) == false

xtest = randn(10)

result = g1(xtest) + g2(xtest)
@test g(xtest) ≈ result

grad1, val1 = gradient_test(g1, xtest)
grad2, val2 = gradient_test(g2, xtest)

gradsum, valsum = gradient_test(g, xtest)
@test gradsum ≈ grad1 + grad2


using DesignAutomation

@testset "Cyclic Coordinate Search" begin
    fn(x) = (x[1]-5)^2+2*(x[2]-5)^2
    x0 = [0., 0.]
    res = ccs(fn, x0)

    @test res.minimizer ≈ [5.0, 5.0]
    @test res.minimum + 1 ≈ 1 # answer is approx 0
    @test res.iterations == 3
    @test res.converged == true

    fn(x) = (2*x[2]-3*x[1]^2)^4 + (5-2*x[1])^2
    x0 = [0., 0.]
    res = ccs(fn, x0)

    @test res.minimizer ≈ [2.5, 9.375]
    @test res.minimum + 1 ≈ 1 # answer is approx 0
    @test res.iterations == 4
    @test res.converged == true
end


"Calculate the inverse of the S-matrix star"
function inv_smat_star(S)
    M = inv(S)
    mortar(
        (M[Block(2, 2)], M[Block(2, 1)]),
        (M[Block(1, 2)], M[Block(1, 1)]),
    )
#    mortar(
#        (inv(S[Block(2,2)] - S[Block(2,1)] * inv(S[Block(1,1)]) * S[Block(1,2)]), inv(S[Block(1,2)] - S[Block(1,1)] * inv(S[Block(2,1)]) * S[Block(2,2)])),
#        (inv(S[Block(2,1)] - S[Block(2,2)] * inv(S[Block(1,2)]) * S[Block(1,1)]), inv(S[Block(1,1)] - S[Block(1,2)] * inv(S[Block(2,2)]) * S[Block(2,1)])),
#    )
end

function onehotvector(i::T, n::T, ::S=true) where {T <: Integer, S}
    out = zeros(S, n)
    out[i] = one(S)
    out
end

for n in 2:5, m in 2:5
    # setup rectangular matrices
    Q = [(rand(n, n), rand(m, n), rand(n, m), rand(m, m)) for _ in 1:3]
    S = [mortar(e[[1, 3]], e[[2, 4]]) for e in Q]
    Lid = mortar(reshape([Matrix(a*I, n, n) for a in [0., 1., 1., 0.]], (2, 2)))
    Rid = mortar(reshape([Matrix(a*I, m, m) for a in [0., 1., 1., 0.]], (2, 2)))
    @testset "verify identity" for M in S
        @test M == smat_star(M, Rid) == smat_star(Lid, M)
    end
    @testset "verify associativity" for A in S, B in S, C in S
        # for dimension compatibility for star product of non-square blocks
        D = Array(transpose(reshape(B[end:-1:1], size(B))))
        D = BlockMatrix(D, [m, n], [m, n])
        @test smat_star(A, smat_star(D, C)) ≈ smat_star(smat_star(A, D), C)
    end
    if n == m
        T = [[LinearMap(e[1]) e[3]; e[2] e[4]] for e in Q]
        R = [LinearMap([e[1] e[3]; e[2] e[4]]) for e in Q]
        @testset "compare BlockMatrix and BlockMap" for A in zip(S,T,R), B in zip(S,T,R)
            BA = A[1] ⋆ B[1]
            BM = A[2] ⋆ B[2]
            LM = A[3] ⋆ B[3]
            @testset "verify star product action on standard basis" for i in 1:(n+m)
                eᵢ = onehotvector(i, n+m)
                @test BA * eᵢ ≈ BM * eᵢ ≈ LM * eᵢ
            end
        end
        @testset "verify inverse" for M in S
            M⁻¹ = inv_smat_star(M)
            @test smat_star(M, M⁻¹) ≈ smat_star(M⁻¹, M) ≈ Lid ≈ Rid
        end
    end
end


x = range(0, stop=1, length=100)
y1 = @. x^1.5
y2 = @. x^5
z1 = @. x^(1/1.5)
z2 = @. x^(1/5)
fill_between(x, y1, y2, fill="cornflowerblue", alpha=0.2)
fill_between!(x, z2, z1, fill="seagreen", alpha=0.2)
a1 = x.^3
a2 = x.^(1/3)
plot!(x, a1, lw=0.05, color="cornflowerblue")
plot!(x, a2, lw=0.05, color="seagreen")
fill_between!(x, x, a1, color="red", alpha=0.2)
fill_between!(x, x, a2, color="orange", alpha=0.2)
plot!(x, x, color="black")
plot!(x, z1, color="orange", ls="--")
plot!(x, y1, color="red", ls="--")


#= ######################################################
inside functions
###################################################### =#

const sigma = 5.670374419E-8

function create_identity_matrix(n::T) where T<:Integer
    id_mat = zeros(T,n,n)
    for i = 1:n
        id_mat[i,i] = 1
    end
    return id_mat
end

#= ######################################################
final functions
###################################################### =#

function tempsolver(m, vfmat, temp, epsilon)
    # old algorithm used in matlab (slightly optimized)
    # prepare matrix for LGS
    # J --> outgoing radiation of element
    # G --> incoming radiation of element
    mat = ((epsilon .- 1) .* vfmat)
    # id_mat = Matrix{Int64}(I, m.nelem, m.nelem) # LinearAlgebra Pkg
    id_mat = create_identity_matrix(m.nelem)
    mat .= mat .+ id_mat
    # solve LGS
    B = (temp[:,1].^4) .* sigma .* epsilon
    J = mat \ B
    # Bestimmung der einfallenden Strahlung G
    G = zeros(Float64,m.nelem,1)
    for i = 1:m.nelem
        G[i,1] = sum(J[:,1] .* vfmat[i, :])
    end
    # Qp Radiation per element
    area = [m.elem[i].area for i = m.elem2par[1].first:m.elem2par[end].last]
    Qp = (J[:,1] - G[:,1]) .* area[:,1]
    Ga = G[:,1] .* area[:,1]
    return Qp, Ga
end

function set_bc_part!(m, bcv, parts, bcp)
    # give all face elements one temperature
    for i = parts
        e1 = m.elem2par[i].first
        e2 = m.elem2par[i].last
        bcv[e1:e2,1] .= bcp
    end
end


function _obj_function(p,parameter_dictionary::Dict{String,Any}, v_variable_data::Array{Float64,1},
    experimental_data_array::Array{Float64,2})

    parameter_dictionary["KD"] = p[1]
    parameter_dictionary["n"] = p[2]

    # compute the model v-variable -
    model_v_variable = compute_activity_variable(experimental_data_array[:,1], parameter_dictionary)

    # compute the residual between the data and model estimated v-variable -
    error = sum((v_variable_data .- model_v_variable).^2)

    # return -
    return error
end

function estimate_parameters(parameter_dictionary::Dict{String,Any}, v_variable_data::Array{Float64,1},
    experimental_data_array::Array{Float64,2})

    try

        # get unkown parameter guess from config -
        KD = parameter_dictionary["KD"]
        n = parameter_dictionary["n"]

        # setup the calculation -
        xinitial = [KD n]
        OF(p) = _obj_function(p,parameter_dictionary, v_variable_data, experimental_data_array)

        # call the optimizer -
        opt_result = optimize(OF,xinitial,BFGS())

        # return -
        return Optim.minimizer(opt_result)
    catch error
        throw(error)
    end
end


"""
## mocouq

This subroutine forms the derivatives of a Mohr-Coulomb potential
function with respect to the three invariants (psi in degrees).

### Function
```julia
(dq1,dq2,dq3) = mocouq(psi,dsbar,theta)
```

### Arguments
```julia
* psi::Float64               : Local force and momemts (Updated)
* dsbar::Float64             : Globale forces and moments
* theta::Float64             : Element orientation angle (3D)
```

### Return values
```julia
* dq1::Float64               : Local force and momemts (Updated)
* dq2::Float64               : Globale forces and moments
* dq3::Float64               : Element orientation angle (3D)
```
"""
function mocouq(psi,dsbar,theta)
 snth=sin(theta)
 snps=sind(psi)
 sq3=sqrt(3.0)
 dq1=snps
 if abs(snth) > 0.49
   c1 = 1.0
   snth < 0.0 && (c1 = -1.0)
   dq2 = (sq3*0.5-c1*snps*0.5/sq3)*sq3*0.5/dsbar
   dq3 = 0.0
 else
   csth = cos(theta)
   cs3th = cos(3theta)
   tn3th = tan(3theta)
   tnth = snth/csth
   dq2 = sq3*csth/dsbar*((1.0+tnth*tn3th)+snps*(tn3th-tnth)/sq3)*0.5
   dq3 = 0.5*3.0*(sq3*snth+snps*csth)/(cs3th*dsbar*dsbar)
 end
 (dq1, dq2, dq3)
end


# ---
# title: 337. House Robber III
# id: problem337
# author: Tian Jun
# date: 2020-10-31
# difficulty: Medium
# categories: Dynamic Programming, Tree, Depth-first Search
# link: <https://leetcode.com/problems/house-robber-iii/description/>
# hidden: true
# ---
#
# The thief has found himself a new place for his thievery again. There is only
# one entrance to this area, called the "root." Besides the root, each house has
# one and only one parent house. After a tour, the smart thief realized that
# "all houses in this place forms a binary tree". It will automatically contact
# the police if two directly-linked houses were broken into on the same night.
#
# Determine the maximum amount of money the thief can rob tonight without
# alerting the police.
#
# **Example 1:**
#
#
#
#     Input: [3,2,3,null,3,null,1]
#
#          3
#         / \
#        2   3
#         \   \
#          3   1
#
#     Output: 7
#     Explanation:  Maximum amount of money the thief can rob = 3 + 3 + 1 = **7**.
#
# **Example 2:**
#
#
#
#     Input: [3,4,5,1,3,null,1]
#
#          3
#         / \
#        4   5
#       / \   \
#      1   3   1
#
#     Output: 9
#     Explanation:  Maximum amount of money the thief can rob = 4 + 5 = **9**.
#
#
#
## @lc code=start
using LeetCode

## add your code here:
## @lc code=end


using LBFGSB
using Test

# define a function
function f(x)
    n = length(x)
    y = 0.25 * (x[1] - 1)^2
    for i = 2:n
        y += (x[i] - x[i-1]^2)^2
    end
    4y
end

# and its gradient function that maps a vector x to a vector z
function g!(z, x)
    n = length(x)
    t₁ = x[2] - x[1]^2
    z[1] = 2 * (x[1] - 1) - 1.6e1 * x[1] * t₁
    for i = 2:n-1
        t₂ = t₁
        t₁ = x[i+1] - x[i]^2
        z[i] = 8 * t₂ - 1.6e1 * x[i] * t₁
    end
    z[n] = 8 * t₁
end

@testset "wrapper" begin
    # the first argument is the dimension of the largest problem to be solved
    # the second argument is the maximum number of limited memory corrections
    optimizer = L_BFGS_B(1024, 17)
    n = 25  # the dimension of the problem
    x = fill(Cdouble(3e0), n)  # the initial guess
    # set up bounds
    bounds = zeros(3, n)
    for i = 1:n
        bounds[1,i] = 2  # represents the type of bounds imposed on the variables:
                         #  0->unbounded, 1->only lower bound, 2-> both lower and upper bounds, 3->only upper bound
        bounds[2,i] = isodd(i) ? 1e0 : -1e2  #  the lower bound on x, of length n.
        bounds[3,i] = 1e2  #  the upper bound on x, of length n.
    end
    # call the optimizer
    # - m: the maximum number of variable metric corrections used to define the limited memory matrix
    # - factr: the iteration will stop when (f^k - f^{k+1})/max{|f^k|,|f^{k+1}|,1} <= factr*epsmch,
    #     where epsmch is the machine precision, which is automatically generated by the code. Typical values for factr:
    #     1e12 for low accuracy, 1e7 for moderate accuracy, 1e1 for extremely high accuracy
    # - pgtol: the iteration will stop when max{|proj g_i | i = 1, ..., n} <= pgtol where pg_i is the ith component of the projected gradient
    # - iprint: controls the frequency of output. iprint < 0 means no output:
    #     iprint = 0 print only one line at the last iteration
    #     0 < iprint < 99 print also f and |proj g| every iprint iterations
    #     iprint = 99 print details of every iteration except n-vectors
    #     iprint = 100 print also the changes of active set and final x
    #     iprint > 100 print details of every iteration including x and g
    # - maxfun: the maximum number of function evaluations
    # - maxiter: the maximum number of iterations
    fout, xout = optimizer(f, g!, x, bounds, m=5, factr=1e7, pgtol=1e-5, iprint=-1, maxfun=15000, maxiter=15000)
    @test fout ≈ 1.083490083518441e-9

    #testing the original wrapper:
    function g(x)
        n = length(x)
        z = zeros(n)
        t₁ = x[2] - x[1]^2
        z[1] = 2 * (x[1] - 1) - 1.6e1 * x[1] * t₁
        for i = 2:n-1
            t₂ = t₁
            t₁ = x[i+1] - x[i]^2
            z[i] = 8 * t₂ - 1.6e1 * x[i] * t₁
        end
        z[n] = 8 * t₁
        z
    end
    func(x) = (f(x), g(x))
    fout2, xout2 = optimizer(func, x, bounds, m=5, factr=1e7, pgtol=1e-5, iprint=-1, maxfun=15000, maxiter=15000)
    @test fout2 ≈ 1.083490083518441e-9
end

@testset "lbfgsb" begin
    #  0->unbounded, 1->only lower bound, 2-> both lower and upper bounds, 3->only upper bound
    lb=[-Inf,-Inf,-2.,6f0]
    ub=[Inf,3.,2,Inf]
    btyps=[0,3,2,1]

    @test LBFGSB.typ_bnd.(lb,ub) == btyps

    @test_throws ErrorException LBFGSB.typ_bnd(6,-1.)

    n,m=4,6
    o,b=LBFGSB._opt_bounds(n,m,lb,ub)

    @test size(b)==(3,n)
    @test b[2,:] == lb
    @test b[3,:] == ub
    @test b[1,:] == btyps

    od=L_BFGS_B(n,m)
    @test all(getproperty(od,i)==getproperty(o,i) for i in fieldnames(L_BFGS_B))

    fout,xout=lbfgsb(x->0.5sum(abs,x),(g,x)->g.=x,100*rand(3),lb=[0.5,-Inf,-0.1],ub=[Inf,0.2,Inf])
    @test fout ≈ 0.25
    @test xout ≈ [0.5,0,0]
 end


# ---
# jupyter:
#   jupytext:
#     formats: ipynb,jl:hydrogen
#     text_representation:
#       extension: .jl
#       format_name: hydrogen
#       format_version: '1.3'
#       jupytext_version: 1.11.2
#   kernelspec:
#     display_name: Julia 1.8.0-DEV
#     language: julia
#     name: julia-1.8
# ---

# %%
VERSION

# %%
using Plots
using Zygote

# %%
meshgrid(x, y) = reim(complex.(x', y))

x = 1:4
y = 10(1:3)
X, Y = meshgrid(x, y)
display(X)
display(Y)

# %%
function plotvf!(x, y, f; scale=1, kwargs...)
    X, Y = meshgrid(x, y)
    u(x, y) = scale * f(x, y)[1]
    v(x, y) = scale * f(x, y)[2]
    U = u.(X, Y)
    V = v.(X, Y)
    X -= U/2
    Y -= V/2
    quiver!(vec(X), vec(Y); quiver = (vec(U), vec(V)), kwargs...)
end

plotvf(x, y, f; kwargs...) = (plot(); plotvf!(x, y, f; kwargs...))

# %%
g(x, y) = (-y, x)
x = y = range(-2, 2, length=11)
plotvf(x, y, g; scale=0.2, size=(400, 400))

# %%
f(x, y) = x^3 - 3x + y^2
df(x, y) = gradient(f, x, y)
xs = ys = range(-2, 2, length=101)
heatmap(xs, ys, f; color=:rainbow)
x = y = range(-2, 2, length=21)[2:2:end]
plotvf!(x, y, df; scale=0.05, color=:white)
plot!(xlim=extrema(xs), ylim=extrema(ys))

# %% [markdown]
# For Plots.@recipe, see
#
# * https://docs.juliaplots.org/latest/recipes/
# * https://github.com/JuliaPlots/ExamplePlots.jl/blob/master/notebooks/usertype_recipes.ipynb
# * https://github.com/JuliaPlots/ExamplePlots.jl/blob/master/notebooks/type_recipes.ipynb
# * https://github.com/JuliaPlots/ExamplePlots.jl/blob/master/notebooks/series_recipes.ipynb
# * https://github.com/JuliaPlots/Plots.jl/blob/master/src/recipes.jl
# * https://nbviewer.jupyter.org/gist/genkuroki/521c4bf9160caae8f8c6591e78a9f1d1

# %%
module O

using Plots
meshgrid(x, y) = reim(complex.(x', y))

struct VectorField{X, Y, F, S} x::X; y::Y; f::F; scale::S end
VectorField(x, y, f; scale=0.2) = VectorField(x, y, f, scale)

@recipe function F(vf::VectorField)
    x, y, f, scale = vf.x, vf.y, vf.f, vf.scale
    X, Y = meshgrid(x, y)
    u(x, y) = scale * f(x, y)[1]
    v(x, y) = scale * f(x, y)[2]
    U = u.(X, Y)
    V = v.(X, Y)
    X -= U/2
    Y -= V/2

    seriestype := :quiver
    quiver --> (vec(U), vec(V))
    (vec(X), vec(Y))
end

end

# %%
x = y = range(-2, 2, length=11)
g(x, y) = (-y, x)
plot(O.VectorField(x, y, g); size=(400, 400))

# %%
f(x, y) = x^3 - 3x + y^2
df(x, y) = gradient(f, x, y)
xs = ys = range(-2, 2, length=101)
heatmap(xs, ys, f; color=:rainbow)
x = y = range(-2, 2, length=21)[2:2:end]
plot!(O.VectorField(x, y, df, 0.05); color=:white)
plot!(xlim=extrema(xs), ylim=extrema(ys))

# %%