#' ---
#' title: Benders Decomposition for Norm-1 Regression with ParameterJuMP.jl
#' author: Joaquim Dias Garcia
#' date: March 9th 2019
#' options:
#'     github:
#'         out_path : markdown
#'     md2html:
#'         out_path : html
#'     md2pdf:
#'         out_path : pdf
#' ---

#' # Introduction

#' This notebook is part of the talk on ParameterJuMP.jl in the
#' third annual JuMP-dev workshop, held in Santiago, Chile, 2019

#' The main purpose of this notebook is to show an application of
#' [ParameterJuMP.jl](https://github.com/JuliaStochOpt/ParameterJuMP.jl).
#' ParameterJuMP is well suited for Benders like decompositions
#' therefore we shall try to demonstrate the usage of the library
#' in one of the simplest problems that fits in the Benders
#' decomposition framework, Norm-1 regression, which is a particular
#' case of quantile regression.
#'
#' Note that this is NOT the standard technique to solve Norm-1
#' regressions. Taylor made methods are available
#' [here](https://cran.r-project.org/web/packages/quantreg/quantreg.pdf)
#' for instance.
#'

#' This notebook will require the following libraries:

#' ParameterJuMP itself
using ParameterJuMP

#' JuMP: the julia mathematical programming modeling tool
using JuMP

#' GLPK: A linear programming solver (other solvers could be used -
#' such as Clp, Xpress, Gurobi, CPLEX and so on)

#+ echo = false
# using Xpress
# const OPTIMIZER = Xpress.Optimizer;
#+

using GLPK
const OPTIMIZER = GLPK.Optimizer;

#' TimerOutputs: a time measuring library to demonstrate the advantage of using ParameterJuMP
using TimerOutputs

#' The following two julia default libraries
using LinearAlgebra # just use the dot function
using Random # to use random number generators

#' Plots library
using Plots
gr(); # plotting backend

#' # Norm-1 regression

#' We will apply Norm-1 regression to the
#' [Linear Regression](https://en.wikipedia.org/wiki/Linear_regression) problem.
#' Linear regression is a statistical tool to obtain the relation
#' between one **dependent variable** and other **explanatory variables**.
#' In other words, given a set of $n$ explanatory variables $X = \{ X_1, \dots, X_n \}$
#' we would like to obtain the best possible estimate for $Y$.
#' In order to accomplish such a task we make the hypothesis that $Y$
#' is approximately linear function of $X$:
#'
#' $Y = \sum_{j =1}^n \beta_j X_j + \varepsilon$
#'
#' where $\varepsilon$ is some random error.
#'
#' The estimation of the $\beta$ values relies on observations of the variables:
#' $\{y^i, x_1^i, \dots, x_n^i\}_i$
#'
#' In this notebook we will solve a problem where the explanatory variables
#' are sinusoids of differents frequencies

#' First, we define the number of explanatory variables and observations

#+ echo = false

# GLPK small
# const N_Candidates = 100
# const N_Observations = 600
# const N_Nodes = 100

# GLPK large
# const Candidates = 600
# const Observations = 5000
# const Nodes = 1000

# Xpress
# const N_Candidates = 700
# const N_Observations = 5000
# const N_Nodes = 1000

#+

const N_Candidates = 100
const N_Observations = 600
const N_Nodes = 100

const Observations = 1:N_Observations
const Candidates = 1:N_Candidates
const Nodes = 1:N_Nodes
;

#' Initialize a random number generator to keep results deterministic

rng = Random.MersenneTwister(123);

#' Building regressors (explanatory) sinusoids

const X = zeros(N_Candidates, N_Observations)
const time = [obs / N_Observations * 1 for obs in Observations]
for obs in Observations, cand in Candidates
    t = time[obs]
    f = cand
    X[cand, obs] = sin(2 * pi * f * t)
end

#' Define coefficients

β = zeros(N_Candidates)
for i ∈ Candidates
    if rand(rng) <= (1-i/N_Candidates)^2 && i<=100
        β[i] = 4*rand(rng)/i
    end
end
println("First coefs: $(β[1:min(10, N_Candidates)])")

#' Create noisy observations
const y = X' * β .+ 0.1*randn(rng, N_Observations)

plt = plot(time, y,
    xlabel = "Time (s)", ylabel = "Amplitude")
plot!(plt, time, X'[:,1])
plot!(plt, time, X'[:,3])
plot!(plt, time, X'[:,9])

#' The classic tool to estimate linear regression models is the
#' [Least Squares](https://en.wikipedia.org/wiki/Least_squares) method.
#'
#' The least squares method relies on solving the optimization problem:
#'
#' $\max \Bigg\{ \sum_{i \in Observations} \Big( y_i - \sum_{j \in Candidates} \beta_j x_{i,j} \Big) ^2 \Bigg\}$
#'
#' In Norm-1 regression, the quadratic functions are replaced by absolute values:
#'
#' $\max\Bigg\{ \sum_{i \in Observations} \Big| y_i - \sum_{j \in Candidates} \beta_j x_{i,j} \Big| \Bigg\}$
#'
#' This optimization problem can be recast as a [Linear Programming](https://en.wikipedia.org/wiki/Linear_programming) Problem:
#'
#' $$
#'\begin{align}
#'    & \min_{\varepsilon^{up}, \varepsilon^{dw}, \beta}  &&  \sum_{i \in Observations} {\varepsilon^{up}}_i + {\varepsilon^{dw}}_i  && \notag \\
#'    & \text{subject to}     &&  {\varepsilon^{up}}_i \geq + y_i - \sum_{j \in Candidates} \beta_j x_{i,j} && \forall i \in Observations \notag \\
#'    &                       &&  {\varepsilon^{dw}}_i \geq - y_i + \sum_{j \in Candidates} \beta_j x_{i,j} && \forall i \in Observations \notag \\
#'    &                       &&  {\varepsilon^{up}}_i, {\varepsilon^{dw}}_i \geq 0 && \forall i \in Observations \notag \\
#'\end{align}
#' $$
#'
#' Where $Observations$ is the set of all observations.
#'

#' This linear programming problem can be described in julia with JuMP
function full_model_regression()
    time_build = @elapsed begin # measure time to create a model

        # initialize a optimization model
        full_model = Model(with_optimizer(OPTIMIZER))

        # create optimization variables of the problem
        @variables(full_model, begin
            ɛ_up[Observations] >= 0
            ɛ_dw[Observations] >= 0
            β[1:N_Candidates]
            # 0 <= β[Candidates] <= 8
        end)

        # define constraints of the model
        @constraints(full_model, begin
            ɛ_up_ctr[i in Observations],
                ɛ_up[i] >= + sum(X[j,i] * β[j] for j ∈ Candidates) - y[i]
            ɛ_dw_ctr[i in Observations],
                ɛ_dw[i] >= - sum(X[j,i] * β[j] for j ∈ Candidates) + y[i]
        end)

        # construct the objective function to be minimized
        @objective(full_model, Min, sum(ɛ_up[i] + ɛ_dw[i] for i ∈ Observations))
    end

    # solve the problem
    time_solve = @elapsed optimize!(full_model)

    println("First coefficients in solution: $(value.(β)[1:min(10, N_Candidates)])")
    println("Objective value: $(objective_value(full_model))")
    println("Time in solve: $time_solve")
    println("Time in build: $time_build")

    return nothing
end

#' Now we execute the function that builds the model and solves it

N_Observations*N_Candidates < 10_000_000 && full_model_regression()

#' # Benders decomposition

#' Benders decomposition is used to solve large optimization problems
#' with some special characteristics.
#' LP's can be solved with classical linear optimization methods
#' such as the Simplex method or Interior point methods provided by
#' solvers like GLPK.
#' However, these methods do not scale linearly with the problem size.
#' In the Benders decomposition framework we break the problem in two pieces:
#' A master and a slave problem.
#'
#' Of course some variables will belong to both problems, this is where the
#' cleverness of Benders kicks in:
#' The master problem is solved and passes the shared variables to the slave.
#' The slave problem is solved with the shared variables FIXED to the values
#' given by the master problem. The solution of the slave problem can be used
#' to generate a constraint to the master problem to describe the linear
#' approximation of the cost function of the shared variables.
#' In many cases, like stochastic programming, the slave problems have a
#' interesting structure and might be broken in smaller problem to be solved
#' in parallel.

#' We will descibe the decomposition similarly to what is done in:
#' Introduction to Linear Optimization, Bertsimas & Tsitsiklis (Chapter 6.5):
#' Where the problem in question has the form
#'
#' $$
#'\begin{align}
#'    & \min_{x, y_k}     &&  c^T x && + f_1^T y_1 && + \dots && + f_n^T y_n  &&  \notag \\
#'    & \text{subject to} &&  Ax    &&             &&         &&              && = b \notag \\
#'    &                   &&  B_1 x && + D_1 y_1   &&         &&              && = d_1 \notag \\
#'    &                   &&  \dots &&             &&  \dots  &&              &&       \notag \\
#'    &                   &&  B_n x &&             &&         && + D_n y_n    && = d_n \notag \\
#'    &                   &&   x,   &&     y_1,    &&         &&       y_n    && \geq 0 \notag \\
#'\end{align}
#' $$
#'

#' ### Slave

#' Given a solution for the $x$ variables we can define the slave problem as
#'
#'
#' $$
#'\begin{align}
#' z_k(x) \ = \ & \min_{y_k}        &&  f_k^T y_k &&                  \notag \\
#'              & \text{subject to} &&  D_k y_k   &&  = d_k - B_k x \notag \\
#'              &                   &&  y_k       && \geq 0          \notag \\
#'\end{align}
#' $$
#'
#'
#' The $z_k(x)$ function represents the cost of the subproblem given a
#' solution for $x$. This function is a convex function because $x$
#' affects only the right hand side of the problem (this is a standard
#' results in LP theory).
#'
#' For the special case of the Norm-1 reggression the problem is written as:
#'
#' $$
#'\begin{align}
#' z_k(\beta) \ = \ & \min_{\varepsilon^{up}, \varepsilon^{dw}}         &&  \sum_{i \in ObsSet(k)} {\varepsilon^{up}}_i + {\varepsilon^{dw}}_i               && \notag \\
#'                  & \text{subject to}     &&  {\varepsilon^{up}}_i \geq + y_i - \sum_{j \in Candidates} \beta_j x_{i,j} && \forall i \in ObsSet(k) \notag \\
#'                  &                       &&  {\varepsilon^{dw}}_i \geq - y_i + \sum_{j \in Candidates} \beta_j x_{i,j} && \forall i \in ObsSet(k) \notag \\
#'                  &                       &&  {\varepsilon^{up}}_i, {\varepsilon^{dw}}_i \geq 0                             && \forall i \in ObsSet(k) \notag \\
#'\end{align}
#' $$
#'
#' The collection $ObsSet(k)$ is a sub-set of the N_Observations.
#' Any partition of the N_Observations collection is valid.
#' In this notebook we will partition with the function:

function ObsSet(K)
    obs_per_block = div(N_Observations, N_Nodes)
    return (1 + (K - 1) * obs_per_block):(K * obs_per_block)
end

#'
#' Which can be written in JuMP as follows.
#'
#' At this point we make a small detour to highlight the ParameterJuMP
#' application. Every time you a find a IF block with the flag `PARAM`
#' it means that we have two different implementations of the method:
#' one relying on ParameterJuMP and the other using pure JuMP.
#'
#'

function slave_model(PARAM, K)

    # initialize the JuMP model
    slave = if PARAM
        # special constructor exported by ParameterJuMP
        # to add the functionality to the model
        ModelWithParams(with_optimizer(OPTIMIZER))
    else
        # regular JuMP constructor
        Model(with_optimizer(OPTIMIZER))
    end

    # Define local optimization variables for norm-1 error
    @variables(slave, begin
        ɛ_up[ObsSet(K)] >= 0
        ɛ_dw[ObsSet(K)] >= 0
    end)

    # create the regression coefficient representation
    if PARAM
        # here is the main constructor of the Parameter JuMP packages
        # It will create model *parameters* instead of variables
        # Variables are added to the optimization model, while parameters
        # are not. Parameters are merged with LP problem constants and do not
        # increase the model dimensions.
        β = add_parameters(slave, zeros(N_Candidates))
    else
        # Create fixed variables
        @variables(slave, begin
            β[Candidates]
            β_fixed[1:N_Candidates] == 0
        end)
        @constraint(slave, β_fix[i in Candidates], β[i] == β_fixed[i])
    end

    # create local constraints
    # Note that *parameter* algebra is implemented just like variables
    # algebra. We can multiply parameters by constants, add parameters,
    # sum parameters and varaibles and so on.
    @constraints(slave, begin
        ɛ_up_ctr[i in ObsSet(K)],
            ɛ_up[i] >= + sum(X[j,i] * β[j] for j ∈ Candidates) - y[i]
        ɛ_dw_ctr[i in ObsSet(K)],
            ɛ_dw[i] >= - sum(X[j,i] * β[j] for j ∈ Candidates) + y[i]
    end)
    # ATTENTION β[j] * X[j,i] Is much slower

    # create local objective function
    @objective(slave, Min, sum(ɛ_up[i] + ɛ_dw[i] for i ∈ ObsSet(K)))

    # return the correct group of parameters
    if PARAM
        return (slave, β)#β_upper, β_lower)
    else
        return (slave, β, β_fixed, β_fix)#β_upper, β_lower)
    end
end

#' ### Master

#' Now that all pieces of the original problem can be representad by
#' the convex $z_k(x)$ functions we can recast the problem in the the equivalent form:
#'
#' $$
#'\begin{align}
#'  & \min_{x}          &&  c^T x + z_1(x) + \dots + z_n(x) && \notag \\
#'  & \text{subject to} &&  Ax = b                          && \notag \\
#'  &                   &&  x \geq 0                        && \notag \\
#'\end{align}
#' $$
#'
#' However we cannot pass a problem in this form to a linear programming
#' solver (it could be passed to other kinds of solvers).
#'
#' Another standart result of optimization theory is that a convex function
#' can be represented by its supporting hyper-planes:
#'
#' $$
#'\begin{align}
#'  z_k(x) \ = \ & \min_{z, x}       &&  z && \notag \\
#'               & \text{subject to} &&  z \geq \pi_k(\hat{x}) (x - \hat{x}) + z_k(\hat{x}), \ \forall \hat{x} \in dom(z_k) && \notag \\
#'\end{align}
#' $$
#'
#' Then we can re-write (again) the master problem as
#'
#' $$
#'\begin{align}
#'  & \min_{x, z_k}     &&  c^T x + z_1 + \dots + z_n \notag \\
#'  & \text{subject to} &&  z_i \geq \pi_i(\hat{x}) (x - \hat{x}) + z_i(\hat{x}), \ \forall \hat{x} \in dom(z_i), i \in \{1, \dots, n\} \notag \\
#'  &                   &&  Ax = b \notag \\
#'  &                   &&  x \geq 0 \notag \\
#'\end{align}
#' $$
#'
#' Which is a linear program!
#'
#' However, it has infinitely many constraints !!!
#'
#' We can relax the infinite constraints and write:
#'
#' $$
#'\begin{align}
#'  & \min_{x, z_k}     &&  c^T x + z_1 + \dots + z_n \notag \\
#'  & \text{subject to} &&  Ax = b \notag \\
#'  &                   &&  x \geq 0 \notag \\
#'\end{align}
#' $$
#'
#' But now its only an underestimated problem.
#' In the case of our problem it can be written as:
#'
#' $$
#'\begin{align}
#'  & \min_{\varepsilon, \beta} &&  \sum_{i \in Nodes} \varepsilon_i \notag \\
#'  & \text{subject to} &&  \varepsilon_i \geq 0 \notag \\
#'\end{align}
#' $$
#'
#' This model can be written in JuMP
#'

function master_model(PARAM)
    master = Model(with_optimizer(OPTIMIZER))
    @variables(master, begin
        ɛ[Nodes] >= 0
        β[1:N_Candidates]
        # 0 <= β[Candidates] <= 8
    end)
    @objective(master, Min, sum(ɛ[i] for i ∈ Nodes))
    sol = zeros(N_Candidates)
    return (master, ɛ, β, sol)
end

#' The method to solve the master problem and query its solution
#' is given here:

function master_solve(PARAM, master_model)
    model = master_model[1]
    β = master_model[3]
    optimize!(model)
    return (value.(β), objective_value(model))
end

#' ### Supporting Hyperplanes

#' With these building blocks in hand, we can start building the algorithm.
#'
#' So far we know how to:
#' - Solve the relaxed master problem
#' - Obtain the solution for the $\hat{x}$ (or $\beta$ in our case)
#'
#'
#' Now we can:
#' - Fix the values of $\hat{x}$ in the slave problems
#' - Solve the slave problems
#' - query the solution of the slave problems to obtain the supporting hyperplane
#'
#' the value of $z_k(\hat{x})$, which is the objective value of the slave problem
#'
#' and the derivative $\pi_k(\hat{x}) = \frac{d z_k(x)}{d x} \Big|_{x = \hat{x}}$
#' The derivative is the dual variable associated to the variable $\hat{x}$,
#' which results by applying the chain rule on the constraints duals.
#'
#' These new steps are executed by the function:

function slave_solve(PARAM, model, master_solution)
    β0 = master_solution[1]
    slave = model[1]

    # The first step is to fix the values given by the master problem
    @timeit "fix" if PARAM
        # *parameters* can be set to new values and the optimization
        # model will be automatically updated
        β = model[2]
        fix.(β, β0)
    else
        # JuMP also has the hability to fix variables to new values
        β_fixed = model[3]
        β_fix = model[4]
        fix.(β_fixed, β0)
    end

    # here the slave problem is solved
    @timeit "opt" optimize!(slave)

    # query dual variables, which are sensitivities
    # They represent the subgradient (almost a derivative)
    # of the objective function for infinitesimal variations
    # of the constants in the linear constraints
    @timeit "dual" if PARAM
        # we can query dual values of *parameters*
        π = dual.(β)
    else
        # or, in pure JuMP, we query the duals form
        # constraints that fix the values of our regression
        # coefficients
        π = dual.(β_fix)
    end

    # π2 = shadow_price.(β_fix)
    # @show sum(π .- π2)
    obj = objective_value(slave)
    rhs = obj - dot(π, β0)
    return (rhs, π, obj)
end

#'
#' Now that we have cutting plane in hand we can add them to the master problem:
#'

function master_add_cut(PARAM, master_model, cut_info, node)
    master = master_model[1]
    ɛ = master_model[2]
    β = master_model[3]

    rhs = cut_info[1]
    π = cut_info[2]

    @constraint(master,
        ɛ[node] >= sum(π[j] * β[j] for j ∈ Candidates) + rhs)
end

#' ### Algorithm wrap up

#'
#' The complete algorithm is
#'
#'
#' - Solve the relaxed master problem
#' - Obtain the solution for the $\hat{x}$ (or $\beta$ in our case)
#' - Fix the values of $\hat{x}$ in the slave problems
#' - Solve the slave problem
#' - query the solution of the slave problem to obtain the supporting hyperplane
#' - add hyperplane to master problem
#' - repeat
#'

#' Now we grab all the pieces that we built and we write the benders
#' algorithm by calling the above function in a proper order.
#'
#' The macros `@timeit` are use to time each step of the algorithm.

function decomposed_model(PARAM)
    reset_timer!() # reset timer fo comparision
    time_init = @elapsed @timeit "Init" begin
        println("Initialize decomposed model")

        # Create the master problem with no cuts
        println("Build master problem")
        @timeit "Master" master = master_model(PARAM)

        # initialize solution for the regression coefficients in zero
        println("Build initial solution")
        @timeit "Sol" solution = (zeros(N_Candidates), Inf)
        best_sol = deepcopy(solution)

        # Create the slave problems
        println("Build slave problems")
        @timeit "Slaves" slaves = [slave_model(PARAM, i) for i ∈ Candidates]

        # Save initial version of the slave problems and create
        # the first set of cuts
        println("Build initial cuts")
        @timeit "Cuts" cuts = [slave_solve(PARAM, slaves[i], solution) for i ∈ Candidates]
    end

    UB = +Inf
    LB = -Inf

    println("Initialize Iterative step")
    time_loop = @elapsed @timeit "Loop"  for k in 1:80

        # Add cuts generated from each slave problem to the master problem
        @timeit "add cuts" for i ∈ Candidates
            master_add_cut(PARAM, master, cuts[i], i)
        end

        # Solve the master problem with the new set of cuts
        # Obtain new solution candidate for the regression coefficients
        @timeit "solve master" solution = master_solve(PARAM, master)

        # Pass the new candidate solution to each of the slave problems
        # Solve the slave problems and obtain cutting planes
        # @show solution[2]
        @timeit "solve nodes" for i ∈ Candidates
            cuts[i] = slave_solve(PARAM, slaves[i], solution)
        end

        LB = solution[2]
        new_UB = sum(cuts[i][3] for i ∈ Candidates)
        if new_UB <= UB
            best_sol = deepcopy(solution)
        end
        UB = min(UB, new_UB)
        println("Iter = $k, LB = $LB, UB = $UB")

        if abs(UB - LB)/(abs(UB)+abs(LB)) < 0.05
            println("Converged!")
            break
        end
    end
    println("First coefficients in solution: $(solution[1][1:min(10, N_Candidates)])")
    println("Objective value: $(solution[2])")
    println("Time in loop: $time_loop")
    println("Time in init: $time_init")

    print_timer()

    return best_sol[1]
end

#' Run benders decomposition with pure JuMP

GC.gc()
β1 = decomposed_model(false);

#' Run benders decomposition with ParameterJuMP

GC.gc()
β2 = decomposed_model(true);

#' Plot resulting time series from the benders base estimations

const y1 = X' * β1
const y2 = X' * β2

plt = plot(time, y,
    xlabel = "Time (s)", ylabel = "Amplitude")
plot!(plt, time, y1)
plot!(plt, time, y2)

#' # Acknowledgments

#' ParameterJuMP was developed by Joaquim Dias Garcia (@joaquimg) and Benoît Legat (@blegat)

#+ echo=false
# using Weave; cd("C:\\Users\\joaquimgarcia\\.julia\\dev\\ParameterJuMP\\examples"); weave("benders_quatile_regression.jl"); weave("benders_quatile_regression.jl", doctype = "github");# weave("benders_quatile_regression.jl", doctype = "md2pdf")
#+


# This file is a part of RunStatistics.jl, licensed under the MIT License (MIT).

"""
    t_obs(X::AbstractArray, μ::Real, σ2::Real)

Compute the value of the *Squares test statistic* `T_obs` i.e. the largest `χ2` of any run of consecutive successes (above expectation)
in a sequence of `N` independent trials with Gaussian uncertainty. `μ` and `σ2` are the expectation and variance of the observations.

Find the location(s) of the run(s) that produces `T_obs`.

Returns a tuple containing `T_obs` and one or more arrays containing the indices of the runs that produce `T_obs`.

For the Squares statistic to be calculable, the observed data must satisfy following conditions:

        All observations {X_i} are independent.
        Each observation is normally distributed, X_i ∼ N( µ_i, σ^2_i ).
        Mean µ_i and variance σ^2_i are known.

In case the observations {X_i} have individual expectations and variances, use:

    t_obs(X::AbstractArray, μ::AbstractArray, σ2::AbstractArray)

With `μ[i]` and `σ2[i]` being the mean and variance of the i-th element of `X`.

See:

Frederik Beaujean and Allen Caldwell. *A Test Statistic for Weighted Runs.*
Journal of Statistical Planning and Inference 141, no. 11 (November 2011): 3437–46.

https://www.sciencedirect.com/science/article/abs/pii/S0378375811001935?via%3Dihub

https://arxiv.org/abs/1005.3233
"""
function t_obs(X::AbstractArray, μ::Real, σ2::Real)

    T = float(promote_type(eltype(X), eltype(μ), eltype(σ2)))
    χ2 = T[]
    χi = zero(T)

    locs_cache = Array[]
    locs_run = Integer[]
    locs = Any[]

    @inbounds for i in eachindex(X)

        if X[i] > μ

            χi += (X[i] - μ)^2 / σ2
            append!(locs_run, i)

        else
            χi > 0 && append!(χ2, χi)
            isempty(locs_run) || append!(locs_cache, [locs_run])
            χi = 0
            locs_run = Integer[]
        end

    end

    locs_ids = findall(x -> x == maximum(χ2), χ2)

    if length(locs_ids) == 1

        return maximum(χ2), locs_cache[locs_ids[1]]

    else

        for i in locs_ids

            append!(locs, locs_cache[i])
        end

    end

    return maximum(χ2), locs
end

function t_obs(X::AbstractArray, μ::AbstractArray, σ2::AbstractArray)

    @argcheck axes(X) == axes(μ) == axes(σ2)
    T = float(promote_type(eltype(X), eltype(μ), eltype(σ2)))

    χ2 = T[]
    χi = zero(T)

    locs_cache = AbstractArray[]
    locs_run = Integer[]
    locs = AbstractArray[]

    @inbounds for i in eachindex(X)

        if  X[i] > μ[i]

            χi += (X[i] - μ[i])^2 / σ2[i]
            append!(locs_run, i)
        else

            χi > 0 && append!(χ2, χi)
            isempty(locs_run) || append!(locs_cache, [locs_run])
            χi = 0
            locs_run = Integer[]
        end
    end

    locs_ids = findall(x -> x == maximum(χ2), χ2)

    if length(locs_ids) == 1

        return maximum(χ2), locs_cache[locs_ids[1]]

    else
        for i in locs_ids

            append!(locs, locs_cache[i])
        end
    end

    return maximum(χ2), locs
end


using PyPlot

struct Vertex
    neighbors::Vector{Int}       # Indices of neighbors of this Vertex
    coordinates::Vector{Float64} # 2D coordinates of this Vertex - only for plotting
    Vertex(neighbors; coordinates=[0,0]) = new(neighbors, coordinates)
end

function Base.show(io::IO, v::Vertex)
    print(io, "Neighbors = ", v.neighbors)
end

struct Graph
    vertices::Vector{Vertex}
end

function Base.show(io::IO, g::Graph)
    for i = 1:length(g.vertices)
        println(io, "Vertex $i, ", g.vertices[i])
    end
end

function PyPlot.plot(g::Graph; scale=1.0)
    fig, ax = subplots()
    ax.set_aspect("equal")

    xmin = minimum(v.coordinates[1] for v in g.vertices)
    xmax = maximum(v.coordinates[1] for v in g.vertices)
    ymin = minimum(v.coordinates[2] for v in g.vertices)
    ymax = maximum(v.coordinates[2] for v in g.vertices)
    sz = max(xmax-xmin, ymax-ymin)
    cr = scale*0.05sz
    hw = cr/2
    axis([xmin-2cr,xmax+2cr,ymin-2cr,ymax+2cr])
    axis("off")

    for i in 1:length(g.vertices)
        c = g.vertices[i].coordinates
        ax.add_artist(matplotlib.patches.Circle(c, cr, facecolor="none", edgecolor="k"))
        ax.text(c[1], c[2], string(i),
                horizontalalignment="center", verticalalignment="center", fontsize=round(Int, 14*scale))
        for nb in g.vertices[i].neighbors
            cnb = g.vertices[nb].coordinates
            dc = cnb .- c
            L = sqrt(sum(dc.^2))
            c1 = c .+ cr/L * dc
            c2 = cnb .- cr/L * dc
            arrow(c1[1], c1[2], c2[1]-c1[1], c2[2]-c1[2],
                  head_width=hw, length_includes_head=true, facecolor="k")
        end
    end
end

function circle_graph(nv=8)
    g = Graph([])
    for i = 1:nv
        th = 2π*i/nv
        v = Vertex([mod(i,nv)+1, mod(i-2,nv)+1], coordinates=[cos(th), sin(th)])
        push!(g.vertices, v)
    end
    return g
end

g = circle_graph(10)

# Add edges by growing the neighbors array
push!(g.vertices[1].neighbors, 6);       # Add edge 1->6
append!(g.vertices[4].neighbors, [2,7]); # Add 2 new edges

# Change edges by modifying the neighbors array
g.vertices[3].neighbors[1] = 1           # Change edge, 3->1 instead of 3->4
pop!(g.vertices[5].neighbors)            # Remove edge

# Add a new vertex by growing the vertices array, and if needed
# updating some of the existing neighbors array
# Note that the new vertex should be numbered last, otherwise
# all node numbers must be reordered
newv = Vertex([1,4,9], coordinates=[-.1,.3])
push!(g.vertices, newv)
push!(g.vertices[2].neighbors, 11)

# Deleting a vertex is difficult using adjancency lists
# (unless it is the last vertex), since it requires renumbering
# of the vertices. But as a compromise, we can remove all edges
# to/from a vertex:
resize!(g.vertices[8].neighbors,0)
for v in g.vertices
    filter!(i -> i != 8, v.neighbors)
end


# kuramoto_7-27-19.jl

# Starting off the day strong with some code from my other project,
# grok-the-dot.

function f(x)
  return 2x+1
end

function g(x)
  return x^2
end

function h(x)
  return 4x^2 + 4x + 11   # (2x + 1)^2 = 4x^2 + 4x + 11
end

G = collect(1:1_000_000)  # You can use underscores to
                          # separate out numbers for easier
                          # reading. It's lowkey one of my
                          # favorite features.

@time h.(G)
@time g.(f.(G))

@time map(λ -> λ^2, f.(G))
@time map(λ -> λ^2, map(λ -> 2λ+1, G))

#----------------------------------------------------------#


using LinearAlgebra

export Scale, factor

"""
    Scale{S <: Union{Number, Val}, D, BT <: AbstractBlock{D}} <: TagBlock{BT, D}

`Scale` a block with scalar. it can be either a `Number` or a compile time `Val`.

# Example

```jldoctest; setup=:(using YaoBlocks)
julia> 2 * X
[scale: 2] X

julia> im * Z
[+im] Z

julia> -im * Z
[-im] Z

julia> -Z
[-] Z
```
"""
mutable struct Scale{S<:Union{Number,Val},D,BT<:AbstractBlock{D}} <: TagBlock{BT,D}
    alpha::S
    content::BT
end

content(x::Scale) = x.content
factor(x::Scale{<:Number}) = x.alpha
factor(x::Scale{Val{X}}) where {X} = X

# parameter interface
getiparams(s::Scale{<:Number}) = (factor(s),)
setiparams(s::Scale{<:Number}, alpha::Number) = Scale(alpha, s.content)
setiparams!(s::Scale{T1}, alpha::T2) where {T1<:Number, T2<:Number} = (s.alpha = T1(alpha); s)

Base.copy(x::Scale) = Scale(x.alpha, copy(x.content))
Base.adjoint(x::Scale{<:Number}) = Scale(adjoint(x.alpha), adjoint(content(x)))
Base.adjoint(x::Scale{Val{X}}) where {X} = Scale(Val(adjoint(X)), adjoint(content(x)))

YaoBase.ishermitian(s::Scale) =
    (ishermitian(s |> content) && ishermitian(s |> factor)) || ishermitian(mat(s))
YaoBase.isunitary(s::Scale) =
    (isunitary(s |> content) && isunitary(s |> factor)) || isunitary(mat(s))
YaoBase.isreflexive(s::Scale) =
    (isreflexive(s |> content) && isreflexive(s |> factor)) || isreflexive(mat(s))
YaoBase.iscommute(x::Scale, y::Scale) = iscommute(x |> content, y |> content)
YaoBase.iscommute(x::AbstractBlock, y::Scale) = iscommute(x, y |> content)
YaoBase.iscommute(x::Scale, y::AbstractBlock) = iscommute(x |> content, y)

Base.:(==)(x::Scale, y::Scale) = (factor(x) == factor(y)) && (content(x) == content(y))

chsubblocks(x::Scale, blk::AbstractBlock) = Scale(x.alpha, blk)
cache_key(x::Scale) = (factor(x), cache_key(content(x)))

mat(::Type{T}, x::Scale) where {T} = T(x.alpha) * mat(T, content(x))
mat(::Type{T}, x::Scale{Val{S}}) where {T,S} = T(S) * mat(T, content(x))

function _apply!(r::AbstractArrayReg, x::Scale{S}) where {S}
    _apply!(r, content(x))
    regscale!(r, factor(x))
    return r
end


using DiffEqJump, DelaySSAToolkit
using Random, Distributions

rn = @reaction_network begin
    kon, Goff --> Gon
    koff, Gon --> Goff
    ρ, Gon --> Gon + N
end kon koff ρ
jumpsys = convert(JumpSystem, rn, combinatoric_ratelaws = false)

u0 = [1,0,0]
de_chan0 = [[]]
tf = 2000.
tspan = (0,tf)
p = [0.0282,0.609,2.11]
dprob = DiscreteProblem(u0, tspan, p)

delay_trigger_affect! = function (integrator, rng)
    τ=rand(LogNormal(1,sqrt(2)))+120
    append!(integrator.de_chan[1], τ)
end
delay_trigger = Dict(3=>delay_trigger_affect!)
delay_complete = Dict(1=>[3=>-1])
delay_interrupt = Dict()
delayjumpset = DelayJumpSet(delay_trigger,delay_complete,delay_interrupt)

djprob = DelayJumpProblem(jumpsys, dprob, DelayRejection(), delayjumpset, de_chan0, save_positions=(false,false))

ensprob = EnsembleProblem(djprob)
@time ens = solve(ensprob, SSAStepper(), EnsembleThreads(), trajectories=10^5)

using DifferentialEquations.EnsembleAnalysis, Plots; theme(:vibrant)
last_slice = componentwise_vectors_timepoint(ens, tf)
histogram(last_slice[3],bins=0:1:60,normalize=:pdf, label = "LogNormal(1,sqrt(2))",fillalpha = 0.6, linecolor = :orange,fmt=:svg,xlabel = "# of products", ylabel = "Probability")
# savefig("docs/src/assets/stochastic_delay1.svg")

delay_trigger_affect2! = function (integrator, rng)
    τ=rand(LogNormal(0,2))+120
    append!(integrator.de_chan[1], τ)
end
delay_trigger2 = Dict(3=>delay_trigger_affect2!)
delayjumpset2 = DelayJumpSet(delay_trigger2,delay_complete,delay_interrupt)

djprob2 = DelayJumpProblem(dprob, DelayRejection(), jumpset, delayjumpset2, de_chan0, save_positions=(false,false))

@time ens2 = solve(EnsembleProblem(djprob2), SSAStepper(), EnsembleThreads(), trajectories=10^5)

last_slice2 = componentwise_vectors_timepoint(ens2, tf)
histogram(last_slice2[3],bins=0:1:60,normalize=:pdf, label = "LogNormal(0,2)",fillalpha = 0.6, linecolor = :orange,fmt=:svg,xlabel = "# of products", ylabel = "Probability")
# savefig("docs/src/assets/stochastic_delay2.svg")


"""
    StudentTLikelihood(ν::T, σ::Real=one(T))

## Arguments
- `ν::Real` : degrees of freedom of the student-T
- `σ::Real` : standard deviation of the local scale

[Student-t likelihood](https://en.wikipedia.org/wiki/Student%27s_t-distribution) for regression:
```math
    p(y|f,ν,σ) = Γ(0.5(ν+1))/(sqrt(νπ) σ Γ(0.5ν)) * (1+(y-f)^2/(σ^2ν))^(-0.5(ν+1))
```
`ν` is the number of degrees of freedom and `σ` is the standard deviation for local scale of the data.

---

For the analytical solution, it is augmented via:
```math
    p(y|f,ω) = N(y|f,σ^2 ω)
```
Where `ω ~ IG(0.5ν,,0.5ν)` where `IG` is the inverse gamma distribution
See paper [Robust Gaussian Process Regression with a Student-t Likelihood](http://www.jmlr.org/papers/volume12/jylanki11a/jylanki11a.pdf)
"""
mutable struct StudentTLikelihood{T<:Real,A<:AbstractVector{T}} <: RegressionLikelihood{T}
    ν::T
    α::T
    σ::T
    c::A
    θ::A
    function StudentTLikelihood{T}(ν::T, σ::T=one(T)) where {T<:Real}
        ν > 0.5 || error("ν should be greater than 0.5")
        return new{T,Vector{T}}(ν, (ν + one(T)) / 2.0, σ)
    end
    function StudentTLikelihood{T}(
        ν::T, σ::T, c::A, θ::A
    ) where {T<:Real,A<:AbstractVector{T}}
        return new{T,A}(ν, (ν + one(T)) / 2.0, σ, c, θ)
    end
end

function StudentTLikelihood(ν::T, σ::T=one(T)) where {T<:Real}
    return StudentTLikelihood{T}(ν, σ)
end

function implemented(
    ::StudentTLikelihood, ::Union{<:AnalyticVI,<:QuadratureVI,<:GibbsSampling}
)
    return true
end

function init_likelihood(
    likelihood::StudentTLikelihood{T},
    inference::AbstractInference{T},
    ::Int,
    nSamplesUsed::Int,
) where {T}
    if inference isa AnalyticVI || inference isa GibbsSampling
        StudentTLikelihood{T}(
            likelihood.ν, likelihood.σ, rand(T, nSamplesUsed), zeros(T, nSamplesUsed)
        )
    else
        StudentTLikelihood{T}(likelihood.ν, likelihood.σ)
    end
end

function (l::StudentTLikelihood)(y::Real, f::Real)
    return gamma(l.α) / (sqrt(l.ν * π) * gamma(l.ν / 2)) * (1 + abs2((y - f) / l.σ))^(-l.α)
    # tdistpdf(l.ν, (y - f) / l.σ) uses R so not differentiable
end

function Distributions.loglikelihood(l::StudentTLikelihood, y::Real, f::Real)
    return log(l(y, f))
    # tdistlogpdf(l.ν, (y - f) / l.σ) uses R so not differentiable
end

function Base.show(io::IO, model::StudentTLikelihood{T}) where {T}
    return print(io, "Student-t likelihood")
end

function compute_proba(
    l::StudentTLikelihood{T}, μ::AbstractVector{<:Real}, σ²::AbstractVector{<:Real}
) where {T<:Real}
    return μ, max.(σ², zero(σ²)) .+ 0.5 * l.ν * l.σ^2 / (0.5 * l.ν - 1)
end

## Local Updates ##

function local_updates!(
    l::StudentTLikelihood{T}, y::AbstractVector, μ::AbstractVector, diag_cov::AbstractVector
) where {T}
    l.c .= 0.5 * (diag_cov + abs2.(μ - y) .+ l.σ^2 * l.ν)
    return l.θ = l.α ./ l.c
end

function sample_local!(
    l::StudentTLikelihood{T}, y::AbstractVector, f::AbstractVector
) where {T}
    l.c .= rand.(InverseGamma.(l.α, 0.5 * (abs2.(f - y) .+ l.σ^2 * l.ν)))
    set_ω!(l, inv.(l.c))
    return nothing
end

## Global Gradients ##

@inline ∇E_μ(l::StudentTLikelihood, ::AOptimizer, y::AbstractVector) = (l.θ .* y,)
@inline ∇E_Σ(l::StudentTLikelihood, ::AOptimizer, ::AbstractVector) = (0.5 .* l.θ,)

## ELBO Section ##

function expec_loglikelihood(
    l::StudentTLikelihood{T},
    ::AnalyticVI,
    y::AbstractVector,
    μ::AbstractVector,
    diag_cov::AbstractVector,
) where {T}
    tot = -0.5 * length(y) * (log(twoπ * l.σ^2))
    tot += -sum(log.(l.c) .- digamma(l.α))
    tot +=
        -0.5 * (
            dot(l.θ, diag_cov) + dot(l.θ, abs2.(μ)) - 2.0 * dot(l.θ, μ .* y) +
            dot(l.θ, abs2.(y))
        )
    return tot
end

AugmentedKL(l::StudentTLikelihood, ::AbstractVector) = InverseGammaKL(l)

function InverseGammaKL(l::StudentTLikelihood{T}) where {T}
    α_p = l.ν / 2
    β_p = α_p * l.σ^2
    return InverseGammaKL(l.α, l.c, α_p, β_p)
end

## PDF and Log PDF Gradients ## (verified gradients)

function ∇loglikelihood(l::StudentTLikelihood{T}, y::Real, f::Real) where {T<:Real}
    return (one(T) + l.ν) * (y - f) / ((f - y)^2 + l.σ^2 * l.ν)
end

function hessloglikelihood(l::StudentTLikelihood{T}, y::Real, f::Real) where {T<:Real}
    v = l.ν * l.σ^2
    Δ² = (f - y)^2
    return (one(T) + l.ν) * (-v + Δ²) / (v + Δ²)^2
end


using BenchmarkTools
using LinearAlgebra

function g(x::Array{Float64, 1})
    y = zeros(length(x))
    z = Diagonal(ones(length(x)))
    q = ones(length(x))
    y .= z * x .+ q
end

function g_notyped(x)
    y = similar(x)
    z = I
    q = ones(eltype(x), length(x))
    y  .= z * x .+ q
end


function day8()
    day8lines = filter(!isempty, strip.(readlines("AoCdata/AoC_2021_day8.txt")))
    part = [0, 0]

    patterns = [
        [String.(sort.(collect.(split(a)))) for a in split(line, "|")] for line in day8lines
    ]

    count1 = sum(count(length(x) == 2 for x in a[2]) for a in patterns)
    count4 = sum(count(length(x) == 4 for x in a[2]) for a in patterns)
    count7 = sum(count(length(x) == 3 for x in a[2]) for a in patterns)
    count8 = sum(count(length(x) == 7 for x in a[2]) for a in patterns)
    part[1] = sum([count1, count4, count7, count8])

    eachlinedig = Vector{String}[]
    for line in patterns
        dig = ["" for _ in 1:10]
        while any(dig .== "")
            for s in vcat(line...)
                length(s) == 2 && (dig[2] = s) # 1
                length(s) == 3 && (dig[8] = s) # 7
                length(s) == 4 && (dig[5] = s) # 4
                length(s) == 7 && (dig[9] = s) # 8
                if length(s) == 6 # 0, 6 or 9
                    if dig[5] != ""
                        if count(c -> c in s, dig[5]) == 4
                            dig[10] = s  # 9
                        elseif dig[2] != ""
                            if count(c -> c in s, dig[2]) == 2
                                dig[1] = s # 0
                            else
                                dig[7] = s # 6
                            end
                        end
                    end
                elseif length(s) == 5  # 2 or 3 or 5
                    if dig[2] != ""
                        if count(c -> c in s, dig[2]) == 2
                            dig[4] = s # 3
                        elseif dig[5] != ""
                            if count(c -> c in s, dig[5]) == 3
                                dig[6] = s # 5
                            else
                                dig[3] = s # 2
                            end
                        end
                    end
                end
            end
        end
        push!(eachlinedig, dig)
    end

    for (i, line) in enumerate(patterns)
        outputdigits = [findfirst(x -> x == s, eachlinedig[i]) - 1 for s in line[2]]
        part[2] += evalpoly(10, reverse(outputdigits))
    end

    println("Part 1:", part[1])
    println("Part 2:", part[2])
end

day8()


if !isdefined(Main, :CustomStencil)
    include("../module/CustomStencil.jl")
    using Main.CustomStencil
end
include("misc.jl")
include("cpu_stencils.jl")

## Stencil Definition
coefs = Float64.([0.25; 0.125; 0.06])
dense_stencil = @def_stencil_expression(
            @sum(i,-2,2,
                @sum(j,-2,2,
                    @sum(k,-2,2, c[max(abs(i),abs(j),abs(k))]*D[x+i,y+j,z+k]))))
st_def = CreateStencilDefinition(dense_stencil, coefs)
st_inst = NewStencilInstance(st_def)

## Input Data size Definition
radius = 2
nx = 5
ny = 5
nz = 5
dx = 1<<(nx)
dy = 1<<(ny)
dz = 1<<(nz)

t_steps = 3
## GPU Implementation
original_data = CreateData(dx,dy,dz)
data = PadData(radius, original_data)
dout = ApplyStencil(st_inst, data, t_steps)
gpu_out = Array(dout)

## CPU Implementation
cpu_out =  cpu_dense_stencil(original_data, radius, coefs, t_steps)

## Check Result
mean_er = sum(abs.(cpu_out .- gpu_out))./(reduce(*, size(gpu_out)))
@assert(sum(abs.(cpu_out .- gpu_out))./(reduce(*, size(gpu_out))) < 0.001)
## Visual Check
# using Plots
# shift = -10
# Int(round(dz/2))+shift
# heatmap(cout[:,:,Int(round(dz/2))+shift])
# heatmap(o[:,:,Int(round(dz/2))+shift])


using ExpFamily
using Base.Test

srand(123)

Cd = [0.5, 1.0]
Pd = [2.0, 1.0]
m  = [1.0,-0.5]

dgNP = DiagGaussianNatParam(mean=m, cov=Cd)
dgMP = DiagGaussianMeanParam(mean=m, cov=Cd)

@test isapprox(dgNP.theta1, Pd.*m)
@test isapprox(dgNP.theta2, -Pd)
@test isapprox(dgMP.mu1, m)
@test isapprox(dgMP.mu2, (m.^2+Cd)/2)

gNP = full(dgNP)
gMP = full(dgMP)

@test isapprox(gNP.theta2, diagm(-Pd))
@test isapprox(gMP.mu2, diagm(m.^2+Cd)/2)

@test isvalid(dgNP)
@test isvalid(dgMP)

mpFromNP  = meanparam(dgNP)
npFromMP  = natparam(dgMP)

@test isapprox(mpFromNP.mu1, dgMP.mu1)
@test isapprox(mpFromNP.mu2, dgMP.mu2)
@test isapprox(npFromMP.theta1, dgNP.theta1)
@test isapprox(npFromMP.theta2, dgNP.theta2)

@test isapprox(mean(dgNP), m)
@test isapprox(mean(dgMP), m)
@test isapprox(cov(dgNP), Cd)
@test isapprox(cov(dgMP), Cd)
@test isapprox(var(dgMP), Cd)
@test isapprox(var(dgNP), Cd)
@test isapprox(std(dgMP), sqrt.(Cd))
@test isapprox(std(dgNP), sqrt.(Cd))

@test_throws DomainError DiagGaussianNatParam(mean=m,cov=-Cd,check=true)
@test_throws DomainError DiagGaussianMeanParam(mean=m,cov=-Cd,check=true)

dim = 4
dgNPones = ones(DiagGaussianNatParam, dim)
dgMPones = ones(DiagGaussianMeanParam, dim)

@test isapprox(mean(dgNPones), zeros(dim))
@test isapprox(cov(dgNPones),  ones(dim))
@test isapprox(mean(dgMPones), zeros(dim))
@test isapprox(cov(dgMPones),  ones(dim))

@test length(dgNPones) == dim
@test length(dgMPones) == dim

rNP = rand(dgNPones,5000)
rMP = rand(dgMPones,5000)

@test isapprox(mean(rNP,2), zeros(dim), atol=1e-1)
@test isapprox(mean(rMP,2), zeros(dim), atol=1e-1)
@test isapprox(cov(rNP,2),  eye(dim),  atol=1e-1)
@test isapprox(cov(rMP,2),  eye(dim),  atol=1e-1)

dgNP2 = dgNP+dgNP
dgMP2 = dgMP+dgMP

@test isapprox(dgNP2.theta1, 2dgNP.theta1)
@test isapprox(dgNP2.theta2, 2dgNP.theta2)
@test isapprox(dgMP2.mu1, 2dgMP.mu1)
@test isapprox(dgMP2.mu2, 2dgMP.mu2)

m1  = randn(dim)
m2  = randn(dim)
C1  = randn(dim,dim)
C1 *= C1'
C2  = randn(dim,dim)
C2 *= C2'

dgNPa = DiagGaussianNatParam(mean=m1, cov=diag(C1))
dgNPb = DiagGaussianNatParam(mean=m2, cov=diag(C2))
dgMPa = DiagGaussianMeanParam(mean=m1, cov=diag(C1))
dgMPb = DiagGaussianMeanParam(mean=m2, cov=diag(C2))

dgNPtest = dgNPa + dgNPb - natparam(dgMPb)
dgMPtest = dgMPa + dgMPb - meanparam(dgNPb)

@test isapprox(dgNPtest.theta1, dgNPa.theta1)
@test isapprox(dgNPtest.theta2, dgNPa.theta2)
@test isapprox(dgMPtest.mu1, dgMPa.mu1)
@test isapprox(dgMPtest.mu2, dgMPa.mu2)

@test isapprox(dgMPtest, dgMPa)

@test isapprox(DiagGaussianNatParam(vec(dgNPa),dim),dgNPa)
@test isapprox(DiagGaussianMeanParam(vec(dgMPa),dim),dgMPa)

@test isapprox(2dgNPa + dgNPa*2 - 3dgNPa, dgNPa)
@test isapprox(2dgMPa + dgMPa*2 - 3dgMPa, dgMPa)
@test isapprox(dgNPa/2 + dgNPa/2, dgNPa)
@test isapprox(dgMPa/2 + dgMPa/2, dgMPa)

nP = -[1.5;1e7]
m  = [1.0;-0.5]
lm = 1e-6

dgNPproj = project(DiagGaussianNatParam(-nP.*m, nP), minvar=lm)
dgMPproj = project(DiagGaussianMeanParam(mean=m, cov=-1.0./nP), minvar=lm)

nPthresh = -[1.5; 1./lm]

@test isapprox(dgNPproj.theta1, -nPthresh.*m)
@test isapprox(dgNPproj.theta2, -[1.5; 1./lm])
@test isapprox(dgMPproj.mu1, m)
@test isapprox(cov(dgMPproj), [1./1.5; lm])

x = randn(dim)

Ca = cov(dgNPa)
ma = mean(dgNPa)
l  = exp( -dot(x-ma, (x-ma)./Ca) / 2 )/sqrt( (2pi)^dim * prod(Ca) )
ll = log(l)

gNPa = GaussianNatParam(mean=ma, cov=diagm(Ca))

@test isapprox( loglik(dgNPa, x), ll )
@test isapprox( loglik(dgMPa, x), ll )

@test isapprox( gradloglik(dgNPa, x),  gradloglik(gNPa, x) )
@test isapprox( gradloglik(dgMPa, x), gradloglik(dgNPa, x) )


function show(io::IO, mC::MapUnitGrp)
  println(io, "UnitGroup map of $(codomain(mC))")
end

function unit_group_disc_exp(x::GrpAbFinGenElem, U::UnitGrpCtx)
  K = nf(order(U))
  y = FacElem([K(U.torsion_units_gen)], [x.coeff[1,1]])
  for i=1:length(U.units)
    mul!(y, y, U.units[i]^x.coeff[1,i+1])
  end
  return y
end

function unit_group_disc_log(x::FacElem{nf_elem, AnticNumberField} , U::UnitGrpCtx, G::GrpAbFinGen) #snf

  if length(U.units) == 0
    r = [-1]
  else
    r = _add_dependent_unit!(U, x, Val{true})
  end
  @assert r[end] == -1

  y = deepcopy(x)
  for i=1:length(r)-1
    y *= U.units[i]^-r[i]
  end

  K = nf(order(U))

  p = next_prime(2^30)
  while (p-1) % U.torsion_units_order != 0
    p = next_prime(p)
  end
  P = prime_decomposition(order(U), p)[1][1]
  F, mF = ResidueField(order(U), P)
  mK = extend(mF, K)

  yp = F(1)
  for (k,v) = y.fac
    yp *= mK(k)^v
  end

  zp = mF(U.torsion_units_gen)
  res = fmpz[]
  for i=0:U.torsion_units_order-1
    if zp^i == yp
      push!(res, i)
    end
  end
  @assert length(res) == 1

  for i = 1:length(r)-1
    push!(res, r[i])
  end
  return G(res)
end

function unit_group_fac_elem(c::ClassGrpCtx; redo::Bool = false)
  u = unit_group_ctx(c, redo = redo)
  if isdefined(u, :unit_map)
    mU = u.unit_map
    U = domain(mU)
    return U, mU
  end
  return unit_group_fac_elem(u)
end

function unit_group_fac_elem(u::UnitGrpCtx)
  O = order(u)

  zo = u.torsion_units_order
  if zo == -1
    u.torsion_units_gen, u.torsion_units_order = torsion_units_gen_order(O)
    zo = u.torsion_units_order
  end
  r = unit_group_rank(O)
  d = fmpz[zo]
  for i=1:r
    push!(d, fmpz(0))
  end
  U = abelian_group(d)

  r = MapUnitGrp{typeof(FacElemMon(nf(O)))}()

  r.header = MapHeader(U, FacElemMon(nf(O)),
    x->unit_group_disc_exp(x, u),
    x->unit_group_disc_log(x, u, U))

  u.unit_map = r

  return U, r
end


@testset "Poisson 2D" begin

    eta1_min = 0.0
    eta1_max = 2π
    eta2_min = 0.0
    eta2_max = 2π

    nc_eta1 = 128
    nc_eta2 = 128

    ex = zeros(Float64, nc_eta1, nc_eta2)
    ey = zeros(Float64, nc_eta1, nc_eta2)

    rho = zeros(Float64, nc_eta1, nc_eta2)
    phi = zeros(Float64, nc_eta1, nc_eta2)

    ex_exact = zeros(nc_eta1, nc_eta2)
    ey_exact = zeros(nc_eta1, nc_eta2)
    phi_exact = zeros(nc_eta1, nc_eta2)

    grid = TwoDGrid(eta1_min, eta1_max, nc_eta1, eta2_min, eta2_max, nc_eta2)

    poisson = TwoDPoissonPeriodic(grid)

    mode = 2
    for i = 1:nc_eta1, j = 1:nc_eta2
        x1 = eta1_min + (i - 1) * (eta1_max - eta1_min) / nc_eta1
        x2 = eta2_min + (j - 1) * (eta2_max - eta2_min) / nc_eta2
        phi_exact[i, j] = -mode * sin(mode * x1) * cos(mode * x2)
        ex_exact[i, j] = mode^2 * cos(mode * x1) * cos(mode * x2)
        ey_exact[i, j] = -mode^2 * sin(mode * x1) * sin(mode * x2)
        rho[i, j] = -2 * mode^3 * sin(mode * x1) * cos(mode * x2)
    end

    rhs = copy(rho)
    solve!(phi, poisson, rhs)
    @test phi_exact ≈ phi

    rhs = copy(rho)
    solve!(ex, ey, poisson, rhs)
    @test ex_exact ≈ ex
    @test ey_exact ≈ ey

    x1_min = 0
    x1_max = 1

    x2_min = 0
    x2_max = 1

    nc_x1 = 32
    nc_x2 = 64

    phi = zeros(nc_x1, nc_x2)
    e1 = zeros(nc_x1, nc_x2)
    e2 = zeros(nc_x1, nc_x2)
    rho = ones(nc_x1, nc_x2)

    grid = TwoDGrid(x1_min, x1_max, nc_x1, x2_min, x2_max, nc_x2)

    poisson = TwoDPoissonPeriodic(grid)

    solve!(phi, poisson, rho)

    solve!(e1, e2, poisson, rho)

    nx, ny = 32, 32
    xmin, xmax = 0.0, 2π
    ymin, ymax = 0.0, 2π
    degree_smoother = 3
    n_particles = 1000

    particles = ParticleGroup{2,2}(n_particles, charge = 1.0, mass = 1.0, n_weights = 1)

    grid = TwoDGrid(xmin, xmax, nx, ymin, ymax, ny)

    poisson = TwoDPoissonPeriodic(grid)

    # Initialize the kernel smoother
    kernel_smoother = ParticleMeshCoupling2D(particles, grid, degree_smoother, :collocation)

    @test true

    # Initialize the PIC field solver
    pic_poisson = (poisson, kernel_smoother)

    @test true

end


using Test
using WaveProp
using WaveProp.Nystrom
using WaveProp.Geometry
using WaveProp.Integration
using WaveProp.Mesh

@testset "Density test" begin
    pde  = Helmholtz(;dim=3,k=1)
    Ω,M  = WaveProp.IO.gmsh_sphere(dim=2)
    Γ    = boundary(Ω)
    mesh = NystromMesh(view(M,Γ),order=1)
    σ     = γ₀(x->x[1],mesh)
    @test eltype(σ) == Float64
    σ     = γ₀(x->exp(im*x[3]),mesh)
    @test eltype(σ) == ComplexF64
    σ     = γ₁((x,n)->sum(x.*n),mesh)
    @test eltype(σ) == Float64
end


using MultivariateOrthogonalPolynomials, DifferentialEquations, Plots
pyplot() # pyplot supports disks

Z = Zernike(1)
W = Weighted(Z)
xy = axes(W,1)
x,y = first.(xy),last.(xy)
Δ = Z \ Laplacian(xy) * W
S = Z \ W

# initial condition is (1-r^2) * exp(-(x-0.1)^2 - (y-0.2)^2)

K = Block.(Base.OneTo(11))

Δₙ = Δ[K,K]
Sₙ = S[K,K]
Zₙ = Z[:,K]
Wₙ = W[:,K]
c₀ = Zₙ \ @.(exp(-(x-0.1)^2 - (y-0.2)^2))

diskheat(c, (Δₙ, Sₙ), t) = Sₙ \ (Δₙ * c)
u = solve(ODEProblem(diskheat, c₀, (0.,1.), (Δₙ, Sₙ)), Tsit5(), reltol=1e-8, abstol=1e-8)

surface(Wₙ * u(1.0))

diskheat(c, (Δₙ, Sₙ), t) = Δₙ * c
u = solve(ODEProblem(ODEFunction(diskheat; jac=(u, (Δₙ, Sₙ), t) -> Δₙ, mass_matrix=Sₙ), c₀, (0.,1.), (Δₙ, Sₙ)), reltol=1e-8, abstol=1e-8)


#! /usr/bin/julia

# Rosetta Code, Knapsack problem/0-1

using MathProgBase

immutable KPDSupply{S<:String, T<:Integer}
    item::S
    weight::T
    value::T
    quant::T
end
function KPDSupply{S<:String, T<:Integer}(item::S, weight::T, value::T)
    KPDSupply(item, weight, value, one(T))
end

function solve{S<:String, T<:Integer}(gear::Array{KPDSupply{S,T},1},
                                      capacity::T)
    w = map(x->x.weight, gear)
    v = map(x->x.value, gear)
    sol = mixintprog(-v, w', '<', capacity, :Bin, 0, 1)
    sol.status == :Optimal || error("This Problem could not be solved")
    gear[sol.sol .== 1.0]
end

gear = [KPDSupply("map", 9, 150),
        KPDSupply("compass", 13, 35),
        KPDSupply("water", 153, 200),
        KPDSupply("sandwich", 50, 160),
        KPDSupply("glucose", 15, 60),
        KPDSupply("tin", 68, 45),
        KPDSupply("banana", 27, 60),
        KPDSupply("apple", 39, 40),
        KPDSupply("cheese", 23, 30),
        KPDSupply("beer", 52, 10),
        KPDSupply("suntan cream", 11, 70),
        KPDSupply("camera", 32, 30),
        KPDSupply("T-shirt", 24, 15),
        KPDSupply("trousers", 48, 10),
        KPDSupply("umbrella", 73, 40),
        KPDSupply("waterproof trousers", 42, 70),
        KPDSupply("waterproof overclothes", 43, 75),
        KPDSupply("note-case", 22, 80),
        KPDSupply("sunglasses", 7, 20),
        KPDSupply("towel", 18, 12),
        KPDSupply("socks", 4, 50),
        KPDSupply("book", 30, 10)]

pack = solve(gear, 400)

println("The hiker should pack:")
for s in pack
    println("    ", s.item)
end
println()
println("Packed Weight:  ", mapreduce(x->x.weight, +, pack))
println("Packed Value:  ", mapreduce(x->x.value, +, pack))


function main()
    order = [50,48,46,44,42,40,38,36,34,32,30,31,33,35,37,39,41,43,45,47,49]
    l = order[1] + order[21] + sum(sqrt(200 * (order[i] + order[i+1] - 50)) for i in 1:20)
    println(Int(round(l * 1000)))
end

main()


push!(LOAD_PATH, "./src")

using ArgParse
using image_transform
using neural_network
using LinearAlgebra

function get_parsed_args()
    settings = ArgParseSettings(
        description="A command line tool to recognise a handwritten digit from an image."
    )
    @add_arg_table settings begin
        "path"
            help = "path to the image"
            required = true
    end
    return parse_args(settings)
end

function main()
    parsed_args = get_parsed_args()
    path = parsed_args["path"]
    image = image_to_vector(path)

    network = NeuralNetwork([784, 200, 75, 10], "./src/params")
    feed_forward!(network, image)
    total = sum(network.layers[end])
    predictions = sort([zip(network.layers[end] ./ total, 0:9)...], by = pair -> pair[1], rev = true)
    for (probability, label) ∈ predictions
        println("$(label): $(round(probability * 100, digits=2))%")
    end
end

main()


module Polynomial

export Poly, geterrorcorrection

"""
Data structure to encode polynomials to generate the error correction codewords.
"""
struct Poly
    coeff::Array{Int64,1}
end

"""
    makelogtable()

Retrun a list of logarithm values for the Galois Field GF(256).
"""
function makelogtable()::Array{Int64}
    t = ones(Int64, 256)
    v = 1
    for i in 2:256
        v = 2 * v
        if v > 255
            v = xor(v, 285) # According to the specs
        end
        t[i] = v
    end
    return t
end

"""
Logarithm table for GF(256).
"""
const logtable = Dict{Int64, Int64}(zip(0:255, makelogtable()))

"""
Anti-logarithm table for GF(256).
"""
const antilogtable = Dict{Int64, Int64}(zip(makelogtable(), 0:254))

"""
    function mult(a::Int64, b::Int64)

Multiplies two integers in GF(256).
"""
function mult(a::Int64, b::Int64)::Int64
    if a == 0 || b == 0
        return 0
    end
    xa = antilogtable[a]
    xb = antilogtable[b]
    return logtable[(xa + xb) % 255]
end

import Base: length, iterate, ==, <<, +, *

"""
    length(p::Poly)

Return the degree of the polynomial.
"""
length(p::Poly)::Int64 = length(p.coeff)

iterate(p::Poly) = iterate(p.coeff)
iterate(p::Poly, i) = iterate(p.coeff, i)

==(a::Poly, b::Poly)::Bool = a.coeff == b.coeff

"""
    <<(p::Poly, n::Int64)

Increase the degree of `p` by `n`.
"""
<<(p::Poly, n::Int64)::Poly = Poly(vcat(zeros(n), p.coeff))

+(p::Poly) = p

function +(a::Poly, b::Poly)::Poly
    l = max(length(a), length(b))
    return Poly([xor(get(a.coeff, i, 0), get(b.coeff, i, 0)) for i in 1:l])
end

*(a::Int64, p::Poly)::Poly = Poly(map(x->mult(a, x), p.coeff))

function *(a::Poly, b::Poly)::Poly
    return sum([ c * (a << (p - 1)) for (p, c) in enumerate(b.coeff)])
end

"""
    generator(n::Int64)

Create the Generator Polynomial of degree `n`.
"""
function generator(n::Int64)::Poly
    prod([Poly([logtable[i - 1], 1]) for i in 1:n])
end

"""
    lead(p::Poly)

Return the leading coefficient of `p`.
"""
lead(p::Poly)::Int64 = last(p.coeff)

"""
    init!(p::Poly)

Delete the leading coefficient of `p`.
"""
init!(p::Poly)::Poly = Poly(deleteat!(p.coeff, length(p)))

"""
    tail!(p::Poly)

Decrease the degree of `p` by one.
"""
tail!(p::Poly)::Poly = Poly(deleteat!(p.coeff, 1))

"""
    geterrorcorrection(a::Poly, n::Int64)

Return a polynomial containing the `n` error correction codewords of `a`.
"""
function geterrorcorrection(a::Poly, n::Int64)::Poly
    la = length(a)
    a = a << n
    g = generator(n) << la

    for _ in 1:la
        tail!(g)
        a = init!(lead(a) * g + a)
    end
    return a
end

end # module


using ApproxFun, Test

@testset "Complex" begin
    @testset "Differentiation" begin
        f=Fun(x->exp(im*x))
        @test norm(f'-im*f) < ncoefficients(f)*100*eps()
        @test norm(integrate(f)+im*f-f.coefficients[1]*im) < 100eps()
        @test norm(real(f)-Fun(cos)) < 10eps()
        @test norm(real(f-Fun(cos))) < 10eps()
    end

    @testset "Other intervals" begin
        f=Fun(x->exp(im.*x),1..2)
        @test norm(f'-im*f) < 1000eps()
        @test norm(integrate(f)+im*f-f.coefficients[1]*im) < 100eps()
        @test norm(real(f)-Fun(cos,domain(f))) < eps()
        @test norm(real(f-Fun(cos,domain(f)))) < eps()
    end

    @testset "Complex segments" begin
        f=Fun(x->exp(im.*x),Segment(1im,2+.5im))
        @test sum(f) ≈ (0.5515167681675808 + 0.6202852564797062im)
        @test f(1im) ≈ exp(im.*im)
        @test f(1+.75im) ≈ exp(im.*(1+.75im))
        @test norm(f'-im*f) < 1000eps()
        @test norm(integrate(f)+im*f-f.coefficients[1]*im) < 100eps()
    end
end


@testset "Test CG poisson example" begin

using HDiscontinuousGalerkin
using Tensors
using SparseArrays

mesh = rectangle_mesh(TriangleCell, (10,10), Vec{2}((0.0,0.0)), Vec{2}((1.0,1.0)))

# ### Initiate function Spaces
dim = 2
Wh = ScalarFunctionSpace(mesh, ContinuousLagrange{dim,RefTetrahedron,1}(); update_face_data = false)

# Declare variables
u_h = TrialFunction(Wh)

# ### Degrees of freedom
dh = DofHandler([u_h],mesh)
K = create_sparsity_pattern(dh);

# ### Boundary conditions
dbc = Dirichlet(u_h, dh, "boundary", x -> 0)

# ### RHS function
f(x::Vec{dim}) = 2*π^2*sin(π*x[1])*sin(π*x[2])

# ### Assembling the linear system
# Now we have all the pieces needed to assemble the linear system, $K u = f$.
function doassemble(Wh, K::SparseMatrixCSC, dh::DofHandler)
    # We allocate the element stiffness matrix and element force vector
    # just once before looping over all the cells instead of allocating
    # them every time in the loop.
    n_basefuncs = getnbasefunctions(Wh)
    Ke = zeros(n_basefuncs, n_basefuncs)
    fe = zeros(n_basefuncs)
    b = zeros(ndofs(dh))
    cell_dofs = Vector{Int}(undef, ndofs_per_cell(dh))

    # Next we define the global force vector `f` and
    # create an assembler. The assembler
    # is just a thin wrapper around `f` and `K` and some extra storage
    # to make the assembling faster.
    assembler = start_assemble(K, b)

    # It is now time to loop over all the cells in our grid.
    @inbounds for (cellcount, cell) in enumerate(CellIterator(mesh))
        # Always remember to reset the element stiffness matrix and
        # force vector since we reuse them for all elements.
        fill!(Ke, 0)
        fill!(fe, 0)
        reinit!(Wh, cell)

        # It is now time to loop over all the quadrature points in the cell and
        # assemble the contribution to `Ke` and `fe`. The integration weight
        # can be queried from `cellvalues` by `getdetJdV`.
        for q_point in 1:getnquadpoints(Wh)
            dΩ = getdetJdV(Wh, q_point)
            fh = function_value(f, Wh, cell, q_point)
            # For each quadrature point we loop over all the (local) shape functions.
            # We need the value and gradient of the testfunction `v` and also the gradient
            # of the trial function `u`. We get all of these from `cellvalues`.
            for i in 1:n_basefuncs
                v  = shape_value(Wh, q_point, i)
                ∇v = shape_gradient(Wh, q_point, i)
                fe[i] += fh*v * dΩ
                for j in 1:n_basefuncs
                    ∇u = shape_gradient(Wh, q_point, j)
                    Ke[i, j] += (∇v ⋅ ∇u) * dΩ
                end
            end
        end
        # The last step in the element loop is to assemble `Ke` and `fe`
        # into the global `K` and `f` with `assemble!`.
        celldofs!(cell_dofs, dh, cell)
        assemble!(assembler, cell_dofs, fe, Ke)
    end
    return K, b
end
#md nothing # hide

# ### Solution of the system
# The last step is to solve the system. First we call `doassemble`
# to obtain the global stiffness matrix `K` and force vector `f`.
K, b = doassemble(Wh, K, dh);

apply!(K,b,dbc)
u = K \ b;

reconstruct!(u_h, u, dh)

# ### Compute errors
u_ex(x::Vec{dim}) = sin(π*x[1])*sin(π*x[2])
Etu_h = errornorm(u_h, u_ex)
@test Etu_h <= 0.0002

end


include("euler/euler.jl")
using .Grid: max_path_sum

compute(triangle::String)::Int = max_path_sum([parse.(Int, split(line, " ")) for line ∈ split(triangle, "\n")])


function Base.real{DD}(f::Fun{Laurent{DD}})
    n=ncoefficients(f)
    cfs=f.coefficients

    ret=Array(Float64,iseven(n)?n+1:n)
    ret[1]=real(cfs[1])

    for k=2:2:n
        # exp(1im(k-1)/2*x)=cos((k-1)/2 x) +i sin((k-1)/2 x)
        ret[k]=imag(cfs[k])
        ret[k+1]=real(cfs[k])
    end
    for k=3:2:n
        # exp(1im(k-1)/2*x)=cos((k-1)/2 x) +i sin((k-1)/2 x)
        ret[k]+=real(cfs[k])
        ret[k-1]-=imag(cfs[k])
    end

    Fun(ret,Fourier(domain(f)))
end

function Base.imag{DD}(f::Fun{Laurent{DD}})
    n=ncoefficients(f)
    cfs=f.coefficients

    ret=Array(Float64,iseven(n)?n+1:n)
    ret[1]=imag(cfs[1])

    for k=2:2:n
        # exp(1im(k-1)/2*x)=cos((k-1)/2 x) +i sin((k-1)/2 x)
        ret[k]=-real(cfs[k])
        ret[k+1]=imag(cfs[k])
    end
    for k=3:2:n
        # exp(1im(k-1)/2*x)=cos((k-1)/2 x) +i sin((k-1)/2 x)
        ret[k]+=imag(cfs[k])
        ret[k-1]+=real(cfs[k])
    end

    Fun(ret,Fourier(domain(f)))
end


const XY = collect(range(-3, 3, length=100))

function showplot()
    GLMakie.activate!()
    fig = Figure()
    buildplot!(fig[2, 1], fig[3, 1], Makie.Slider(fig[1, 1], range=XY, startvalue=0))
    fig
end

function buildplot!(pa1, pa2, slider)
    ρ = 0.7

    dist = MvNormal([0, 0], [1 ρ; ρ 1])

    grid = zeros(Float64, (100, 100))

    for (x, i) in enumerate(XY)
        for (y, j) in enumerate(XY)
            grid[x, y] = pdf(dist, [i, j])
        end
    end

    # compute shading and integral of portion above cutoff
    integrated = lift(slider.value) do ŷ
        integrated = zeros(Float64, 100)

        for (x, i) in enumerate(XY)
            for (y, j) in enumerate(XY)
                # probit likelihood function: P(ŷ + ϵ > 0), ϵ ~ N(0, 1)
                #                               = P(ϵ > -ŷ)
                if i > -ŷ
                    integrated[y] += grid[x, y]
                end
            end
        end

        integrated = integrated ./ sum(integrated)

        integrated
    end

    color = lift(slider.value) do ŷ
        color = Array{Any}(undef, 100, 100)

        for (x, i) in enumerate(XY)
            for (y, j) in enumerate(XY)
                color[x, y] = i > -ŷ ? colorant"#4B9CD3" : colorant"#AAAAAA"
            end
        end

        color
    end

    vertices = lift(slider.value) do ŷ
        top = maximum(grid)
        vertices = [
            3 -3 0;  # 1
            3 3 0;  # 2
            -ŷ 3 0; # 3
            -ŷ -3 0; # 4
            3 -3 top;  # 5
            3 3 top;  # 6
            -ŷ 3 top; # 7
            -ŷ -3 top; # 8
        ]
    end

    faces = [
        1 2 3; 4 1 3; # bottom
        5 6 7; 8 7 5; # top
        1 2 5; 5 6 2;
        2 3 6; 7 6 3;
        3 4 7; 8 7 4;
        4 1 5; 8 5 4;
    ]

    ax1 = Axis3(pa1, xlabel="Selection error term", ylabel="Outcome error term")
    surface!(ax1, XY, XY, grid)
    mesh!(ax1, vertices, faces, color=colorant"rgba(75, 156, 211, 0.5)")
    ax2 = Axis(pa2, ylims=(0, 15), yticks=[0])
    lines!(ax2, XY, integrated)
end

function wglplot(s, x...)
    WGLMakie.activate!()

    App() do session::Session
        fig = Figure(resolution=(1200, 900))
        slider = Makie.Slider(fig[1, 1], range=XY, startvalue=0)
        buildplot!(fig[2, 1], fig[3, 1], slider)
        return JSServe.record_states(session, DOM.div(slider, fig))
    end
end

function exportplot()
    JSServe.export_standalone(wglplot, joinpath(Base.source_dir(), "output"))
end


# Example of the piecewise linear regration with Mosek, MosekTools, and JuMP
# See Mosek manual 9.2.3 Convex piecewise linear regression

using JuMP, Mosek, MosekTools
using RDatasets, Plots

# prepare data
data = dataset("datasets", "pressure");
Y = data.Temperature ./1000;    Y = Y[8:end]
X = data.Pressure ./1000;       X = X[8:end]
n = length(X)
k = 3
(M_lb, M_ub) = (-5, 5)

# build and solve the model
m = Model(Mosek.Optimizer)
@variable(m, t)
@variable(m, M_lb <= f[1:n] <= M_ub)
@variable(m, a[1:k])
@variable(m, b[1:k])
@variable(m, z[1:n,1:k], Bin)
@objective(m, Min,  t)
@constraint(m, [t;Y - f] in SecondOrderCone())
@constraint(m, cut_one[i=1:n,j=1:k], (a[j]*X[i]+b[j]) - f[i] <= M_ub * (1 - z[i,j]))
@constraint(m, cut_two[i=1:n,j=1:k], (a[j]*X[i]+b[j]) - f[i] >= M_lb * (1 - z[i,j]))
@constraint(m, sum_bin[i=1:n], sum(z[i,:]) == 1)
optimize!(m)
status=termination_status(model)

@show a_ = JuMP.value.(a)
@show b_ = JuMP.value.(b)

# make a plot
Plots.gr(ylims=(0,1))
Plots.scatter(X,Y,legend=false)
for j in 1:k
  f_z(x) = a_[j]*x + b_[j]
  Plots.plot!(X, f_z)
end
current()


@doc raw"
`struct RK_W1 <: ReproducingKernel_1`

Define a type of reproducing kernel of Sobolev space ``W^2_1 [0,1]``:
```math
V(\eta, \xi) =
  \begin{cases}
       1 + \eta \,  , &  0 \le \eta \le \xi \le 1 \\
       1 + \xi \,  , &  0 \le \xi \le \eta \le 1
    \end{cases}
```
"
struct RK_W1 <: ReproducingKernel_1
   RK_W1() = new()
   RK_W1(p::Int64) = new()
   RK_W1(p::AbstractFloat) = new()
end

@doc raw"
`struct RK_W2 <: ReproducingKernel_2`

Define a type of reproducing kernel of Sobolev space ``W^2_2 [0,1]``:
```math
V(\eta, \xi) =
  \begin{cases}
       1 + (\eta + \eta^2 / 2)\xi - \eta^3 / 6  \,  , &  0 \le \eta \le \xi \le 1  \\
       1 + (\xi + \xi^2 / 2)\eta - \xi^3 / 6 \,  , &  0 \le \xi \le \eta \le 1
    \end{cases}
```
"
struct RK_W2 <: ReproducingKernel_2
   RK_W2() = new()
   RK_W2(p::Int64) = new()
   RK_W2(p::AbstractFloat) = new()
end

@doc raw"
`struct RK_W3 <: ReproducingKernel_3`

Define a type of reproducing kernel of Sobolev space ``W^2_3 [0,1]``:
```math
V(\eta, \xi) =
  \begin{cases}
       \sum_{i=0}^2 \frac{\xi^i}{!i} \left ( \frac{\eta^i}{!i} + (-1)^{i} \frac {\eta^{5 - i}}{(5 - i)!}  \right )  \,  , &  0 \le \eta \le \xi \le 1
        \\
       \sum_{i=0}^2 \frac{\eta^i}{!i} \left ( \frac{\xi^i}{!i} + (-1)^{i} \frac {\xi^{5 - i}}{(5 - i)!}  \right ) \,  , &  0 \le \xi \le \eta \le 1   \end{cases}
```
"
struct RK_W3 <: ReproducingKernel_3
   RK_W3() = new()
   RK_W3(p::Int64) = new()
   RK_W3(p::AbstractFloat) = new()
end

#
@doc raw"
`struct RK_H1{T <: AbstractFloat} <: ReproducingKernel_1`

Define a type of reproducing kernel of Bessel Potential space ``H^1_ε (R)``:
```math
V(\eta , \xi, \varepsilon) = \exp (-\varepsilon |\xi - \eta|) \, .
```
# Arguments
- no argument: means that parameter `ε` will be estimated in course of
               interpolation procedure execution.
- `ε::T`: 'scaling parameter' from the Bessel Potential space definition,
           it must be greater than zero.
"
struct RK_H1{T <: AbstractFloat} <: ReproducingKernel_1
     ε::T
     RK_H1() = new{Float64}(0.0)
     function RK_H1(ε::T) where T <: AbstractFloat
        if ε <= 0
          throw(DomainError(ε, "Parameter ε must be a positive number."))
        end
        new{T}(ε)
     end
     function RK_H1(ε::Integer)
        if ε <= 0
           throw(DomainError(ε, "Parameter ε must be a positive number."))
        end
        new{Float64}(convert(Float64, ε))
     end
end

@doc raw"
`struct RK_H2{T <: AbstractFloat} <: ReproducingKernel_2`

Define a type of reproducing kernel of Bessel Potential space ``H^2_ε (R)``:
```math
V(\eta , \xi, \varepsilon) = \exp (-\varepsilon |\xi - \eta|)
             (1 + \varepsilon |\xi  - \eta|) \, .
```
# Arguments
- no argument: means that parameter `ε` will be estimated in course of
               interpolation procedure execution.
- `ε::T`: 'scaling parameter' from the Bessel Potential space definition,
           it must be greater than zero.
"
struct RK_H2{T <: AbstractFloat} <: ReproducingKernel_2
     ε::T
     RK_H2() = new{Float64}(0.0)
     function RK_H2(ε::T) where T <: AbstractFloat
        if ε <= 0
           throw(DomainError(ε, "Parameter ε must be a positive number."))
        end
        new{T}(ε)
     end
     function RK_H2(ε::Integer)
        if ε <= 0
           throw(DomainError(ε, "Parameter ε must be a positive number."))
        end
        new{Float64}(convert(Float64, ε))
     end
end

@doc raw"
`struct RK_H3{T <: AbstractFloat} <: ReproducingKernel_3`

Define a type of reproducing kernel of Bessel Potential space ``H^3_ε (R)``:
```math
V(\eta , \xi, \varepsilon) = \exp (-\varepsilon |\xi - \eta|)
             (3 + 3\varepsilon |\xi  - \eta| + \varepsilon ^2 |\xi - \eta| ^2 ) \, .
```
# Arguments
- no argument: means that parameter `ε` will be estimated in course of
               interpolation procedure execution.
- `ε::T`: 'scaling parameter' from the Bessel Potential space definition,
           it must be greater than zero.
"
struct RK_H3{T <: AbstractFloat} <: ReproducingKernel_3
     ε::T
     RK_H3() = new{Float64}(0.0)
     function RK_H3(ε::T) where T <: AbstractFloat
        if ε <= 0
           throw(DomainError(ε, "Parameter ε must be a positive number."))
        end
        new{T}(ε)
     end
     function RK_H3(ε::Integer)
        if ε <= 0
           throw(DomainError(ε, "Parameter ε must be a positive number."))
        end
        new{Float64}(convert(Float64, ε))
     end
end

@inline function _rk(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_1}
   value::T = 0.0
   defined::Bool  = false
   if η <= ξ
      if isa(kernel, RK_H3)
         defined = true
         x = kernel.ε * (ξ - η)
         value = (3 + x * (3 + x)) * exp(-x)
      end
      if isa(kernel, RK_H2)
         defined = true
         x = kernel.ε * (ξ - η)
         value = (1 + x) * exp(-x)
      end
      if isa(kernel, RK_H1)
         defined = true
         x = kernel.ε * (ξ - η)
         value = exp(-x)
      end
      if isa(kernel, RK_W3)
         defined = true
         value = 1 + η * (ξ + η * (η^2 * (η - 5 * ξ) + 10 * ξ^2 * (η + 3))/120)
      end
      if isa(kernel, RK_W2)
         defined = true
         value = 1 + (1 + η/2) * η * ξ - η^3/6
      end
      if isa(kernel, RK_W1)
         defined = true
         value = 1 + η
      end
   else
      value = _rk(kernel, ξ, η)
      return value
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂rk_∂ξ(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_2}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = kernel.ε * (ξ - η)
      value = -kernel.ε * x * (1 + abs(x)) * exp(-abs(x))
   end
   if isa(kernel, RK_H2)
      defined = true
      x = kernel.ε * (ξ - η)
      value = -kernel.ε * x * exp(-abs(x))
   end
   if isa(kernel, RK_W3)
       defined = true
       if η <= ξ
          value = η + η * (4 * η * ξ * (η + 3) - η^3)/24
       else
          value = (ξ^4 - 4 * η * (ξ^3 - 6) + 6 * ξ * η^2 * (ξ + 2))/24
       end
   end
   if isa(kernel, RK_W2)
      defined = true
      if η <= ξ
         value = (1 + η/2) * η
      else
         value = ξ * (η - ξ/2) + η
      end
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂²rk_∂η∂ξ(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_2}
   value::T = 0.0
   defined::Bool  = false
   if η <= ξ
      if isa(kernel, RK_H3)
         defined = true
         x = abs(kernel.ε * (ξ - η))
         value = -(kernel.ε^2) * (x * (x - 1) - 1) * exp(-x)
      end
      if isa(kernel, RK_H2)
         defined = true
         x = abs(kernel.ε * (ξ - η))
         value = kernel.ε^2 * (1 - x) * exp(-x)
      end
      if isa(kernel, RK_W3)
         defined = true
         value = -(η^3)/6 + η * ξ * (η + 2)/2 + 1
      end
      if isa(kernel, RK_W2)
         defined = true
         value = 1 + η
      end
   else
      value = _∂²rk_∂η∂ξ(kernel, ξ, η)
      return value
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂²rk_∂ξ²(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_3}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = abs(kernel.ε * (ξ - η))
      value = kernel.ε^2 * (x * (x - 1) - 1) * exp(-x)
   end
   if isa(kernel, RK_W3)
      defined = true
      if η <= ξ
         value = η^2 * (η + 3)/6
      else
         value = (ξ^2 * (ξ - 3 * η) + 3 * η^2 * (ξ + 1))/6
      end
   end

   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂³rk_∂η∂ξ²(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_3}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = kernel.ε * (ξ - η)
      value = kernel.ε^3 * (x * (abs(x) - 3)) * exp(-abs(x))
   end
   if isa(kernel, RK_W3)
      defined = true
      if η <= ξ
          value = η * (1 + η/2)
      else
         value = ξ * (η - ξ/2) + η
      end
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂⁴rk_∂η²∂ξ²(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_3}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = abs(kernel.ε * (ξ - η))
      value = kernel.ε^4 * (x * (x - 5) + 3) * exp(-x)
   end
   if isa(kernel, RK_W3)
      defined = true
      value = η + 1
      if η <= ξ
         value = η + 1
      else
         value = ξ + 1
      end
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂rk_∂η(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_2}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = kernel.ε * (ξ - η)
      value = kernel.ε * x * (1 + abs(x)) * exp(-abs(x))
   end
   if isa(kernel, RK_H2)
      defined = true
      x = kernel.ε * (ξ - η)
      value = kernel.ε * x * exp(-abs(x))
   end
   if isa(kernel, RK_W3)
       defined = true
       if η <= ξ
          value = (η * (η^3 + 6 * ξ^2 * (η + 2)) - 4 * ξ * (η^3 - 6))/24
       else
          value = ξ + ξ * (4 * η * ξ * (ξ + 3) - ξ^3)/24
       end
   end
   if isa(kernel, RK_W2)
      defined = true
      if η <= ξ
         value = η * (ξ - η/2) + ξ
      else
         value = (1 + ξ/2) * ξ
      end
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂²rk_∂η²(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_3}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = abs(kernel.ε * (ξ - η))
      value = kernel.ε^2 * (x * (x - 1) - 1) * exp(-x)
   end
   if isa(kernel, RK_W3)
      defined = true
      if η <= ξ
         value = (η^2 * (η - 3 * ξ) + 3 * ξ^2 * (η + 1))/6
      else
         value = ξ^2 * (ξ + 3)/6
      end
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end

@inline function _∂³rk_∂η²∂ξ(kernel::RK, η::T, ξ::T) where {T <: AbstractFloat, RK <: ReproducingKernel_3}
   value::T = 0.0
   defined::Bool  = false
   if isa(kernel, RK_H3)
      defined = true
      x = kernel.ε * (ξ - η)
      value = -kernel.ε^3 * (x * (abs(x) - 3)) * exp(-abs(x))
   end
   if isa(kernel, RK_W3)
      defined = true
      if η <= ξ
          value = η * (ξ - η/2) + ξ
      else
          value = ξ * (1 + ξ/2)
      end
   end
   if !defined
      Throw(ArgumentError("kernel: Incorrect parameter type."))
   end
   return value
end


using Dorayaki
using Random
using Plots
using BenchmarkTools

Random.seed!(0)

const lr = 0.1
const iters = 100

x = rand(1, 100)
y = 5 .+ 2 .* x .+ rand(1, 100)

function main(x, y)
    W = Var(zeros(1, 1))
    b = Var(zeros(1))

    predict(x) = W * x + b

    for i in 1:iters
        y_pred = predict(x)
        loss = mean_squared_error(y, y_pred)

        cleargrad!(W)
        cleargrad!(b)

        gradient!(loss)
        W.data .-= lr .* W.grad.data
        b.data .-= lr .* b.grad.data
        # println(W.data, b.data, loss.data)
    end
    return W.data[1], b.data[1]
end

ret = @benchmark main(x, y)
#=
f(a) = ret[1] * a + ret[2]
fig = plot(x, y, st=:scatter)
plot!(fig, f) =#


# # [A classification task: the prediction of  plant species from floreal measures (the iris dataset)](@id clustering_tutorial)
# The task is to estimate the species of a plant given some floreal measurements. It use the classical "Iris" dataset.
# Note that in this example we are using clustering approaches, so we try to understand the "structure" of our data, without relying to actually knowing the true labels ("classes" or "factors"). However we have chosen a dataset for which the true labels are actually known, so to compare the accuracy of the algorithms we use, but these labels will not be used during the algorithms training.

#
# Data origin:
# - dataset description: [https://en.wikipedia.org/wiki/Iris_flower_data_set](https://en.wikipedia.org/wiki/Iris_flower_data_set)
# - data source we use here: [https://github.com/JuliaStats/RDatasets.jl](https://github.com/JuliaStats/RDatasets.jl)

# ## Library and data loading
# We load the Beta Machine Learning Toolkit as well as some other packages that we use in this tutorial
using BetaML
using Random, Statistics, Logging, BenchmarkTools, RDatasets, Plots, DataFrames

# We are also going to compare our results with two other leading packages in Julia for clustering analysis, [`Clustering.jl`](https://github.com/JuliaStats/Clustering.jl) that provides (inter alia) kmeans and kmedoids algorithms and [`GaussianMixtures.jl`](https://github.com/davidavdav/GaussianMixtures.jl) that provides, as the name says, Gaussian Mixture Models. So we import them (we "import" them, rather than "use", not to bound their full names into namespace as some would collide with BetaML).
import Clustering, GaussianMixtures
using  Test     #src

# We do a few tweeks for the Clustering and GaussianMixtures packages. Note that in BetaML we can also control both the random seed and the verbosity in the algorithm call, not only globally
Random.seed!(123)
#logger  = Logging.SimpleLogger(stdout, Logging.Error); global_logger(logger); ## For suppressing GaussianMixtures output

# Differently from the [regression tutorial](@ref regression_tutorial), we load the data here from [`RDatasets`](https://github.com/JuliaStats/RDatasets.jl](https://github.com/JuliaStats/RDatasets.jl), a package providing standard datasets.
iris = dataset("datasets", "iris")
describe(iris)

# The iris dataset  provides floreal measures in columns 1 to 4 and the assigned species name in column 5. There are no missing values

# ## Data preparation
# The first step is to prepare the data for the analysis. We collect the first 4 columns as our _feature_ `x` matrix and the last one as our `y` label vector.
# As we are using clustering algorithms, we are not actually using the labels to train the algorithms, we'll behave like we do not know them, we'll just let the algorithm "learn" fro mthe structure of the data itself. We'll however use it to judge the accuracy that they did reach.
x       = Matrix{Float64}(iris[:,1:4]);
yLabels = unique(iris[:,5]);
# As the labels are expressed as strings, the first thing we do is encode them as integers for our analysis using the function [`integerEncoder`](@ref).
y       = integerEncoder(iris[:,5],factors=yLabels);

# The dataset from RDatasets is ordered by species, so we need to shuffle it to avoid biases.
# Shuffling happens by default in crossValidation, but we are keeping here a copy of the shuffled version for later.
# Note that the version of [`shuffle`](@ref) that is included in BetaML accepts several n-dimensional arrays and shuffle them (by default on rows, by we can specify the dimension) keeping the association  between the various arrays in the shuffled output.
(xs,ys) = shuffle([x,y]);

# ## Main analysis

# We will try 3 BetaML models ([`kmeans`](@ref), [`kmedoids`](@ref) and [`gmm`](@ref)) and we compare them with `kmeans` from Clusterings.jl and `GMM` from GaussianMixtures.jl
# `Kmeans` and `kmedoids` works by first initialising the centers of the k-clusters (the "representative" (step a ) . For `kmeans` they must be selected within one of the data, for kmeans they are the geometrical center) n a nutshell. Then ( b ) iterate for each point to assign the point to the cluster of the closest representative (according with a user defined distance metric, default to Euclidean), and ( c ) move each representative at the center of its newly acquired cluster (where "center" depends again from the metric). Steps ( b ) and ( c ) are reiterated until the algorithm converge, i.e. the tentative k representative points (and their relative clusters) don't move any more. The result (output of the algorithm) is that each point is assigned to one of the clusters (classes).
# The `gmm` algorithm is similar in that it employs an iterative approach (the Expectation_Minimisation algorithm, "em") but here we make the hipothesis that the data points are the observed outcomes of some _mixture_ probabilistic models where we have first a k-categorical variables whose outcomes are the (unobservble) parameters of a probabilistic distribution from which the data is finally drawn. Because the parameters of each of the k-possible distributions is unobservable this is also called a model with latent variables.
# Most `gmm` models use the Gaussain distribution as the family of the mixture components, so we can tought the `gmm` acronym to indicate _Gaussian Mixture Model_. In BetaML we do implemented only Gaussain components, but any distribution could be used by just subclassing `AbstractMixture` and implementing a couple of methids (you are invited to contribute or just ask for a distribution family you are interested), so I prefer to think "gmm" as an acronym for _Generative Mixture Model_.
# The algorithm try to find the mixture that maximises the likelihood that the data has been generated indeed from such mixture, where the "E" step refers to computing the probability that each point belongs to each of the k-composants (somehow similar to the step _b_ in the kmeans/kmedoids algorithm), and the "M" step estimates, giving the association probabilities in step "M", the parameters of the mixture and of the individual components (similar to step _c_).
# The result here is that each point has a categorical distribution (PMF) representing the probabilities that it belongs to any of the k-components (our classes or clusters). This is interesting, as `gmm` can be used for many other things that clustering. It forms the backbone of the [`predictMissing`](@ref) function to impute missing values (on some or all dimensions) based to how close the record seems to its pears. For the same reasons, `predictMissing` can also be used to predict user's behaviours (or users' appreciation) according to the behaviour/ranking made by pears ("collaborative filtering").
# While the result of `gmm` is a vector of PMFs (one for each record), error measures and reports with the true values (if known) can be directly applied, as in BetaML they internally call `mode()` to retrieve the class with the highest probability for each record.

# As we are here, we also try different versions of the BetaML models, even if the default "versions" should be fine. For `kmeans` and `kmedoids` we will try different initialisation strategies ("gird", the default one, "random" and "shuffle"), while for the `gmm` model we'll choose different distributions of the Gaussain family (`SphericalGaussian` - where the variance is a scalar, `DiagonalGaussian` - with a vector variance, and `FullGaussian`, where the covariance is a matrix).

# As the result would depend on stochasticity both in the data selected and in the random initialisation, we use a cross-validation approach to run our models several times (with different data) and then we average their results.
# Cross-Validation in BetaML is very flexible and it is done using the [`crossValidation`](@ref) function.
# crossValidation works by calling the function `f`, defined by the user, passing to it the tuple `trainData`, `valData` and `rng` and collecting the result of the function f. The specific method for which `trainData`, and `valData` are selected at each iteration depends on the specific `sampler`.
# We start by selectign a k-fold sampler that split our data in 5 different parts, it uses 4 for training and 1 part (not used here) for validation. We run the simulations twice and, to be sure to have replicable results, we fix the random seed (at the whole crossValidaiton level, not on each iteration).
sampler = KFold(nSplits=5,nRepeats=3,shuffle=true, rng=copy(FIXEDRNG))

# We can now run the cross-validation with our models. Note that instead of defining the function `f` and then calling `crossValidation[f(trainData,testData,rng),[x,y],...)` we use the Julia `do` block syntax and we write directly the content of the `f` function in the `do` block.
# Also, by default crossValidation already returns the mean and the standard deviation of the output of the user-provided `f` function (or the `do` block). However this requires that the `f` function return a single scalar. Here we are returning a vector of the accuracies of the different models (so we can run the cross-validation only once), and hence we indicate with `returnStatistics=false` to crossValidation not to attempt to generate statistics but rather report the whole output.
# We'll compute the statistics ex-post.

# Inside the `do` block we do 4 things:
# - we recover from `trainData` (a tuple, as we passed a tuple to `crossValidation` too) the `xtrain` features and `ytrain` labels;
# - we run the various clustering algorithms
# - we use the real labels to compute the model accuracy. Note that the clustering algorithm know nothing about the specific label name or even their order. This is why [`accuracy`](@ref) has the parameter `ignoreLabels` to compute the accuracy oven any possible permutation of the classes found.
# - we return the various models' accuracies
cOut = crossValidation([x,y],sampler,returnStatistics=false) do trainData,testData,rng
          ## For unsupervised learning we use only the train data.
          ## Also, we use the associated labels only to measure the performances
         (xtrain,ytrain)  = trainData;
         ## We run the clustering algorithm...
         clusteringOut     = kmeans(xtrain,3,rng=rng) ## init is grid by default
         ## ... and we compute the accuracy using the real labels
         kMeansAccuracy    = accuracy(clusteringOut[1],ytrain,ignoreLabels=true)
         clusteringOut     = kmeans(xtrain,3,rng=rng,initStrategy="random")
         kMeansRAccuracy   = accuracy(clusteringOut[1],ytrain,ignoreLabels=true)
         clusteringOut     = kmeans(xtrain,3,rng=rng,initStrategy="shuffle")
         kMeansSAccuracy   = accuracy(clusteringOut[1],ytrain,ignoreLabels=true)
         clusteringOut     = kmedoids(xtrain,3,rng=rng)   ## init is grid by default
         kMedoidsAccuracy  = accuracy(clusteringOut[1],ytrain,ignoreLabels=true)
         clusteringOut     = kmedoids(xtrain,3,rng=rng,initStrategy="random")
         kMedoidsRAccuracy = accuracy(clusteringOut[1],ytrain,ignoreLabels=true)
         clusteringOut     = kmedoids(xtrain,3,rng=rng,initStrategy="shuffle")
         kMedoidsSAccuracy = accuracy(clusteringOut[1],ytrain,ignoreLabels=true)
         clusteringOut     = gmm(xtrain,3,mixtures=[SphericalGaussian() for i in 1:3], verbosity=NONE, rng=rng)
         gmmSpherAccuracy  = accuracy(clusteringOut.pₙₖ,ytrain,ignoreLabels=true, rng=rng)
         clusteringOut     = gmm(xtrain,3,mixtures=[DiagonalGaussian() for i in 1:3], verbosity=NONE, rng=rng)
         gmmDiagAccuracy   = accuracy(clusteringOut.pₙₖ,ytrain,ignoreLabels=true, rng=rng)
         clusteringOut     = gmm(xtrain,3,mixtures=[FullGaussian() for i in 1:3], verbosity=NONE, rng=rng)
         gmmFullAccuracy   = accuracy(clusteringOut.pₙₖ,ytrain,ignoreLabels=true, rng=rng)
         ## For comparision with Clustering.jl
         clusteringOut     = Clustering.kmeans(xtrain', 3)
         kMeans2Accuracy   = accuracy(clusteringOut.assignments,ytrain,ignoreLabels=true)
         ## For comparision with GaussianMistures.jl - sometimes GaussianMistures.jl em! fails with a PosDefException
         dGMM              = GaussianMixtures.GMM(3, xtrain; method=:kmeans, kind=:diag)
         GaussianMixtures.em!(dGMM, xtrain)
         gmmDiag2Accuracy  = accuracy(GaussianMixtures.gmmposterior(dGMM, xtrain)[1],ytrain,ignoreLabels=true)
         fGMM              = GaussianMixtures.GMM(3, xtrain; method=:kmeans, kind=:full)
         GaussianMixtures.em!(fGMM, xtrain)
         gmmFull2Accuracy  = accuracy(GaussianMixtures.gmmposterior(fGMM, xtrain)[1],ytrain,ignoreLabels=true)
         ## Returning the accuracies
         return kMeansAccuracy,kMeansRAccuracy,kMeansSAccuracy,kMedoidsAccuracy,kMedoidsRAccuracy,kMedoidsSAccuracy,gmmSpherAccuracy,gmmDiagAccuracy,gmmFullAccuracy,kMeans2Accuracy,gmmDiag2Accuracy,gmmFull2Accuracy
 end

## We transform the output in matrix for easier analysis
accuracies = fill(0.0,(length(cOut),length(cOut[1])))
[accuracies[r,c] = cOut[r][c] for r in 1:length(cOut),c in 1:length(cOut[1])]
μs = mean(accuracies,dims=1)
σs = std(accuracies,dims=1)

@test all(μs .> 0.7) #src

@test μs[1] > 0.89 &&  μs[4] > 0.89 &&  μs[9] > 0.96 #src
modelLabels=["kMeansG","kMeansR","kMeansS","kMedoidsG","kMedoidsR","kMedoidsS","gmmSpher","gmmDiag","gmmFull","kMeans (Clustering.jl)","gmmDiag (GaussianMixtures.jl)","gmmFull (GaussianMixtures.jl)"]
report = DataFrame(mName = modelLabels, avgAccuracy = dropdims(round.(μs',digits=3),dims=2), stdAccuracy = dropdims(round.(σs',digits=3),dims=2))

#src plot(modelLabels,μs',seriestype=:scatter)
#src yerror=collect(zip(rand(12), rand(12))

# ### BetaML model accuracies

# From the output We see that the gmm models perform for this dataset generally better than kmeans or kmedoids algorithms, also with very low variances.
# In detail, it is the (default) `grid` initialisation that leads to the better results for `kmeans` and `kmedoids`, while for the `gmm` models it is the `FullGaussian` to perform better.

# ### Comparisions with `Clustering.jl` and `GaussianMixtures.jl`
# For this specific case, both `Clustering.jl` and `GaussianMixtures.jl` report substantially worst accuracies, and with very high variances. But we maintain the ranking that Full Gaussian gmm > Diagonal Gaussian > Kmeans accuracy.
# I suspect the reason that BetaML gmm works so weel is in relation to the usage of kmeans algorithm with itself the grid initialisation.
# The grid initialisation "guarantee" indeed that the initial means of the mixture components are well spread across the multidimensional space defined by the data, and it helps avoiding the EM algoritm to converge to a bad local optimus.

# ## Working without the labels

# Up to now we used the real labels to compare the model accuracies. But in real clustering examples we don't have the true classes, or we wouln't need to do clustering in the first instance, so we don't know the number of classes to use.
# There are several methods to judge clusters algorithms goodness, perhaps the simplest one, at least for the expectation-maximisation algorithm employed in `gmm` to fit the data to the unknown mixture, is to use a information criteria that trade the goodness of the lickelyhood with the parameters used to do the fit.
# BetaML provide by default in the gmm clustering outputs both the _Bayesian information criterion_  ([`BIC`](@ref bic)) and the _Akaike information criterion_  ([`AIC`](@ref aic)), where for both a lower value is better.

# We can then run the model with different number of classes and see which one leads to the lower BIC or AIC.
# We run hence `crossValidation` again with the `FullGaussian` gmm model
# Note that we use the BIC/AIC criteria here for establishing the "best" number of classes but we could have used it also to select the kind of Gaussain distribution to use. This is one example of hyper-parameter tuning that we developed more in detail (but without using cross-validation) in the [regression tutorial](@ref regression_tutorial).

# Let's try up to 8 possible classes:

K = 8
sampler = KFold(nSplits=5,nRepeats=2,shuffle=true, rng=copy(FIXEDRNG))
cOut = crossValidation([x,y],sampler,returnStatistics=false) do trainData,testData,rng
    (xtrain,ytrain)  = trainData;
    clusteringOut  = [gmm(xtrain,k,mixtures=[FullGaussian() for i in 1:k], verbosity=NONE, rng=rng) for k in 1:K]
    BICS           = [clusteringOut[i].BIC for i in 1:K]
    AICS           = [clusteringOut[i].AIC for i in 1:K]
    return (BICS,AICS)
end

## Transforming the output in matrices for easier analysis
Nit = length(cOut)

BICS = fill(0.0,(Nit,K))
AICS = fill(0.0,(Nit,K))
[BICS[r,c] = cOut[r][1][c] for r in 1:Nit,c in 1:K]
[AICS[r,c] = cOut[r][2][c] for r in 1:Nit,c in 1:K]

μsBICS = mean(BICS,dims=1)
#-
σsBICS = std(BICS,dims=1)
#-
μsAICS = mean(AICS,dims=1)
#-
σsAICS = std(AICS,dims=1)
#-
plot(1:K,[μsBICS' μsAICS'], labels=["BIC" "AIC"], title="Information criteria by number of classes", xlabel="number of classes", ylabel="lower is better")

# We see that following the "lowest AIC" rule we would indeed choose three classes, while following the "best AIC" criteria we would have choosen only two classes. This means that there is two classes that, concerning the floreal measures used in the database, are very similar, and opur models are unsure about them. Perhaps the biologists will end up one day with the conclusion that it is indeed only one specie :-).

# We could study this issue more in detail by analysing the [`ConfusionMatrix`](@ref), but the one used in BetaML does not account for the ignoreLabels option (yet).

# ## Benchmarking computational efficiency

# We now benchmark the time and memory required by the various models by using the `@btime` macro of the `BenchmarkTools` package:

# ```
# @btime kmeans($xs,3);
# # 261.540 μs (3777 allocations: 442.53 KiB)
# @btime kmedoids($xs,3);
# 4.576 ms (97356 allocations: 10.42 MiB)
# @btime gmm($xs,3,mixtures=[SphericalGaussian() for i in 1:3], verbosity=NONE);
# # 5.498 ms (133365 allocations: 8.42 MiB)
# @btime gmm($xs,3,mixtures=[DiagonalGaussian() for i in 1:3], verbosity=NONE);
# # 18.901 ms (404333 allocations: 25.65 MiB)
# @btime gmm($xs,3,mixtures=[FullGaussian() for i in 1:3], verbosity=NONE);
# # 49.257 ms (351500 allocations: 61.95 MiB)
# @btime Clustering.kmeans($xs', 3);
# # 17.071 μs (23 allocations: 14.31 KiB)
# @btime begin dGMM = GaussianMixtures.GMM(3, $xs; method=:kmeans, kind=:diag); GaussianMixtures.em!(dGMM, $xs) end;
# # 530.528 μs (2088 allocations: 488.05 KiB)
# @btime begin fGMM = GaussianMixtures.GMM(3, $xs; method=:kmeans, kind=:full); GaussianMixtures.em!(fGMM, $xs) end;
# # 4.166 ms (58910 allocations: 3.59 MiB)
# ```
# (_note: the values reported here are of a local pc, not of the GitHub CI server, as sometimes - depending on data and random initialisation - `GaussainMixtures.em!`` fails with a `PosDefException`. This in turln would lead the whole documentation to fail to compile_)

# Like for supervised models, dedicated models are much better optimized than BetaML models, and are order of magnitude more efficient. However even the slowest BetaML clusering model (gmm using full gaussians) is realtively fast and can handle mid-size datasets (tens to hundreds of thousand records) without significant slow downs.

# ## Conclusions

# We have shown in this tutorial how we can easily run clustering almgorithms in BetaML with just one line of code `choosenModel(x,k)`, but also how can we use cross-validation in order to help the model or parameter selection, with or whithout knowing the real classes.
# We retrieve here what we observed with supervised models. Globally the accuracy of BetaML models are comparable to those of leading specialised packages (in this case they are even better), but there is a significant gap in computational efficiency that restricts the pratical usage of BetaML to mid-size datasets. However we trade this relative inefficiency with very flexible model definition and utility functions (for example the BetaML gmm works with missing data, allowing it to be used as the backbone of the [`predictMissing`](@ref) missing imputation function, or for collaborative reccomendation systems).


export adjointTest

function adjointTest(m0,pFor,out::Bool=true)

end

import jInv.Utils.checkDerivative
function checkDerivative(m0,pFor::ForwardProbType;kwargs...)
    function testFun(x,v=[])
        fc = vec(getData(x,pFor)[1])
		if !(isempty(v))
			return fc, getSensMatVec(v,x,pFor)
		else
			return fc
		end
	end
    return checkDerivative(testFun,m0;kwargs...)
end

"""
function jInv.adjointTest

automatic adjoint test for forward problems. If sensitivity matrix is m x n then
this method generates vector v,m of lengths n and m, respectively and computes

err = abs(dot(v,JTw) - dot(w,Jv))  / abs(dot(w,Jv))

Input:

	sig::Vector           - current model
	pFor::ForwardProbType - forward problem

Optional Inputs:

	out::Bool            - controls verbosity (default=false)
	tol::Real            - tol on relative error (default=1e-10)

Output:

	passed              - true/false depending whether test passed or not
	err                 - absolute error
"""
function adjointTest(sig,pFor::ForwardProbType;out::Bool=false,tol::Real=1e-10)

    (out) && println("calling forward problem")
    dobs, pFor = getData(sig,pFor)

    (out) && println("calling getSensMatVec")
    v = getRandomTestDirection(sig)
    Jv = getSensMatVec(v,sig,pFor)
    w = getRandomTestDirection(Jv)
    t1 = dot(w,Jv)
    (out) && println("t1 = $(t1)")

    (out) && println("calling getSensTMatVec")
    JTw = getSensTMatVec(w,sig,pFor)
    t2 = dot(v,JTw)
    (out) && println("t2 = $(t2)")

    err = abs(t1-t2)/abs(t1)
    return err < tol, err
end


# ---
# title: 1307. Verbal Arithmetic Puzzle
# id: problem1307
# author: Tian Jun
# date: 2020-10-31
# difficulty: Hard
# categories: Math, Backtracking
# link: <https://leetcode.com/problems/verbal-arithmetic-puzzle/description/>
# hidden: true
# ---
#
# Given an equation, represented by `words` on left side and the `result` on
# right side.
#
# You need to check if the equation is solvable under the following rules:
#
#   * Each character is decoded as one digit (0 - 9).
#   * Every pair of different characters they must map to different digits.
#   * Each `words[i]` and `result` are decoded as one number **without** leading zeros.
#   * Sum of numbers on left side (`words`) will equal to the number on right side (`result`).
#
# Return `True` if the equation is solvable otherwise return `False`.
#
#
#
# **Example 1:**
#
#
#
#     Input: words = ["SEND","MORE"], result = "MONEY"
#     Output: true
#     Explanation: Map 'S'-> 9, 'E'->5, 'N'->6, 'D'->7, 'M'->1, 'O'->0, 'R'->8, 'Y'->'2'
#     Such that: "SEND" + "MORE" = "MONEY" ,  9567 + 1085 = 10652
#
# **Example 2:**
#
#
#
#     Input: words = ["SIX","SEVEN","SEVEN"], result = "TWENTY"
#     Output: true
#     Explanation: Map 'S'-> 6, 'I'->5, 'X'->0, 'E'->8, 'V'->7, 'N'->2, 'T'->1, 'W'->'3', 'Y'->4
#     Such that: "SIX" + "SEVEN" + "SEVEN" = "TWENTY" ,  650 + 68782 + 68782 = 138214
#
# **Example 3:**
#
#
#
#     Input: words = ["THIS","IS","TOO"], result = "FUNNY"
#     Output: true
#
#
# **Example 4:**
#
#
#
#     Input: words = ["LEET","CODE"], result = "POINT"
#     Output: false
#
#
#
#
# **Constraints:**
#
#   * `2 <= words.length <= 5`
#   * `1 <= words[i].length, result.length <= 7`
#   * `words[i], result` contains only upper case English letters.
#   * Number of different characters used on the expression is at most 10.
#
#
## @lc code=start
using LeetCode

## add your code here:
## @lc code=end


"""
    interwine{T}(v1::Vector{T}, v2::Vector{T})

Interwine the elements of two vectors. The resulting
vector has double the length of the original ones.

**Example**
```
interwine([1,1,1], [2,2,2]) == [1,2,1,2,1,2]
```
"""
function interwine(v1::Vector{T}, v2::Vector{T}) where T
    @assert size(v1) == size(v2)
    v = Vector{T}(2length(v1))
    v[1:2:end] = v1
    v[2:2:end] = v2
    return v
end

"""
    interwine{T}(m1::Matrix{T}, m2::Matrix{T})

Interwine the columns of two matrices.
"""
function interwine(m1::Matrix{T}, m2::Matrix{T}) where T
    @assert size(m1) == size(m2)
    v = Vector{Vector{T}}(2size(m1,2))
    for j=1:size(m1,2)
        v[2j-1] = m1[:,j]
        v[2j] = m2[:,j]
    end
    return hcat(v...)
end