"
`FitnessFunction` represents a fitness function that is going to be evaluated
over the individuals of a population.

# Fields
- `_fitnessFunction::Function`: the actual function that is going to be run.
- `_varArgs::Array`: arguments of the fitness function (if any).
- `_weight::Float32`: the weight of the fitness value. If this value is positive,
    the fitness function will be maximized, if it is negative, the fitness
    function will be minimized.

Several considerations must be taken in order to create a fitness function.

Every fitness function must **always** recieve the individual's genotype as its
first argument. If the function needs other arguments besides the genotype, they
are specified in the `_varArgs` field of the fitness function (see Fields). The
user does **not** need to specify the genotype as one of the fitness function
arguments, as the system already knows it is one of them. Here is an example of
how to create a `FitnessFunction` instance:

```jldoctest
julia> function countBinaryOnes(genotype::BinaryGenotype)
    representation = getRepresentation(genotype)
    tam = length(representation)
    fitness = 0
    for i=1:tam
        if representation[i] == 1
            fitness += 1
        end
    end
    return fitness
end
countBinaryOnes (generic function with 1 method)

julia> FitnessFunction(countBinaryOnes)
FitnessFunction(countBinaryOnes, Any[], 1.0)
```

A `FitnessFunction` instance has been created with `countBinaryOnes` as the
actual function, which receives two arguments (2 and 0.5), and a weight of -1.0,
so it is to be minimized. Notice that, in order to get the actual representation
of the genotype (which is BinaryGenotype), [`getRepresentation`](@ref) function
is called, which returns in this case an array of Bool.

If it is a Genetic Programming problem, the fitness functions must receive
another mandatory argument as their second argument, even if they do not use it,
which is the information about the GP experiment (see [`GPExperimentInfo`](@ref)).
Again, the user does **not** need to specify de GP experiment information as one
of the fitness function arguments. Here is an example of how to create a fitness
function for a GP problem:

```jldoctest
julia> function myFitnessFunction(genotype::STGPGenotype, gpExperimentInfo::STGPInfo, arg1, arg2, arg3)
    #···Code···
    return fitness
end
myFitnessFunction (generic function with 1 method)

julia> FitnessFunction(myFitnessFunction, [1,0,0,1], [5,2,6,1], [6,7,2,1], weight=2)
FitnessFunction(myFitnessFunction, Any[[1,0,0,1], [5,2,6,1], [6,7,2,1]], 2.0)
```

Here, a `FitnessFunction` instance has been created with `myFitnessFunction` as the
actual function, which receives three arrays as arguments, and a weight of 2.0,
so it is to be maximized and it weights twice as much.

For the GP case, the obtention of the genotype's representation can be most of
the time meaningless, so the user might want to obtain the phenotype instead.
The user can obtain the phenotype right from the genotype by calling the
[`getPhenotype`](@ref) function (click to see documentation), and once they
get the phenotype, it can be evaluated by calling the [`evaluate`](@ref) function
(click to see documentation), which returns the result of the phenotype for a
specific set of values. In this example, a fitness function for a STGP problem
that creates decision trees for binary classification is shown:

```jldoctest
julia> function CCR(genotype::STGPGenotype, gpExperimentInfo::STGPInfo, classes::Array{Bool}, xValues::Array{Int64}, yValues::Array{Int64})

    rightGuesses = 0
    phenotype = getPhenotype(genotype); #Obtains the phenotype
    nValues = length(xValues)

    for i=1:nValues
        # Evaluates the phenotype for a set of the values of the variables x and y
        estimated_class = evaluate(phenotype, getVariables(gpExperimentInfo), xValues[i], yValues[i])
        if typeof(estimated_class) <: Bool
            if classes[i] == estimated_class
                rightGuesses += 1
            end
        end
    end

    # Calculates CCR, which is the fitness value of the individual
    fitness = (rightGuesses/nValues)*100
    return fitness
end
CCR (generic function with 1 method)

julia> classes = [true, true, false, true]
4-element Array{Bool,1}:
[...]

julia> xValues = [2, 1, 8, 5]
4-element Array{Bool,1}:
[...]

julia> yValues = [4, 6, 2, 5]
4-element Array{Bool,1}:
[...]

julia> FitnessFunction(CCR, classes, xValues, yValues, weight=1)
FitnessFunction(CCR, Any[Bool[1, 1, 0, 1], [2, 1, 8, 5], [4, 6, 2, 5]], 1.0)
```

As the reader can already guess, the fitness of the individuals of the population
will be their CCR, which is the number of patterns correctly classified. In order
to do so, we have created a fitness function that receives the mandatory arguments,
which are the individual's genotype and the GP experiment information, and, in
addition, it also receives the true classes and the values for the two variable
nodes of the problem, which are \"x\" and \"y\".

To evaluate the code generated from a GP individual, first, the user shall obtain
its phenotype, which is the Julia code translated from a GP tree. Then, the user
can evaluate the phenotype using the actual values as it is shown in the fitness
function by calling `evaluate`.

Once the fitness function and the arguments it is going to receive are created,
the user can create its instance of `FitnessFunction` as it is shown.

See also: [`Evaluator`](@ref)
"
struct FitnessFunction
    _fitnessFunction::Function
    _varArgs::Array
    _weight::Float32

    function FitnessFunction(fitnessFunction::Function, varArgs...;
                             weight::Real = 1.0)
        varArgs = [i for i in varArgs]
        new(fitnessFunction, varArgs, weight)
    end
end # struct

"""
    getFunction(fitnessfunction::FitnessFunction)::Function

Obtains the function of a fitness function.
"""
getFunction(fitnessfunction::FitnessFunction)::Function = fitnessfunction._fitnessFunction
# function

"""
    getFunctionArgs(fitnessfunction::FitnessFunction)::Array

Obtains the arguments of a fitness function.
"""
getFunctionArgs(fitnessfunction::FitnessFunction)::Array = fitnessfunction._varArgs
# function

"""
    getWeight(fitnessFunction::FitnessFunction)::Float32

Obtains the weight of a fitness function.
"""
getWeight(fitnessFunction::FitnessFunction)::Float32 = fitnessFunction._weight
# function

"""
    getWeight(fitnessFunctions::Array{FitnessFunction}, index::Integer=1)::Float32

Obtains the weight of a fitness function within an array of fitness functions.
"""
getWeight(fitnessFunctions::Array{FitnessFunction}, index::Integer=1)::Float32 = getWeight(fitnessFunctions[index])
# function

"""
    getWeights(fitnessFunctions::Array{FitnessFunction})::Array{Float32}

Obtains the weights of all the fitness function of an array.
"""
function getWeights(fitnessFunctions::Array{FitnessFunction})::Array{Float32}
    len = length(fitnessFunctions)
    weights = Array{Float32}(undef, len)
    for i=eachindex(weights)
        @inbounds weights[i] = getWeight(fitnessFunctions, i)
    end
    return weights
end # function

"""
    fitnessEval(fitnessFunc::FitnessFunction, genotype::Genotype)::Float64

Returns the result of a fitness function for a determined genotype.
"""
function fitnessEval(fitnessFunc::FitnessFunction, genotype::Genotype)::Float64
    ff = getFunction(fitnessFunc)
    args = getFunctionArgs(fitnessFunc)
    ff(genotype, args...)
end # function

"""
    fitnessEval(fitnessFunc::FitnessFunction,
                gpExperimentInfo::GPExperimentInfo, genotype::Genotype)::Float64

Returns the result of a fitness function for a determined genotype.
"""
function fitnessEval(fitnessFunc::FitnessFunction,
                     gpExperimentInfo::GPExperimentInfo, genotype::Genotype)::Float64
    ff = getFunction(fitnessFunc)
    args = getFunctionArgs(fitnessFunc)
    ff(genotype, gpExperimentInfo, args...)
end # function

"
This struct evaluates each individual according to a fitness function and an
assigned weight to each value of fitness.

# Fields
- `_fitnessFunctions::Array{FitnessFunction}`: set of fitness functions that are
    going to be evaluated for every individual of the population.
- `_globalFitnessFunction::FitnessFunction`: second-phase fitness function. If set,
    this function will be evaluated once all the individuals of the population
    have been evaluated with their raw fitness values.
- `_compareFunction::Function`: method used for comparison between individuals.
- `_compareFunctionArgs::String`: tells the evaluator what is going to be
    compared. \"raw\" is for fitness comparison, \"rawW\" is for weighted fitness comparison,
    \"global\" is for second-phase fitness comparison, \"globalW\" is for weighted
    second-phase fitness comparison, \"both\" is for both fitness and second-phase
    fitness comparison and \"bothW\" is for both weighted fitness and weighted
    second-phase fitness comparison.

If the user wants to set a second-phase fitness function, there are some
considerations to be taken.

The global fitness function must **always** receive the population of individuals
as its first argument and the fitness weights as its second argument, as they are
used to determine which fitness values weight more and which ones weight less.
The user can specify other arguments needed when creating the `FitnessFunction`
instance, but they do **not** need to specify the population or the weights as
one of them, as the system already know they are. Also, the function must return
an array with the second-phase fitness values of all the individuals in the
population. Here is an example of how a second-phase fitness function should be
created:

```jldoctest
julia> function myGlobalFitnessFunction(population::Array{Individual}, weights::Array{Real})
    #···Code···
    return globalFitnessValues
end
myGlobalFitnessFunction (generic function with 1 method)

julia> FitnessFunction(myGlobalFitnessFunction)
FitnessFunction(myGlobalFitnessFunction, Any[], 1.0)
```

Here, a second-phase fitness function has been created using `myGlobalFitnessFunction`
as the actual function, which takes no arguments, and assigning 1.0 as its weight.
The way the system know this is the second-phase fitness function is
by specifying so when calling `setEvaluator` with the keyword argument
`globalFitnessFunction`.

See also: [`setEvaluator`](@ref), [`FitnessFunction`](@ref)
"
struct Evaluator
    _fitnessFunctions::Array{FitnessFunction}
    _globalFitnessFunction::FitnessFunction
    _compareFunction::Function
    _compareFunctionArgs::String
end # struct

"""
    getNumberFitness(evaluator::Evaluator)

Obtains the number of fitness functions of the problem
"""
getNumberFitness(evaluator::Evaluator) = length(evaluator._fitnessFunctions)
# function

"""
    getFitnessFunction(evaluator::Evaluator, index::Integer)::FitnessFunction

Obtains a fitness function of the evaluator.
"""
getFitnessFunction(evaluator::Evaluator, index::Integer)::FitnessFunction = evaluator._fitnessFunctions[index]
# function

"""
    hasGlobalFitnessFunction(evaluator::Evaluator)::Bool

Checks if a global fitness function has been assigned.
"""
function hasGlobalFitnessFunction(evaluator::Evaluator)::Bool
    return getFunction(evaluator._globalFitnessFunction) != noFunc
end
# function

"""
    getGlobalFitnessFunction(evaluator::Evaluator)::FitnessFunction

Obtains the global fitness function of the evaluator.
"""
getGlobalFitnessFunction(evaluator::Evaluator)::FitnessFunction = evaluator._globalFitnessFunction
# function

"""
    getWeight(evaluator::Evaluator, index::Integer=1)::Float32

Obtains the weight of a fitness function of the evaluator.
"""
getWeight(evaluator::Evaluator, index::Integer=1)::Float32 = getWeight(evaluator._fitnessFunctions, index)
# function

"""
    getWeights(evaluator::Evaluator)::Array{Float32}

Obtains the weights of all the fitness function of the evaluator.
"""
getWeights(evaluator::Evaluator)::Array{Float32} = getWeights(evaluator._fitnessFunctions)
# function

"""
    getCompareFunction(evaluator::Evaluator)::Function

Obtains the method used for comparison of the evaluator.
"""
getCompareFunction(evaluator::Evaluator)::Function = evaluator._compareFunction
# function

"""
    getParametrizedCompareFunction(compareFunction::Function, args::String,
                                   weights::Array{Real})::Function

Obtains the comparison function based on the weights and the number of fitness
functions of the evaluator.
"""
function getParametrizedCompareFunction(compareFunction::Function, args::String,
                                        weights::Array{Float32})::Function

    local compare
    #compare = () -> nothing
    if compareFunction == noFunc
        if length(weights) == 1
            if sign(weights[1]) == 1
                compare = (ind1::Individual, ind2::Individual) -> compareFitnessMaximize(getFitness(ind1, all=true), getFitness(ind2, all=true))
            else
                compare = (ind1::Individual, ind2::Individual) -> compareFitnessMinimize(getFitness(ind1, all=true), getFitness(ind2, all=true))
            end
        else
            compare = (ind1::Individual, ind2::Individual) -> compareFitnessLinearCombination(getFitness(ind1, all=true), getFitness(ind2, all=true), weights)
        end
    else
        weights = length(weights) == 1 ? weights[1] : weights
        if args == "raw"
            compare = (ind1::Individual, ind2::Individual) -> compareFunction(getFitness(ind1, all=true), getFitness(ind2, all=true))
        elseif args == "rawW"
            compare = (ind1::Individual, ind2::Individual) -> compareFunction(getFitness(ind1, all=true), getFitness(ind2, all=true), weights)
        elseif args == "global"
            compare = (ind1::Individual, ind2::Individual) -> compareFunction(getGlobalFitness(ind1), getGlobalFitness(ind2))
        elseif args == "globalW"
            compare = (ind1::Individual, ind2::Individual) -> compareFunction(getGlobalFitness(ind1), getGlobalFitness(ind2), weights)
        elseif args == "both"
            compare = (ind1::Individual, ind2::Individual) -> compareFunction(getFitness(ind1, all=true), getGlobalFitness(ind1), getFitness(ind2, all=true), getGlobalFitness(ind2))
        else
            compare = (ind1::Individual, ind2::Individual) -> compareFunction(getFitness(ind1, all=true), getGlobalFitness(ind1), getFitness(ind2, all=true), getGlobalFitness(ind2), weights)
        end
    end
    return compare
end # function

"""
    computeFitness(evaluator::Evaluator, experimentInfo::ExperimentInfo,
                   genotype::Genotype, stopCondition::StopCondition)::Array{Float64}

Computes the fitness of a single individual.
"""
function computeFitness(evaluator::Evaluator, experimentInfo::ExperimentInfo,
                        genotype::Genotype, stopCondition::StopCondition)::Array{Float64}

    len = length(evaluator._fitnessFunctions)
    fitnesses = Array{Float64}(undef,len)

    if experimentInfo._individualType <: GAGenotype
        for i=eachindex(fitnesses)
            @inbounds fitnesses[i] = fitnessEval(getFitnessFunction(evaluator, i), genotype)
        end
    else
        for i=eachindex(fitnesses)
            @inbounds fitnesses[i] = fitnessEval(getFitnessFunction(evaluator, i), experimentInfo._GPExperimentInfo, genotype)
        end
    end

    if stopCondition._maxEvaluations != -1
        notifyEvaluation(stopCondition)
    end

    return fitnesses
end # function

"""
    globalFitnessEval(fitnessFunc::FitnessFunction,
                      population::Array{Individual},
                      weights::Array{Real})::Float64

Returns the result of a global fitness function for a population.
"""
function globalFitnessEval(fitnessFunc::FitnessFunction,
                           population::Array{Individual},
                           weights::Array{Float32})::Array{Float64}

    ff = getFunction(fitnessFunc)
    args = getFunctionArgs(fitnessFunc)
    ff(population, weights, args...)
end # function

"""
    evaluate!_(evaluator::Evaluator, experimentInfo::ExperimentInfo,
               population::Array{Individual}, stopCondition::StopCondition)

Evaluates the fitness of the individuals of a population and sets their values.
"""
function evaluate!_(evaluator::Evaluator, experimentInfo::ExperimentInfo,
                    population::Array{Individual}, stopCondition::StopCondition)
    for ind in population
        if hasNullFitness(ind)
            ind._fitness = computeFitness(evaluator, experimentInfo, ind._genotype, stopCondition)
        end
    end

    globalFitnessFunc = getGlobalFitnessFunction(evaluator)
    if getFunction(globalFitnessFunc) != noFunc
        fitnesses = getFitness(population)
        weights = getWeights(evaluator)
        globalFitnesses = globalFitnessEval(globalFitnessFunc, population, weights)
        setGlobalFitness!(population, globalFitnesses)
    end
    nothing
end # function

"""
    compareFitnessMaximize(fitness1::Float64, fitness2::Float64)

Compares two values of fitness and returns true if the first one is greater than
the second one, returns false otherwise.
"""
function compareFitnessMaximize(fitness1::Float64, fitness2::Float64)
    return fitness1 > fitness2 ? true : false
end # function

"""
    compareFitnessMinimize(fitness1::Float64, fitness2::Float64)

Compares two values of fitness and returns true if the first one is less than
the second one, returns false otherwise.
"""
function compareFitnessMinimize(fitness1::Float64, fitness2::Float64)
    return fitness1 < fitness2 ? true : false
end # function

"""
    compareFitnessLinearCombination(fitness1::Array{Float64},
                                    fitness2::Array{Float64},
                                    weights::Array{Float32})

Compares two sets of fitness values calculating the dot product between each one
of them and the weights assigned. Returns true if the first set of fitness values
is greater than the second one, returns false otherwise.
"""
function compareFitnessLinearCombination(fitness1::Array{Float64},
                                         fitness2::Array{Float64},
                                         weights::Array{Float32})

    function dot(array1::Array, array2::Array)
        sum(array1 .* array2)
    end

    conjunction1 = dot(fitness1, weights)
    conjunction2 = dot(fitness2, weights)

    return conjunction1 > conjunction2 ? true : false
end # function

"""
    getBestIndividual(population::Array{Individual}, compare::Function,
                      n::Integer = 1)::Array{Int}

Obtains the `n` best individuals of a population according to a comparing function.
"""
function getBestIndividual(population::Array{Individual}, compare::Function,
                           n::Integer = 1)::Array{Int}

    bestIndividual = collect(1:n)
    len = length(population)

    @inbounds for i=1:n
        for j=i+1:n
            if compare(population[bestIndividual[i]], population[bestIndividual[j]])
                bestIndividual[j], bestIndividual[i] = bestIndividual[i],  bestIndividual[j]
            end
        end
    end

    @inbounds for i=n+1:len
        if compare(population[i], population[bestIndividual[1]])
            bestIndividual[1] = i
            for j=1:n-1
                if compare(population[bestIndividual[j]], population[bestIndividual[j+1]]) || break
                    bestIndividual[j], bestIndividual[j+1] = bestIndividual[j+1],  bestIndividual[j]
                end
            end
        end
    end

    return bestIndividual
end # function

"""
    getWorstIndividual(population::Array{Individual}, compare::Function,
                       n::Integer=1)::Array{Int}

Obtains the `n` worst individuals of a population according to a comparing function.
"""
function getWorstIndividual(population::Array{Individual}, compare::Function,
                            n::Integer=1)::Array{Int}

    worstIndividual = collect(1:n)
    len = length(population)

    @inbounds for i=1:n
        for j=i+1:n
            if !compare(population[worstIndividual[i]], population[worstIndividual[j]])
                worstIndividual[j], worstIndividual[i] = worstIndividual[i],  worstIndividual[j]
            end
        end
    end

    @inbounds for i=n+1:len
        if !compare(population[i], population[worstIndividual[1]])
            worstIndividual[1] = i
            for j=1:n-1
                if !compare(population[worstIndividual[j]], population[worstIndividual[j+1]]) || break
                    worstIndividual[j], worstIndividual[j+1] = worstIndividual[j+1],  worstIndividual[j]
                end
            end
        end
    end

    if n == 1
        worstIndividual = worstIndividual[1]
    end

    return worstIndividual
end # function

"""
    pareto(population::Array{Individual}, weights::Array{Float32})::Array{Float64}

Assigns every individual of a population its second-phase fitness which is the
level they are in the pareto front.

# Arguments
- `population::Array{Individual}`: population of individuals that are going to be
    second-phase evaluated.
- `weights::Array{Real}`: array of weights for every single "raw fitness"
    function.

# Returns
An array with the levels of the individuals in the pareto front.
"""
function pareto(population::Array{Individual}, weights::Array{Float32})::Array{Float64}

    weightsSign = sign.(weights)

    function dominates(fitness1, fitness2)
        nFitness = length(fitness1)
        nDominations1 = 0
        nDominations2 = 0

        fitness1 = fitness1 .* weightsSign
        fitness2 = fitness2 .* weightsSign

        for i=1:nFitness
            if fitness1[i] > fitness2[i]
                nDominations1 += 1
            elseif fitness1[i] < fitness2[i]
                nDominations2 += 1
            end
        end
        if nDominations1 == nFitness
            return 1
        elseif nDominations2 == nFitness
            return -1
        end

        return 0
    end # function

    popsize = length(population)
    fitnesses = getFitness(population)
    dominateMatrix = zeros(Integer, popsize, popsize+1)
    dominateMatrix[:,end] = collect(1:popsize)

    @inbounds for i=1:popsize
        for j=i+1:popsize
            dominateMatrix[i,j] = dominates(fitnesses[i,:], fitnesses[j,:])
            dominateMatrix[j,i] = dominateMatrix[i,j] * (-1)
        end
    end

    paretoLevel = zeros(Integer, popsize)

    level = 0

    while (true)
        level+=1
        len = size(dominateMatrix)[1]
        if len == 0
            break
        end
        rows = ones(Bool, len)
        cols = ones(Bool, len+1)
        @inbounds for i=1:len
            n = count(i->(i==-1), view(dominateMatrix, i, :))
            if n == 0
                paretoLevel[dominateMatrix[i,end]] = level
                rows[i] = false
                cols[i] = false
            end
        end
        dominateMatrix = dominateMatrix[rows, cols]
    end

    return paretoLevel
end # function


using NonconvexNLopt
function convert_common_kwargs(opt::NonconvexNLopt.NLoptAlg, opt_kwargs;
    callback=nothing,
    maxiters=nothing,
    maxtime=nothing,
    abstol=nothing,
    reltol=nothing)

    conv_opt_kwargs = (; opt_kwargs...)

    if !isnothing(callback)
        @warn "common callback argument is currently not used by $(opt)"
    end

    if !isnothing(maxiters)
        conv_opt_kwargs = (; conv_opt_kwargs..., maxeval=maxiters)
     end

    if !isnothing(maxtime)
        conv_opt_kwargs = (; conv_opt_kwargs..., maxtime = maxtime)
    end

    if !isnothing(abstol)
        conv_opt_kwargs = (; conv_opt_kwargs..., ftol_abs = abstol)
    end

    if !isnothing(reltol)
        conv_opt_kwargs = (; conv_opt_kwargs..., ftol_rel =reltol)
    end

    return conv_opt_kwargs
end

function __create_options(opt::NonconvexNLopt.NLoptAlg;
    opt_kwargs=nothing)

    options = !isnothing(opt_kwargs) ? NonconvexNLopt.NLoptOptions(;opt_kwargs...) : NonconvexNLopt.NLoptOptions()

    return options
end

function _create_options(opt::NonconvexNLopt.NLoptAlg;
    opt_kwargs=nothing,
    sub_options=nothing,
    convergence_criteria=nothing)

    options = (; options = __create_options(opt, opt_kwargs=opt_kwargs))

    return options
end

check_optimizer_backend(opt::NonconvexNLopt.NLoptAlg) = false


using ChainRulesCore
using Documenter
using DocumenterTools: Themes
using Markdown

DocMeta.setdocmeta!(
    ChainRulesCore,
    :DocTestSetup,
    quote
        using Random
        Random.seed!(0)  # frule doctest shows output

        using ChainRulesCore
        # These rules are all actually defined in ChainRules.jl, but we redefine them here to
        # avoid the dependency.
        @scalar_rule(sin(x), cos(x))  # frule and rrule doctest
        @scalar_rule(sincos(x), @setup((sinx, cosx) = Ω), cosx, -sinx)  # frule doctest
        @scalar_rule(hypot(x::Real, y::Real), (x / Ω, y / Ω))  # rrule doctest
    end
)

Themes.compile(joinpath(@__DIR__, "src/assets/chainrules.scss"))

makedocs(
    modules=[ChainRulesCore],
    format=Documenter.HTML(
        prettyurls=false,
        assets=["assets/chainrules.css"],
        mathengine=MathJax(),
    ),
    sitename="ChainRules",
    authors="Jarrett Revels and other contributors",
    pages=[
        "Introduction" => "index.md",
        "FAQ" => "FAQ.md",
        "Writing Good Rules" => "writing_good_rules.md",
        "Complex Numbers" => "complex.md",
        "Deriving Array Rules" => "arrays.md",
        "Debug Mode" => "debug_mode.md",
        "Gradient Accumulation" => "gradient_accumulation.md",
        "Usage in AD" => [
            "Overview" => "autodiff/overview.md",
            "Operator Overloading" => "autodiff/operator_overloading.md",
        ],
        "Design" => [
            "Changing the Primal" => "design/changing_the_primal.md",
            "Many Differential Types" => "design/many_differentials.md",
        ],
        "API" => "api.md",
    ],
    strict=true,
    checkdocs=:exports,
)

deploydocs(
    repo = "github.com/JuliaDiff/ChainRulesCore.jl.git",
    push_preview=true,
)


# -------------------------------------
# Results and related sub structures
# -------------------------------------

"""
	ResultTimes

Part of the Result object returned by the solver. ResultTimes contains timing results for certain parts of the algorithm:

Time Name  | Description
---  | :---
solver_time | Total time used to solve the problem
setup_time |  Setup time = graph\\_time + init\\_factor\\_time + scaling\\_time
scaling_time | Time to scale the problem data
graph_time | Time used to perform chordal decomposition
init\\_factor\\_time | Time used for initial factorisation of the system of linear equations
factor\\_update\\_time | Sum of times used to refactor the system of linear equations due to rho
iter_time | Time spent in iteration loop
proj_time | Time spent in projection functions
post_time | Time used for post processing
update_time | Time spent in the update! function of the accelerator
accelerate_time | Time spent in the accelerate! function of the accelerator

By default COSMO only measures `solver_time`, `setup_time` and `proj_time`. To measure the other times set `verbose_timing = true`.
"""
mutable struct ResultTimes{T <: AbstractFloat}
	solver_time::T
	setup_time::T
	scaling_time::T
	graph_time::T
	init_factor_time::T
	factor_update_time::T
	iter_time::T
	proj_time::T
	post_time::T
	update_time::T
	accelerate_time::T
end

ResultTimes{T}() where{T} = ResultTimes{T}(T(NaN), T(NaN), T(NaN), T(NaN), T(NaN), T(NaN), T(NaN), T(NaN), T(NaN), T(NaN), T(NaN))
ResultTimes(T::Type = DefaultFloat) = ResultTimes{T}()

function Base.show(io::IO, obj::ResultTimes)
  obj.iter_time != 0 ? verbose = true : verbose = false
  print(io,"Solver time:\t$(round.(obj.solver_time, digits = 4))s ($(round.(obj.solver_time * 1000, digits = 2))ms)\n",
"Setup time:\t$(round.(obj.setup_time, digits = 4))s ($(round.(obj.setup_time * 1000, digits = 2))ms)\n",
"Proj time:\t$(round.(obj.proj_time, digits = 4))s ($(round.(obj.proj_time * 1000, digits = 2))ms)\n")
  if verbose
    print(io,"Iter time:\t$(round.(obj.iter_time, digits = 4))s ($(round.(obj.iter_time * 1000, digits = 2))ms)\n",
    "Scaling time:\t$(round.(obj.scaling_time, digits = 4))s ($(round.(obj.scaling_time * 1000, digits = 2))ms)\n",
    "Graph time:\t$(round.(obj.graph_time, digits = 4))s ($(round.(obj.graph_time * 1000, digits = 2))ms)\n",
    "Initial Factor time:\t$(round.(obj.init_factor_time, digits = 4))s ($(round.(obj.init_factor_time * 1000, digits = 2))ms)\n",
    "Factor update time:\t$(round.(obj.factor_update_time, digits = 4))s ($(round.(obj.factor_update_time * 1000, digits = 2))ms)\n",
    "Post time:\t$(round.(obj.post_time, digits = 4))s ($(round.(obj.post_time * 1000, digits = 2))ms)\n",
    "Accelerator - update time:\t$(round.(obj.update_time, digits = 4))s ($(round.(obj.update_time * 1000, digits = 2))ms)\n",
    "Accelerator - accelerate time:\t$(round.(obj.accelerate_time, digits = 4))s ($(round.(obj.accelerate_time * 1000, digits = 2))ms)\n")
  end
end

"""
    ResultInfo{T <: AbstractFloat}

Object that contains further information about the primal residual, the dual residuals and the rho updates.
"""
struct ResultInfo{T <: AbstractFloat}
	r_prim::T
	r_dual::T
	max_norm_prim::T
	max_norm_dual::T
	rho_updates::Vector{T}
end

ResultInfo(rp, rd, ro, rho_updates) = ResultInfo{DefaultFloat}(rp, rd, rho_updates)

"""
    Result{T <: AbstractFloat}

Object returned by the COSMO solver after calling `optimize!(model)`. It has the following fields:

Fieldname | Type | Description
---  | :--- | :---
x | Vector{T}| Primal variable
y | Vector{T}| Dual variable
s | Vector{T}| (Primal) set variable
obj_val | T | Objective value
iter | Int | Total number of ADMM iterations (incl. safeguarding_iter)
safeguarding_iter | Int | Number of iterations due to safeguarding of accelerator
status | Symbol | Solution status
info | COSMO.ResultInfo | Struct with more information
times | COSMO.ResultTimes | Struct with several measured times
"""
struct Result{T <: AbstractFloat}
    x::Vector{T}
    y::Vector{T}
    s::Vector{T}
    obj_val::T
    iter::Int
    safeguarding_iter::Int
    status::Symbol
    info::ResultInfo
    times::ResultTimes

    function Result{T}() where {T <: AbstractFloat}
      return new(zeros(T, 1), zeros(T, 1), zeros(T, 1), zero(T), 0, 0, :Unsolved, ResultInfo{T}(0., 0., 0., 0., T[]), ResultTimes{T}())
    end

    function Result{T}(x, y, s, obj_val, iter, safeguarding_iter, status, info, times) where {T <: AbstractFloat}
      return new(x, y, s, obj_val, iter, safeguarding_iter, status, info, times)
    end

end

function Base.show(io::IO, obj::Result)
	print(io,">>> COSMO - Results\nStatus: ")
	if obj.status == :Solved
		result_color = :green
	else
		result_color = :red
	end
	printstyled("$(obj.status)\n", color = result_color)
	println("Iterations: $(obj.iter) (incl. $(obj.safeguarding_iter) safeguarding iterations)\nOptimal Objective: $(@sprintf("%.4g", obj.obj_val))\nRuntime: $(round.(obj.times.solver_time * 1000, digits = 2))ms\nSetup Time: $(round.(obj.times.setup_time * 1000, digits = 2))ms\n")
	!isnan(obj.times.iter_time) && print("Avg Iter Time: $(round.((obj.times.iter_time / obj.iter) * 1000, digits = 2))ms")
end

# -------------------------------------
# Problem scaling
# -------------------------------------

struct ScaleMatrices{Tf <: AbstractFloat}
	D::Union{UniformScaling{Bool}, Diagonal{Tf, Vector{Tf}} }
	Dinv::Union{UniformScaling{Bool}, Diagonal{Tf, Vector{Tf}} }
	E::Union{UniformScaling{Bool}, Diagonal{Tf, Vector{Tf}} }
	Einv::Union{UniformScaling{Bool}, Diagonal{Tf, Vector{Tf}} }
	c::Base.RefValue{Tf}
	cinv::Base.RefValue{Tf}
end

ScaleMatrices(args...) = ScaleMatrices{DefaultFloat}(args...)

ScaleMatrices{T}() where {T} = ScaleMatrices(I, I, I, I, Base.RefValue{T}(one(T)), Base.RefValue{T}(one(T)))

function ScaleMatrices{T}(m, n) where {T <: AbstractFloat}
	D    = Diagonal(ones(T, n))
	Dinv = Diagonal(ones(T, n))
	E    = Diagonal(ones(T, m))
	Einv = Diagonal(ones(T, m))
	c    = Base.RefValue{T}(one(T))
	cinv = Base.RefValue{T}(one(T))
	ScaleMatrices(D, Dinv, E, Einv, c, cinv)
end

# -------------------------------------
# Problem data
# -------------------------------------

mutable struct ProblemData{T <: AbstractFloat}
	P::AbstractMatrix{T}
	q::Vector{T}
	A::AbstractMatrix{T}
	b::Vector{T}
	C::CompositeConvexSet{T}
	model_size::Array{Integer,1}

	function ProblemData{T}() where {T <: AbstractFloat}
		return new(
			spzeros(T, 1, 1),             #P
			T[],                        #q
			spzeros(T, 1, 1),             #A
			T[],                        #b
			COSMO.CompositeConvexSet{T}([COSMO.ZeroSet{T}(1)]),     #C
			[0; 0])                 #model size
	end
end

ProblemData(args...) = ProblemData{DefaultFloat}(args...)

# ---------------------------
# Struct to hold clique and sparsity data for a constraint
# ---------------------------

mutable struct SparsityPattern
  sntree::SuperNodeTree
  ordering::Array{Int}
  reverse_ordering::Array{Int}
  row_range::UnitRange{Int} # the starting row of the psd cone in the original problem
  cone_ind::Int # this is the ind of the original psd cone in ws.p.C that is decomposed
  nz_ind_map::SparseVector{Int, Int} # maps a matrix entry k = svec(i, j) to the location of the entry in the sparse data structure

  # constructor for sparsity pattern
  function SparsityPattern(L::SparseMatrixCSC, N::Int, ordering::Array{Int, 1}, merge_strategy::Union{Type{<: AbstractMergeStrategy}, OptionsFactory{<: AbstractMergeStrategy}}, row_range::UnitRange{Int}, cone_ind::Int, nz_ind_map::SparseVector{Int, Int})

    merge_strategy = merge_strategy()
    sntree = SuperNodeTree(L, merge_strategy)

    # clique merging
    sntree.num > 1 && merge_cliques!(sntree)

    # reorder vertices in supernodes to have consecutive order
    # necessary for equal column structure for psd completion
    reorder_snd_consecutively!(sntree, ordering)

    # undo the reordering and sort
    # for iii = 1:sntree.num
    #   sep = get_sep(sntree, iii)
    #   snd = get_snd(sntree, iii)
    #   map!(v -> ordering[v], sep)
    #   map!(v -> ordering[v], snd)
    #   sort!(sep)
    #   sort!(snd)
    # end

    # for each clique determine the number of entries of the block represented by that clique
    calculate_block_dimensions!(sntree)#, merge_strategy)

    return new(sntree, ordering, invperm(ordering), row_range, cone_ind, nz_ind_map)
  end

  # For debugging
  function SparsityPattern(sntree::SuperNodeTree, ordering::Array{Int}, reverse_ordering::Array{Int}, row_range::UnitRange{Int}, cone_ind::Int)
    return new(sntree, ordering, reverse_ordering, row_range, cone_ind)
  end
end

# -------------------------------------
# Chordal Decomposition Information
# -------------------------------------
mutable struct ChordalInfo{T <: AbstractFloat}
  decompose::Bool # an internal flag to check if problem has been decomposed
  originalM::Int
  originalN::Int
  originalC::CompositeConvexSet{T}
  H::SparseMatrixCSC{T}
  sp_arr::Array{COSMO.SparsityPattern}
  psd_cones_ind::Array{Int} # stores the position of decomposable psd cones in the composite convex set
  num_psd_cones::Int # number of psd cones of original problem
  num_decomposable::Int #number of decomposable cones
  num_decom_psd_cones::Int #total number of psd cones after decomposition
  L::SparseMatrixCSC{T} #pre allocate memory for QDLDL
  cone_map::Dict{Int, Int} # map every cone in the decomposed problem to the equivalent or undecomposed cone in the original problem

  function ChordalInfo{T}(problem::COSMO.ProblemData{T}, settings::COSMO.Settings) where {T <: AbstractFloat}
    originalM = problem.model_size[1]
    originalN = problem.model_size[2]
    originalC = deepcopy(problem.C)
    num_psd_cones = length(findall(x -> typeof(x) <: Union{PsdConeTriangle{T}, PsdCone{T}} , problem.C.sets))
    # allocate sparsity pattern for each cone
    sp_arr = Array{COSMO.SparsityPattern}(undef, num_psd_cones)
    cone_map = Dict{Int, Int}()

    return new(settings.decompose, originalM, originalN, originalC, spzeros(1, 1), sp_arr, Int[], num_psd_cones, 0, 0, spzeros(1, 1), cone_map)
  end

	function ChordalInfo{T}() where{T}
		C = COSMO.CompositeConvexSet{T}([COSMO.ZeroSet{T}(1)])
		return new(false, 0, 0, C, spzeros(1, 1), COSMO.SparsityPattern[], [1])
	end

end

# -------------------------------------
# Structure of internal iterate variables
# -------------------------------------

struct Variables{T}
	w::Vector{T}
	w_prev::Vector{T}
	x::SubArray{T}
	s::SplitVector{T}
	μ::Vector{T}

	function Variables{T}(m::Int, n::Int, C::AbstractConvexSet{T}) where {T <: AbstractFloat}
		m == C.dim || throw(DimensionMismatch("set dimension is not m"))
		w = zeros(T, n + m)
		w_prev = zeros(T, n + m)
		x = view(w_prev, 1:n) #x is a view onto w_prev, to keep it in sync with s and μ
		s = SplitVector(zeros(T, m), C)
		μ = zeros(T, m)
		new(w, w_prev, x, s, μ)
	end
end

Variables(args...) = Variables{DefaultFloat}(args...)

mutable struct IterateHistory
	x_data::AbstractMatrix
	s_data::AbstractMatrix
	y_data::AbstractMatrix
	v_data::AbstractMatrix
	r_prim_data::AbstractVector
	r_dual_data::AbstractVector
	eta_data::AbstractArray
	alpha_data::AbstractArray
	cond_data::AbstractVector
	aa_fail_data::AbstractVector

	function IterateHistory(m, n, mem)
		new(zeros(n, 0), zeros(m, 0), zeros(m, 0), zeros(m + n, 0), Float64[], Float64[], zeros(mem, 0), zeros(mem + 1, 0), Float64[], Int64[])
	end
end

function update_iterate_history!(history::IterateHistory, x, s, y, v, r_prim, r_dual, eta::Vector{Float64}, cond::Float64)
	history.x_data = hcat(history.x_data, x)
	history.s_data = hcat(history.s_data, s)
	history.y_data = hcat(history.y_data, y)
	history.v_data = hcat(history.v_data, v)
	history.cond_data = push!(history.cond_data, cond)
	push!(history.r_prim_data, r_prim)
	push!(history.r_dual_data, r_dual)

	# alphas = compute_alphas(eta)
	# history.alpha_data = hcat(history.alpha_data, alphas)
	# history.eta_data = hcat(history.eta_data, eta)
end

struct UtilityVariables{T}
  vec_m::Vector{T}
  vec_n::Vector{T}
  vec_n2::Vector{T}

  function UtilityVariables{T}(m::Int, n::Int) where {T <: AbstractFloat}
    new(zeros(T, m), zeros(T, n), zeros(T, n))
  end
end

UtilityVariables(args...) = UtilityVariables{DefaultFloat}(args...)

# -------------------------------------
# a collection of state flags
# -------------------------------------

mutable struct States
	IS_ASSEMBLED::Bool # the workspace has been assembled with problem data
	IS_OPTIMIZED::Bool # the optimization function has been called on the model
	IS_CHORDAL_DECOMPOSED::Bool # the problem has been decomposed
	KKT_FACTORED::Bool # the KKT matrix has been factored
	IS_SCALED::Bool # the problem data has been scaled
	States() = new(false, false, false, false, false)
end

# -------------------------------------
# Top level container for all solver data
# -------------------------------------
"""
	Workspace{T <: AbstractFloat}()

Initializes an empty COSMO model that can be filled with problem data using `assemble!(model, P, q,constraints; [settings, x0, s0, y0])`.
"""
mutable struct Workspace{T}
	p::ProblemData{T}
	settings::Settings{T}
	sm::ScaleMatrices{T}
	ci::ChordalInfo{T}
	vars::Variables{T}
  	utility_vars::UtilityVariables{T}
	δx::Vector{T}
	δy::SplitVector{T}
	s_tl::Vector{T}
	ls::Vector{T}
	sol::Vector{T}
	x_tl::SubArray{T}
	ν::SubArray{T}
	ρ::T
	ρvec::Vector{T}
	kkt_solver::Union{AbstractKKTSolver,Nothing}
	states::States
	rho_updates::Vector{T} #keep track of the rho updates and the number of refactorisations
	rho_update_due::Bool # a flag that indicates that at the next possible iteration ρ should be updated
	infeasibility_check_due::Bool # a flag that indicates that at the next possible iteration we should check for infeasibility
	times::ResultTimes{Float64} #always 64 bit regardless of data type
	row_ranges::Array{UnitRange{Int}, 1} # store a set_ind -> row_range map
	safeguarding_iter::Int # count extra ADMM iterations due to bad quality accelerated steps
	accelerator::AbstractAccelerator
	accelerator_active::Bool
	activation_reason::AbstractActivationReason
	#constructor
	function Workspace{T}() where {T <: AbstractFloat}
		p = ProblemData{T}()
		sm = ScaleMatrices{T}()
		vars = Variables{T}(1, 1, p.C)
    	uvars = UtilityVariables{T}(1, 1)
		ci = ChordalInfo{T}()
		δx = zeros(T, 0)
		δy = SplitVector(zeros(T, 1), p.C)
		s_tl = zeros(T,0)
		ls = zeros(T, 0)
		sol = zeros(T, 1)
		x_tl = view(sol, 1:1)
		ν = view(sol, 1:1)
		return new(p, Settings{T}(), sm, ci, vars,  uvars, δx, δy, s_tl, ls, sol, x_tl, ν, zero(T), T[], nothing, States(), T[], false, false, ResultTimes(), [0:0], 0, EmptyAccelerator(), false, ImmediateActivation())
	end
end
Workspace(args...) = Workspace{DefaultFloat}(args...)

Base.show(io::IO, model::COSMO.Workspace{T}) where {T} = println(io, "A COSMO Model with Float precision: $(T)")

# Type alias facing the user
"""
	Model{T <: AbstractFloat}()

Initializes an empty COSMO model that can be filled with problem data using `assemble!(model, P, q,constraints; [settings, x0, s0, y0])`.
"""
const Model = Workspace;


#===============================================================================
                            Path imputation
===============================================================================#

function eMCMC.update!(
        update::PathImputation,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
    )
    ws, ws° = local_ws.sub_ws_diff, local_ws.sub_ws_diff°

    num_recordings = length(ws.P)

    # this loop should be entirely parallelizable, as recordings
    # are—by design—conditionally independent
    for i in 1:num_recordings
        _, ws°.ll[i] = forward_guide!(
            ws.P[i], ws°.XX[i], ws°.WW[i], ws.WW[i], update.ρs[i], _LL,
            ws.XX[i][1].x[1]; Wnr=ws°.Wnr[i]
        )
        eMCMC.accept_reject!(
            update, global_ws, local_ws, step, ws.ll[i], ws°.ll[i], i
        )
    end
end

eMCMC.log_transition_density(::Any, updt::PathImputation, args...) = 0.0

eMCMC.log_prior(::Any, updt::PathImputation, args...) = 0.0

#̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶
#               Decide what happens at the Accept/Reject step
#̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶

function eMCMC.register_accept_reject_results!(
        accepted::Bool,
        updt::PathImputation,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
        i=1
    )
    accepted && swap_paths!(local_ws, i)
    save_path!(global_ws, step, i)
    eMCMC.register_accept_reject_results!(accepted, updt, local_ws, step, i)
end

_TEMP_SAVED_PATHS = []

function save_path!(global_ws, step, i)
    save_now = ( step.mcmciter % 100 == 0 && step.pidx == 1 ) #TODO change
    save_now && save!(global_ws.XX_buffer, global_ws.sub_ws_diff.XX)
end

#TODO there is nothing special about this step, incorporate in generic eMCMC
function eMCMC.register_accept_reject_results!(
        _accepted::Bool,
        updt,
        ws::DiffusionLocalWorkspace,
        step,
        i=1,
    )
    register!(updt, _accepted) #TODO
    accepted(ws, step.mcmciter)[i] = _accepted

    # register proposal
    ll°(ws, step.mcmciter)[i] = ll°(ws)[i]

    # update local ll with accepted
    _accepted && ( ll(ws)[i] = ll°(ws)[i] )

    # register accepted
    ll(ws, step.mcmciter)[i] = ll(ws)[i]
end

function register!(updt, accepted::Bool)
    nothing #TODO implement
end

function swap!(A, B)
    for i in eachindex(A, B)
        A[i], B[i] = B[i], A[i]
    end
end

function swap_paths!(ws::DiffusionLocalWorkspace, i)
    s°, s = ws.sub_ws_diff°, ws.sub_ws_diff
    swap!(s°.WW[i], s.WW[i])
    swap!(s°.XX[i], s.XX[i])
end

#=
function recompute_loglikhd!(ws::DiffusionLocalWorkspace, i) #TODO deprecate
    num_segments = length(ws.XX[i])
    ll_tot = 0.0
    for j in 1:num_segments
        ll_tot += loglikhd(ws.XX[i][j], ws.P[i][j])
    end
    ll(ws)[i] = ll_tot
end
=#
#===============================================================================
                        Starting point imputation
===============================================================================#

function eMCMC.update!(
        update::StartingPointsImputation,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
    )
    ws, ws° = ws.sub_ws_diff, ws.sub_ws_diff°
    num_recordings = length(ws.P)

    # this loop should be entirely parallelizable, as recordings
    # are---by design---conditionally independent
    for i in 1:num_recordings
        z°, y° = proposal_start_pt(ws, i) #TODO sample new sp
        _, ws°.ll[i] = solve_and_ll!(y°, ws.WW[i], ws°.XX[i], ws.P[i])
        accept_reject!(
            update, global_ws, local_ws, step, ws.ll[i], ws°.ll[i], i
        )
    end
end

log_transition_density(::Any, updt::StartingPointsImputation, args...) = 0.0

function log_prior(
        ::eMCMC.Proposal,
        updt::StartingPointsImputation,
        ws::DiffusionLocalWorkspace,
        i,

    )
    y° = ws.sub_ws_diff°.XX[i][1].y[1]
    # a little bit of cheating with the second term (not a prior)
    logpdf(ws.x0_prior, y°)# + log_likelihood_obs(ws.sub_ws_gp.P[1], y°) #TODO integrate the latter directly into ll
end

function log_prior(
        ::eMCMC.Previous,
        updt::StartingPointsImputation,
        ws::DiffusionLocalWorkspace,
        i,
    )
    y = ws.sub_ws_diff.XX[i][1].y[1]
    # a little bit of cheating with the second term (not a prior)
    logpdf(ws.x0_prior, y)# + log_likelihood_obs(ws.sub_ws_gp.P[1], y) #TODO integrate the latter directly into ll
end

#̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶
#               Decide what happens at the Accept/Reject step
#̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶

function eMCMC.register_accept_reject_results!(
        accepted::Bool,
        updt::StartingPointsImputation,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
        i=1
    )
    accepted && swap_starting_pts_and_paths!(ws, i)
    eMCMC.register_accept_reject_results!(accepted, updt, local_ws, step, i)
end

function swap_starting_pts_and_paths!(ws::DiffusionLocalWorkspace, i)
    s°, s = ws.sub_ws_diff°, ws.sub_ws_diff
    swap!(s°.z0s[i], s.z0s[i])
    swap!(s°.XX[i], s.XX[i])
end

#===============================================================================
                            Parameter update
===============================================================================#
function eMCMC.set_proposal!(
        updt::eMCMC.MCMCParamUpdate,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
    )
    ws, ws° = local_ws.sub_ws_diff, local_ws.sub_ws_diff°
    num_recordings = length(ws.WW)
    eMCMC._set_local_state°(updt, global_ws, local_ws, step)

    for i in 1:num_recordings
        critical_change = DD.set_parameters!(
            ws.P[i],
            ws°.P[i],
            state°(local_ws),
            local_ws.loc_pnames.vp_names[i],
            local_ws.loc_pnames.vp_names_aux[i],
            local_ws.loc_pnames.θ_local_names[i],
            local_ws.loc_pnames.θ_local_aux_names[i],
            local_ws.loc_pnames.θ_local_obs[i],
            local_ws.critical_param_change[i],
        )
        step.same_layout[i] || (critical_change = true)

        #-----------------------------------------------#
        # this is where most of the computation happens #
        critical_change && backward_filter!(ws°.P[i])
        #-----------------------------------------------#

        success, ws°.ll[i] = GP.solve_and_ll!(
            ws°.XX[i], ws.WW[i], ws°.P[i], ws.XX[i][1].x[1]
        )
        success || return
    end
end

#=

function eMCMC.set_parameters!(
        ::eMCMC.Proposal,
        updt::RandomWalkUpdate,
        global_ws::DiffusionGlobalWorkspace,
        ws::DiffusionLocalWorkspace,
    )
    OBS.set_parameters!(
        global_ws.sub_ws_diff°.P,
        global_ws.data,
        eMCMC.coords(updt),
        eMCMC.invcoords(updt),
        state°(ws)
    )
end

function eMCMC.set_parameters!(
        ::eMCMC.Previous,
        updt::RandomWalkUpdate,
        global_ws::DiffusionGlobalWorkspace,
        ws::DiffusionLocalWorkspace,
    )
    OBS.set_parameters!(
        global_ws.sub_ws_diff.P,
        global_ws.data,
        eMCMC.coords(updt),
        eMCMC.invcoords(updt),
        eMCMC.subidx(state(global_ws), updt)
    )
end
=#

#=
function eMCMC.update!()
    num_recordings = length(ws.P)
    for i in 1:num_recordings
        num_intv = length(ws.P[i])
        success, ll° = forward_guide!(ws, update, i)
        ll = path_loglikhd(ws, ws_cached, i) # use cashed values or recompute
        accept_reject!(updt, ws, ws_global, i, ll, ll°)
    end
end
=#

function eMCMC.compute_ll!(
        updt::eMCMC.MCMCParamUpdate,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
    )
    ll°(local_ws, step.mcmciter) .= ll°(local_ws)
end

function log_prior(
        ::eMCMC.Proposal,
        updt::eMCMC.MCMCParamUpdate,
        global_ws::DiffusionGlobalWorkspace,
        ws::DiffusionLocalWorkspace,
        i,
    )
    y° = ws.sub_ws_gp°.XX[i][1].y[1]
    # possibly get rid of the first one
    logpdf(ws.x0_prior, y°) + logpdf(updt.prior, state°(local_ws))
end

function log_prior(
        ::eMCMC.Previous,
        updt::eMCMC.MCMCParamUpdate,
        global_ws::DiffusionGlobalWorkspace,
        ws::DiffusionLocalWorkspace,
        i,
    )
    y = ws.sub_ws_gp.XX[i][1].y[1]
    # possibly get rid of the first one
    logpdf(ws.x0_prior, y) + logpdf(updt.prior, state(local_ws))
end

#̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶
#               Decide what happens at the Accept/Reject step
#̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶̶

function eMCMC.register_accept_reject_results!(
        accepted::Bool,
        updt::eMCMC.MCMCParamUpdate,
        global_ws::DiffusionGlobalWorkspace,
        local_ws::DiffusionLocalWorkspace,
        step,
        ::Any
    )
    accepted && swap_laws_and_paths!(local_ws)
    eMCMC.register_accept_reject_results!(accepted, updt, local_ws, step)
    eMCMC.set_chain_param!(accepted, updt, global_ws, local_ws, step)
end

function eMCMC.register_accept_reject_results!(
        _accepted::Bool,
        updt::eMCMC.MCMCParamUpdate,
        ws::DiffusionLocalWorkspace,
        step,
    )
    register!(updt, _accepted) #TODO
    set_accepted!(ws, step.mcmciter, _accepted)

    # register proposal
    ll°(ws, step.mcmciter)[1] = ll°(ws)[1]

    # update local ll with accepted
    _accepted && ( ll(ws)[1] = ll°(ws)[1] )

    # register accepted
    ll(ws, step.mcmciter)[1] = ll(ws)[1]
end

function swap_laws_and_paths!(ws::DiffusionLocalWorkspace)
    s°, s = ws.sub_ws_diff°, ws.sub_ws_diff
    for i in eachindex(s°.P, s.P)
        swap!(s°.P[i], s.P[i])
        swap!(s°.XX[i], s.XX[i])
    end
end

#===============================================================================
                Starting point imputation via Langevin updates
===============================================================================#

function eMCMC.update!(
        update::StartingPointsLangevinImputation,
        ws::DiffusionLocalWorkspace,
        ws_global::DiffusionGlobalWorkspace,
        step
    )
    nothing
end

#===============================================================================
                            Temporary measures
===============================================================================#
# temporary measure, make sure online stats work for Diffusions before removing
function eMCMC.update!(
        updt::eMCMC.MCMCParamUpdate,
        global_ws::DiffusionGlobalWorkspace,
        ws::DiffusionLocalWorkspace,
        step,
    )
    #step.mcmciter % 1 == 0 && println("$(step.mcmciter)")
    eMCMC.proposal!(updt, global_ws, ws, step) # specific to updt, see `updates.jl`
    eMCMC.set_proposal!(updt, global_ws, ws, step)
    eMCMC.compute_ll!(updt, global_ws, ws, step)
    eMCMC.accept_reject!(updt, global_ws, ws, step)
    #eMCMC.update_stats!(global_ws, ws, step)
end


using Distributed

# */------------------------------------------------------------------------------------------/*

loc_machine = Dict("hostip" => "")

machine_1   = Dict("hostip" => "000.000.000.000",
                   "grblcs" => "path_to_gurobi_license",
                   "jlpath" => "path_to_julia_exec")

# */------------------------------------------------------------------------------------------/*

cluster = Dict("machine0" => loc_machine, "machine1"=> machine_1)

# */------------------------------------------------------------------------------------------/*

ipkeys = collect(keys(cluster))
wpools = Dict{String,Array{Int64,1}}()
for k in ipkeys
    nw = get_wrkr(k,cluster[k],cs)
    if cluster[k]["hostip"] == ""
        nw > 0 ? wp = addprocs(nw;topology=:master_worker) : wp = zeros(Int64,0)
    else
        nw > 0 ? wp = addprocs([(cluster[k]["hostip"],nw)];tunnel=true,topology=:master_worker,exename=cluster[k]["jlpath"]) : wp = zeros(Int64,0)
    end
    wpools[k] = wp
end
# */------------------------------------------------------------------------------------------/*

@everywhere using Pkg
@everywhere using JuMP
@everywhere using Suppressor
@everywhere using ParallelDataTransfer
@everywhere using Gurobi

# */------------------------------------------------------------------------------------------/*

for k in ipkeys
    if cluster[k]["hostip"] != ""
        for w in wpools[k]
            @spawnat(w, Core.eval(Main, Expr(:(=), :grb_lic, cluster[k]["grblcs"])))
            @spawnat w ENV["GRB_LICENSE_FILE"] = grb_lic
            @spawnat w @suppress Pkg.build("Gurobi")
        end
    end
end

# */------------------------------------------------------------------------------------------/*

@everywhere gurobi_env = @suppress Gurobi.Env()
@everywhere Pkg.instantiate()

# */------------------------------------------------------------------------------------------/*


export iauCr
"""
Copy an r-matrix.

This function is part of the International Astronomical Union's
SOFA (Standards Of Fundamental Astronomy) software collection.

Status:  vector/matrix support function.

Given:
   r        double[3][3]    r-matrix to be copied

Returned:
   c        double[3][3]    copy

Called:
   iauCp        copy p-vector

This revision:  2016 May 19

SOFA release 2018-01-30

Copyright (C) 2018 IAU SOFA Board.  See notes at end.
"""
# void iauCr(double r[3][3], double c[3][3])

function iauCr(r::AbstractMatrix{<:Real})

   # Allocate return value
   c = zeros(Float64, 3, 3)

   ccall((:iauCr, libsofa_c), Cvoid,
         (Ptr{Cdouble}, Ptr{Cdouble}),
         convert(Array{Float64, 2}, r'), c)

   return SMatrix{3,3}(c')
end


#!/usr/bin/env julia
include("soscan.jl")
selc7017 = sosread("data/Statistikdatabasen_2019-02-21 18_39_26.csv")
caplot(selc7017, :Kvinnor, 45)
caplot(selc7017, :Män, 45)
caplot(selc7017, :Kvinnor, 65)
caplot(selc7017, :Män, 65)
grid(1)
legend(loc = 3)
xlabel("Kohort")
ylabel("log(incidens)")
title("Incidens lungcancer Sverige")
savefig("../../images/SvLc7017CohAge.svg")


"""
    readsiggen(file_path::String[, T::Type=Float64])
Read the '*.config' file in 'file_path' for SigGen and returns a dictionary of all parameters.
Non-existing parameteres are set to 0.
...
# Arguments
- `file_path::String`: file path for the SigGen config file.
- `T::Type=Float64`: type of the parameters in the output dictionary.
...
"""
function readsiggen(file_path::String; T::Type=Float64)
    #detector_name = split(basename(file_path), ".config")[1]
    #In order to use the plotting recipe:
    detector_name = "Public Inverted Coax"

    config = Dict("name"       => detector_name,
        "verbosity_level"      => 0,
        "xtal_length"          => 0,
        "xtal_radius"          => 0,
        "top_bullet_radius"    => 0, # not implemented in later conversions
        "bottom_bullet_radius" => 0, # not implemented in later conversions
        "pc_length"            => 0,
        "pc_radius"            => 0,
        "bulletize_PC"         => 0, # not implemented in later conversions
        "wrap_around_radius"   => 0,
        "ditch_depth"          => 0,
        "ditch_thickness"      => 0,
        "bottom_taper_length"  => 0,
        "hole_length"          => 0,
        "hole_radius"          => 0,
        "outer_taper_length"   => 0,
        "inner_taper_length"   => 0,
        "outer_taper_width"    => 0,
        "inner_taper_width"    => 0,
        "taper_angle"          => 0,
        "taper_length"         => 0,
        "Li_thickness"         => 0,
        "energy"               => 0,
        "xtal_grid"            => 0,
        "impurity_z0"          => 0,
        "impurity_gradient"    => 0,
        "impurity_surface"     => 0,
        "xtal_HV"              => 0,
        "max_iterations"       => 0,
        "write_field"          => 0,
        "write_WP"             => 0,
        "drift_name"           => "",
        "field_name"           => "",
        "wp_name"              => "",
        "xtal_temp"            => 0,
        "preamp_tau"           => 0,
        "time_steps_calc"      => 0,
        "step_time_calc"       => 0,
        "step_time_out"        => 0,
        "charge_cloud_size"    => 0,
        "use_diffusion"        => 0,
        "surface_drift_vel_factor" => 0);

    file = open(file_path) do f
        temp          = readlines(f)
        config_file   = Dict()
        for line in temp
            if line  != "" && string(line[1]) != "#"
                line  = strip(split(line, "#")[1])
                name  = ""
                value = ""
                if length(split(line, "\t")) > 1
                    name  = split(line, "\t")[1]
                    value = strip(split(line, "\t")[2])
                else
                    name  = split(line, " ")[1]
                    value = strip(split(line, " ")[end])
                end

                config_file[name] = value

            end
        end
        return config_file
    end

    config_keys = keys(config)
    file_keys   = keys(file)

    for key in file_keys
        if key in config_keys
            if key == "wp_name" || key == "drift_name" || key == "field_name"
                config[key] = string(file[key])
            elseif key == "taper_angle" && parse(Float64, file["taper_angle"]) != 0
                config["outer_taper_width"] = tan(2*pi*parse(T, file[key])/360)*parse(T, file["outer_taper_length"])
                config[key]                 = parse(T, file[key])

            else
                config[key] = parse(T, file[key])
            end
        end

    end
    return config
end

"""
    siggentodict(config::Dict[, units::Dict, detector_name::String])
Converts the dictionary containing the parameters from a SigGen config file
to a SSD config dictionary. This dictionary can be saved as a JSON file using
the JSON package and 'JSON.print(file, config, 4)'.
The 'detector_name' is set to "Public Inverted Coax" by default to inherit
the colour scheme.
...
# Arguments
- `config::Dict`: dictionary containing SigGen parameters (output of readsiggen()).
- `units::Dict`: units used in SigGen file (set to 'mm', 'deg', 'V' and 'K').
- `detector_name::String`: name of the detector.
...
"""
function siggentodict(config::Dict;
        units::Dict = Dict("length" => "mm","angle" => "deg","potential" => "V","temperature" => "K"))

    #>----------------Settings START-------------------------------------------<

    #>----------------Setting the z and r limits-------------------------------<
    z_limit      = ceil(config["xtal_length"]; sigdigits=1);
    if z_limit - config["xtal_length"] <= 0
        z_limit += 10
    end
    r_limit      = ceil(config["xtal_radius"]; sigdigits=1);
    if r_limit - config["xtal_radius"] <= 0
        r_limit += 10
    end

    #>----------------Setting the Axes & Grid dictionaries---------------------<
    axes         = Dict("r"           => Dict("to"         => r_limit,
                                              "boundaries" => "inf"),
                        "phi"         => Dict("from"       => 0,
                                              "to"         => 0,
                                              "boundaries" => "periodic"),
                        "z"           => Dict("from"       => -10,
                                              "to"         => z_limit,
                                              "boundaries" => Dict("left" => "inf", "right" => "inf"))
                        )

    grid         = Dict("coordinates" => "cylindrical",
                        "axes"        => axes);

    #>----------------Settings END---------------------------------------------<
    #<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
    #>----------------Volume START---------------------------------------------<

    #>----------------Main Volume -> Initial Cylinder--------------------------<
    # r and L are given by the input file. Cylindrical symmetry is set to default
    main_vol     = Dict("name"        => "Initial Cylinder",
                        "type"        => "tube",
                        "r"           => Dict("from" => 0.0, "to" => config["xtal_radius"]),
                        "phi"         => Dict("from" => 0.0, "to" => 360.0),
                        "h"           => config["xtal_length"]);

    #>----------------Upper Cone / Taper on top--------------------------------<
    # If not existing, everything will be set to 0
    upper_cone   = Dict("name"        => "Upper Cone",
                        "type"        => "cone",
                        "r"           => Dict("bottom" => Dict("from" => config["xtal_radius"],
                                                               "to"   => ceil(config["xtal_radius"] + 1.0)),
                                              "top"    => Dict("from" => config["xtal_radius"] -
                                                                            config["outer_taper_width"],
                                                               "to"   => ceil(config["xtal_radius"] + 1.0))),
                        "phi"         => Dict("from"   => 0.0, "to" => 360.0),
                        "h"           => config["outer_taper_length"],
                        "translate"   => Dict("z"      => config["xtal_length"] -
                                                                config["outer_taper_length"]));

    #>----------------Lower Cone / Taper on bottom-----------------------------<
    # 45-degree taper at bottom of ORTEC-type crystal
    lower_cone   = Dict("name"        => "Lower Cone",
                        "type"        => "cone",
                        "r"           => Dict("top"    => Dict("from" => config["xtal_radius"],
                                                               "to"   => ceil(config["xtal_radius"] + 1.0)),
                                              "bottom" => Dict("from" => config["xtal_radius"] -
                                                                            config["taper_length"],
                                                               "to"   => ceil(config["xtal_radius"] + 1.0))),
                        "phi"         => Dict("from"   => 0.0, "to" => 360.0),
                        "h"           => config["taper_length"],
                        "translate"   => Dict("z"      => 0.0));

    #>----------------Ditch around the bottom contact--------------------------<
    # If not existing, everything will be set to 0
    ditch        = Dict("name"        => "Ditch",
                        "type"        => "tube",
                        "r"           => Dict("from" => config["wrap_around_radius"] -
                                                            config["ditch_thickness"],
                                              "to"   => config["wrap_around_radius"]),
                        "phi"         => Dict("from" => 0.0, "to" => 360.0),
                        "h"           => config["ditch_depth"]);

    #>----------------Borehole on bottom as contact----------------------------<
    # If not existing, everything will be set to 0
    borehole     = Dict("name"        => "Borehole",
                        "type"        => "tube",
                        "r"           => Dict("from" => 0.0, "to" => config["pc_radius"]),
                        "phi"         => Dict("from" => 0.0, "to" => 360.0),
                        "h"           => config["pc_length"]);

    #>----------------Hole on top of the detector------------------------------<
    # If not existing, everything will be set to 0
    hole = Dict()
    if config["inner_taper_width"] != 0 && config["inner_taper_length"] != 0
        hole         = Dict("name"        => "Inner Cone",
                            "type"        => "cone",
                            "r"           => Dict("top"    => Dict("from" => 0,
                                                                   "to"   => config["hole_radius"] +
                                                                                config["inner_taper_width"]),
                                                  "bottom" => Dict("from" => 0,
                                                                   "to"   => config["hole_radius"])),
                            "phi"         => Dict("from"   => 0.0, "to" => 360.0),
                            "h"           => config["inner_taper_length"],
                            "translate"   => Dict("z"      => config["xtal_length"] -
                                                                config["inner_taper_length"]));
    else
        hole         = Dict("name"        => "Top Hole",
                            "type"        => "tube",
                            "r"           => Dict("from" => 0.0, "to" => config["hole_radius"]),
                            "phi"         => Dict("from" => 0.0, "to" => 360.0),
                            "h"           => config["hole_length"],
                            "translate"   => Dict("z"    => config["xtal_length"] -
                                                            config["hole_length"]));
    end

    #>----------------Subtract volumes-----------------------------------------<
    geometry = main_vol;
    if config["outer_taper_length"] != 0 || config["outer_taper_width"] != 0
        geometry = Dict("type" => "difference", "parts" => [geometry, upper_cone])
    end
    if config["inner_taper_length"] != 0 || config["inner_taper_width"] != 0 || config["hole_radius"] != 0 || config["hole_length"] != 0
        geometry = Dict("type" => "difference", "parts" => [geometry, hole])
    end
    if config["taper_length"] != 0
        geometry = Dict("type" => "difference", "parts" => [geometry, lower_cone])
    end
    if config["ditch_thickness"] != 0
        geometry = Dict("type" => "difference", "parts" => [geometry, ditch])
    end
    if config["pc_radius"] != 0 && config["pc_length"] >= 0.1 # below this, it is the thickness of the layer
        geometry = Dict("type" => "difference", "parts" => [geometry, borehole])
    end

    #>----------------Volume END-----------------------------------------------<
    #<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
    #>----------------Contacts START-------------------------------------------<

    geometry_1   = Any[] # gathers all parts of contact_1

    #>----------------Area/volume of point contact-----------------------------<
    #>----------------Contact 1------------------------------------------------<
    # Depending on whether the contact is a borehole or not
    if config["pc_radius"] != 0 && config["pc_length"] >= 5 # below this, it is the thickness of the layer
        borehole_wall = Dict(    "type" => "tube",
             "r"         => Dict("from" => config["pc_radius"],
                                   "to" => config["pc_radius"]),
             "phi"       => Dict("from" => 0.0,
                                   "to" => 360.0),
             "h"         => config["pc_length"])
        borehole_top = Dict(     "type" => "tube",
             "r"         => Dict("from" => 0.0,
                                   "to" => config["pc_radius"]),
             "phi"       => Dict("from" => 0.0,
                                   "to" => 360.0),
             "h"         => 0.0,
             "translate" => Dict("z"    => config["pc_length"]))
        borehole_around = Dict(  "type" => "tube",
             "r"         => Dict("from" => config["pc_radius"],
                                   "to" => config["wrap_around_radius"] - config["ditch_thickness"]),
             "phi"       => Dict("from" => 0.0,
                                   "to" => 360.0),
             "h"         => 0.0,
             "translate" => Dict("z"    => 0.0))
        push!(geometry_1, borehole_wall)
        push!(geometry_1, borehole_top)
        push!(geometry_1, borehole_around)

    elseif config["pc_radius"] != 0 && config["pc_length"] < 5
        pc_vol = Dict(           "type" => "tube",
             "r"         => Dict("from" => 0.0,
                                   "to" => config["pc_radius"]),
             "phi"       => Dict("from" => 0.0,
                                   "to" => 360.0),
             "h"         => config["pc_length"])
        push!(geometry_1, pc_vol)
    end

    #>----------------Define contact 1-----------------------------------------<

    contact_1 = Dict("type"        => "contact",
                     "material"    => "HPGe",
                     "channel"     => 1,
                     "potential"   => 0.0,
                     "geometry"    => Dict("type"  => "union",
                                           "parts" => geometry_1));

    #>----------------Contact 2------------------------------------------------<
    geometry_2 = Any[]

    if config["wrap_around_radius"] != 0.0
        bottom = Dict("type" => "tube",
                      "r"    => Dict("from" => config["wrap_around_radius"],
                                       "to" => config["xtal_radius"] - config["taper_length"]),
                      "phi"  => Dict("from" => 0.0,
                                       "to" => 360.0),
                      "h"    => 0.0)
        push!(geometry_2, bottom)
    end

    wall   = Dict("type" => "tube",
                  "r"    => Dict("from" => config["xtal_radius"],
                                 "to"   => config["xtal_radius"]),
                  "phi"  => Dict("from" => 0.0,
                                   "to" => 360.0),
                  "h"    => config["xtal_length"] - config["taper_length"] - config["outer_taper_length"],
                  "translate" => Dict("z" => config["taper_length"]))
    push!(geometry_2, wall)

    top    = Dict("type" => "tube",
                  "r"    => Dict("from" => config["hole_radius"] + config["inner_taper_width"],
                                   "to" => config["xtal_radius"] - config["outer_taper_width"]),
                  "phi"  => Dict("from" => 0.0,
                                   "to" => 360.0),
                  "h"    => 0.0,
                  "translate" => Dict("z" => config["xtal_length"]))
    push!(geometry_2, top)
    if config["hole_radius"] != 0 && config["hole_length"] != 0 && config["inner_taper_width"] == 0 && config["inner_taper_length"] == 0
        hole   = Dict("type" => "tube",
                      "r"    => Dict("from" => config["hole_radius"],
                                       "to" => config["hole_radius"]),
                      "phi"  => Dict("from" => 0.0,
                                       "to" => 360.0),
                      "h"    => config["hole_length"],
                      "translate" => Dict("z" => config["xtal_length"] - config["hole_length"]))
        inner_hole_bottom    =  Dict("type" => "tube",
                      "r"    => Dict("from" => 0.0,
                                       "to" => config["hole_radius"]),
                      "phi"  => Dict("from" => 0.0,
                                       "to" => 360.0),
                      "h"    => 0.0,
                      "translate" => Dict("z" => config["xtal_length"] - config["hole_length"]))
        push!(geometry_2, hole)
        push!(geometry_2, inner_hole_bottom)
    end
    if config["hole_radius"] != 0 && config["hole_length"] != 0 && config["inner_taper_width"] != 0 && config["inner_taper_length"] != 0
        inner_taper = Dict("type" => "cone",
                      "r"         => Dict("top"    => Dict("from" => config["hole_radius"] +
                                                                        config["inner_taper_width"],
                                                           "to"   => config["hole_radius"] +
                                                                        config["inner_taper_width"]),
                                          "bottom" => Dict("from" => config["hole_radius"],
                                                           "to"   => config["hole_radius"])),
                      "phi"       => Dict("from"   => 0.0, "to" => 360.0),
                      "h"         => config["inner_taper_length"],
                      "translate" => Dict("z"      => config["xtal_length"] -
                                                        config["inner_taper_length"]));
        inner_hole_bottom   =  Dict("type" => "tube",
                      "r"    => Dict("from" => 0.0,
                                       "to" => config["hole_radius"]),
                      "phi"  => Dict("from" => 0.0,
                                       "to" => 360.0),
                      "h"    => 0.0,
                      "translate" => Dict("z" => config["xtal_length"] - config["hole_length"]))
        push!(geometry_2, inner_taper)
        push!(geometry_2, inner_hole_bottom)
    end
    if config["outer_taper_width"] != 0 && config["outer_taper_length"] != 0
        outer_taper = Dict("type" => "cone",
                      "r"         => Dict("top"    => Dict("from" => config["xtal_radius"] -
                                                                        config["outer_taper_width"],
                                                           "to"   => config["xtal_radius"] -
                                                                        config["outer_taper_width"]),
                                          "bottom" => Dict("from" => config["xtal_radius"],
                                                           "to"   => config["xtal_radius"])),
                      "phi"       => Dict("from"   => 0.0, "to" => 360.0),
                      "h"         => config["outer_taper_length"],
                      "translate" => Dict("z"      => config["xtal_length"] -
                                                        config["outer_taper_length"]));
        push!(geometry_2, outer_taper)
    end
    if config["taper_length"] != 0
        taper = Dict("type"       => "cone",
                      "r"         => Dict("top"    => Dict("from" => config["xtal_radius"],
                                                           "to"   => config["xtal_radius"]),
                                          "bottom" => Dict("from" => config["xtal_radius"] -
                                                                        config["taper_length"],
                                                           "to"   => config["xtal_radius"] -
                                                                        config["taper_length"])),
                      "phi"       => Dict("from"   => 0.0, "to" => 360.0),
                      "h"         => config["taper_length"]);
        push!(geometry_2, taper)
    end

    #>----------------Define contact 2-----------------------------------------<

    contact_2 = Dict("type"        => "contact",
                     "material"    => "HPGe",
                     "channel"     => 2,
                     "potential"   => config["xtal_HV"],
                     "geometry"    => Dict("type"  => "union",
                                           "parts" => geometry_2));

    #>----------------Contacts END---------------------------------------------<
    #<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><
    #>----------------Final configuration--------------------------------------<

    impurity_z0            = config["impurity_z0"]*1e10
    impurity_gradient      = config["impurity_gradient"]*1e10
    if units["length"]     == "mm"
        impurity_z0        /= 1e3
        impurity_gradient  /= 1e4
    elseif units["length"] == "m"
        impurity_z0        *= 1e6
        impurity_gradient  *= 1e8
    end

    objects = Any[Dict("type"        => "semiconductor",
                    "material"    => "HPGe",
                    "bulk_type"   => "p",
                    "temperature" => config["xtal_temp"],
                    "charge_density_model" => Dict(
                                "name"     => "linear",
                                "r"        => Dict("init"     => 0.0,
                                                   "gradient" => 0.0),
                                "phi"      => Dict("init"     => 0.0,
                                                   "gradient" => 0.0),
                                "z"        => Dict("init"     => impurity_z0,
                                                   "gradient" => impurity_gradient)
                            ),
                    "geometry"    => geometry
                    ),
                    contact_1,
                    contact_2];

    conf = Dict();
    conf = Dict("name"    => config["name"],
                "units"   => units,
                "grid"    => grid,
                "medium"  => "vacuum",
                "objects" => objects);

    return conf
end


"""
Describes a dam purpose, such as providing drinking water (i.e. water withdrawn from the reservoir), or ensuring inland
navigation (i.e. constraints on the reservoir level behind the dam).

Each purpose must have a unique identifier, whose type is `Symbol`. It is always available with the `getId()` function.
For implementers, a default implementation uses the field `id`.
"""
abstract Purpose

getId(p::Purpose) = p.id
isDeterministic(p::Purpose) = false

id(p::Purpose) = getId(p)

"""
Described a dam purpose that is fulfilled by taking water from the reservoir, such as providing drinking water.

The given discharges are expressed as 10^6 m^3/s.
"""
abstract WaterWithdrawalPurpose <: Purpose

isWaterWithdrawal(p::Purpose) = false
isWaterWithdrawal(p::WaterWithdrawalPurpose) = true

"""
Describes a deterministic dam purpose, without any uncertainty, which only withdraws water.

Each such purpose corresponds to a given needed discharge. It is always available with the `getNeed()` function.
For implementers, a default implementation uses the field `need`. The given discharges are expressed as 10^6 m^3/s.

The rationale behind this layer of abstraction (instead of making it concrete, with the various purposes having
a different id) is that some purposes may have multiple instances for the same dam. For example, a conveyance network
may extract water from multiple dams for a given quantity of water to reach; and the same dam may participate in multiple
such summations.
"""
abstract DeterministicWaterWithdrawalPurpose <: WaterWithdrawalPurpose

getNeed(d::DeterministicWaterWithdrawalPurpose) = d.need
getNeed(d::DeterministicWaterWithdrawalPurpose, p::Period) = getNeed(d) * Dates.days(p) * 86400.
isDeterministic(d::DeterministicWaterWithdrawalPurpose) = true

need(d::DeterministicWaterWithdrawalPurpose) = getNeed(d)
need(d::DeterministicWaterWithdrawalPurpose, p::Period) = getNeed(d, p)

"""
Represents a deterministic drinking water consumption from the reservoir. It can only be fulfilled by one reservoir.

By default, the id is `:drinkingWater`.
"""
immutable DeterministicDrinkingWater <: DeterministicWaterWithdrawalPurpose
  id::Symbol
  need::Float64 # 10^6 m^3/s
  DeterministicDrinkingWater(v::Discharge) = new(:drinkingWater, _from_unitful(v))
  DeterministicDrinkingWater(id::Symbol, v::Discharge) = new(id, _from_unitful(v))
end

"""
Represents a deterministic environmental flow that must be released from the reservoir into its river.

By default, the id is `:environmentalFlow`.
"""
immutable DeterministicEnvironmentalFlow <: DeterministicWaterWithdrawalPurpose
  id::Symbol
  need::Float64 # 10^6 m^3/s
  DeterministicEnvironmentalFlow(v::Discharge) = new(:environmentalFlow, _from_unitful(v))
  DeterministicEnvironmentalFlow(id::Symbol, v::Discharge) = new(id, _from_unitful(v))
end

"""
Represents a hydropower unit, which has no need for water as the exact discharge is a degree of freedom.

The id is computed with that of the hydropower unit as `getPurposeId(hp)`.

This object is automatically created when creating a `Reservoir` data structure, and thus does not need to be used
outside internal code.
"""
immutable DeterministicHydropower <: DeterministicWaterWithdrawalPurpose
  hp::HydropowerUnit
end

getId(hp::DeterministicHydropower) = getPurposeId(hp.hp)
getNeed(hp::DeterministicHydropower) = 0.
getHydropowerUnit(hp::DeterministicHydropower) = hp.hp
isHydropower(p::Purpose) = false
isHydropower(p::DeterministicHydropower) = true

hydropower(hp::DeterministicHydropower) = hp.hp

"""
Builds an array of deterministic purposes for the most common cases, i.e. one `DeterministicDrinkingWater`
and one `DeterministicEnvironmentalFlow` at most.
"""
function DeterministicPurposes(;drinkingWater::Discharge=0.m^3/s, environmentalFlow::Discharge=0.m^3/s)
  list = Purpose[] # All data structures expect a generic array of purposes, be they deterministic or not.
  if drinkingWater != 0.m^3/s
    push!(list, DeterministicDrinkingWater(drinkingWater))
  end
  if environmentalFlow != 0.m^3/s
    push!(list, DeterministicEnvironmentalFlow(environmentalFlow))
  end
  return list
end


#InspectDR: Store/manipulate info about graph bounds, scales, ...
#-------------------------------------------------------------------------------

#TODO: Move Plot2D here from base.jl, and rename?

#==Main types
===============================================================================#

#TODO: deprecate Plot2DInfo & stick with Graph2DInfo (despite redundancy)???
#Maybe Plot2DInfo just stores array of Graph2DInfo?

#"Evaluated" information about all strips:
#(Cache for bounds, current extents, text formatting, transforms, etc)
mutable struct StripInfo
	graphbb::BoundingBox #Location of graph (device units)
	ext::PExtents2D #Current extents of graph
	yfmt::TickLabelFormatting #y tick label formatting
	yixf::InputXfrm1DSpec #Input transform for y-values
	xf::Transform2D #Transform used to render data
	grid::PlotGrid #Real, displayed ("_eval"-uated) grid - not "coordinate grid"
	#TODO: axis scaling, ...?
end

#"Evaluated" information about all graphs on a plot:
#(Cache for bounds, current extents, text formatting, transforms, etc)
mutable struct Plot2DInfo
#TODO: store BoundingBox for entire plot??
#TODO: store extents???
	xfmt::TickLabelFormatting #x tick label formatting
	xixf::InputXfrm1DSpec #Input transform for x-values
	strips::Vector{StripInfo}
end
Plot2DInfo() = Plot2DInfo(
	TickLabelFormatting(NoRangeDisplayInfo()), InputXfrm1DSpec(:lin), []
)

#Graph-specific version of Plot2DInfo/StripInfo
mutable struct Graph2DInfo
	graphbb::BoundingBox #Location of graph (device units)
	ext::PExtents2D #Current extents of graph

	xfmt::TickLabelFormatting #x tick label formatting
	yfmt::TickLabelFormatting #y tick label formatting

	ixf::InputXfrm2D #Input transform for y-values
	xf::Transform2D #Transform used to render data

	grid::PlotGrid #Real, displayed ("_eval"-uated) grid - not "coordinate grid"
end

#==Accessors
===============================================================================#
Transform2D(pinfo::Plot2DInfo, istrip::Int) = pinfo.strips[istrip].xf
InputXfrm2D(pinfo::Plot2DInfo, istrip::Int) =
	InputXfrm2D(pinfo.xixf, pinfo.strips[istrip].yixf)

#==Constructor-like functions
===============================================================================#
function Graph2DInfo(pinfo::Plot2DInfo, istrip::Int)
	graphbb = pinfo.strips[istrip].graphbb
	ext = pinfo.strips[istrip].ext
	xfmt = pinfo.xfmt
	yfmt = pinfo.strips[istrip].yfmt
	ixf = InputXfrm2D(pinfo, istrip)
	xf = Transform2D(pinfo, istrip)
	grid = pinfo.strips[istrip].grid
	return Graph2DInfo(graphbb, ext, xfmt, yfmt, ixf, xf, grid)
end

function Plot2DInfo(plot::Plot2D)
	xfmt = TickLabelFormatting(NoRangeDisplayInfo())
	if length(plot.strips) > 0
		istrip = 1
		strip = plot.strips[istrip]
		ext = getextents_axis(plot, istrip)
		#Get the rectangular coordinate grid:
		grid = coord_grid(strip.grid, plot.xscale, strip.yscale, ext)
		if !isa(grid.xlines, UndefinedGridLines)
			xfmt = TickLabelFormatting(plot.layout.values.format_xtick, grid.xlines.rnginfo)
		end
	end
	return Plot2DInfo(xfmt, InputXfrm1DSpec(plot.xscale), [])
end

#bb: bounding box of entire plot
function Plot2DInfo(plot::Plot2D, bb::BoundingBox)
	dfltfmt = TickLabelFormatting(NoRangeDisplayInfo()) #WANTCONST
	result = Plot2DInfo(plot)
	databb = databounds(bb, plot.layout.values, grid1(plot))
	nstrips = length(plot.strips)
	graphbblist = graphbounds_list(databb, plot.layout.values, nstrips)
	resize!(result.strips, nstrips)

	for istrip in 1:nstrips #Compute x/y label formats:
		strip = plot.strips[istrip]
		graphbb = graphbblist[istrip]
		ext = getextents_axis(plot, istrip)
		grid = _eval(strip.grid, plot.xscale, strip.yscale, ext)
		cgrid = coord_grid(strip.grid, plot.xscale, strip.yscale, ext)
		yfmt = dfltfmt
		if !isa(cgrid.ylines, UndefinedGridLines)
			yfmt = TickLabelFormatting(plot.layout.values.format_ytick, cgrid.ylines.rnginfo)
		end
		result.strips[istrip] = StripInfo(graphbb, ext, yfmt,
			InputXfrm1DSpec(strip.yscale), Transform2D(ext, graphbb), grid
		)
	end
	return result
end
#Want to define/use coord_grid when displaying coordinates???
#Or maybe we deprecate coord_grid???

#==Helper functions
===============================================================================#
#Get numeric formatting/precision with mouse hover:
function hoverfmt(fmt::TickLabelFormatting)
	result = deepcopy(fmt.fmt)
	#Display a bit more precision than tick labels:
	result.ndigits += 2 #TODO: Better algorithm?
	return result
end

#==Main functions
===============================================================================#
#TODO: move some of the computations from base.jl here?

#Last line


using FileIO, GeometryTypes
using Test
const tf = joinpath(dirname(@__FILE__), "testfiles")

function test_face_indices(mesh)
    for face in faces(mesh)
        for index in face
            @test firstindex(vertices(mesh)) <= index <= lastindex(vertices(mesh))
        end
    end
end

@testset "MeshIO" begin
    dirlen = 1f0
    baselen = 0.02f0
    mesh = [
        AABB{Float32}(Vec3f0(baselen), Vec3f0(dirlen, baselen, baselen)),
        AABB{Float32}(Vec3f0(baselen), Vec3f0(baselen, dirlen, baselen)),
        AABB{Float32}(Vec3f0(baselen), Vec3f0(baselen, baselen, dirlen))
    ]
    mesh = merge(map(GLPlainMesh, mesh))
    mktempdir() do tmpdir
        for ext in ["2dm", "off"]
            @testset "load save $ext" begin
                save(joinpath(tmpdir, "test.$ext"), mesh)
                mesh_loaded = load(joinpath(tmpdir, "test.$ext"), GLPlainMesh)
                @test mesh_loaded == mesh
            end
        end
        @testset "PLY ascii and binary" begin
            f = File(format"PLY_ASCII", joinpath(tmpdir, "test.ply"))
            save(f, mesh)
            mesh_loaded = load(joinpath(tmpdir, "test.ply"), GLPlainMesh)
            @test mesh_loaded == mesh
            save(File(format"PLY_BINARY", joinpath(tmpdir, "test.ply")), mesh)
        end
        @testset "STL ascii and binary" begin
            save(File(format"STL_ASCII", joinpath(tmpdir, "test.stl")), mesh)
            mesh_loaded = load(joinpath(tmpdir, "test.stl"), GLPlainMesh)
            #@test mesh_loaded == mesh
            save(File(format"STL_BINARY", joinpath(tmpdir, "test.stl")), mesh)
            mesh_loaded = load(joinpath(tmpdir, "test.stl"), GLNormalMesh)
            #@test mesh_loaded == mesh
            mesh_loaded = load(joinpath(tmpdir, "test.stl"), GLPlainMesh)
        end
    end
    @testset "Real world files" begin

        @testset "STL" begin
            msh = load(joinpath(tf, "ascii.stl"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 12
            @test length(vertices(msh)) == 36
            @test length(normals(msh)) == 36
            test_face_indices(msh)

            msh = load(joinpath(tf, "binary.stl"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 828
            @test length(vertices(msh)) == 2484
            @test length(normals(msh)) == 2484
            test_face_indices(msh)

            mktempdir() do tmpdir
                save(File(format"STL_BINARY", joinpath(tmpdir, "test.stl")), msh)
                msh1 = load(joinpath(tmpdir, "test.stl"))
                @test typeof(msh1) == GLNormalMesh
                @test faces(msh) == faces(msh1)
                @test vertices(msh) == vertices(msh1)
                @test normals(msh) == normals(msh1)
            end

            msh = load(joinpath(tf, "binary_stl_from_solidworks.STL"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 12
            @test length(vertices(msh)) == 36
            test_face_indices(msh)

            # STL Import
            msh = load(joinpath(tf, "cube_binary.stl"))
            @test length(vertices(msh)) == 36
            @test length(faces(msh)) == 12
            test_face_indices(msh)

            msh = load(joinpath(tf, "cube.stl"))
            @test length(vertices(msh)) == 36
            @test length(faces(msh)) == 12
            test_face_indices(msh)

        end
        @testset "PLY" begin
            msh = load(joinpath(tf, "ascii.ply"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 36
            test_face_indices(msh)
            @test length(vertices(msh)) == 72
            @test length(normals(msh)) == 72
            #msh = load(joinpath(tf, "binary.ply")) # still missing
            #@test typeof(msh) == GLNormalMesh
            #println(msh)
            msh = load(joinpath(tf, "cube.ply")) # quads
            @test length(vertices(msh)) == 24
            @test length(faces(msh)) == 12
            test_face_indices(msh)
        end
        @testset "OFF" begin
            msh = load(joinpath(tf, "test.off"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 28
            @test length(vertices(msh)) == 20
            @test length(normals(msh)) == 20
            test_face_indices(msh)

            msh = load(joinpath(tf, "test2.off"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 810
            @test length(vertices(msh)) == 405
            @test length(normals(msh)) == 405
            test_face_indices(msh)

            msh = load(joinpath(tf, "cube.off"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 12
            @test length(vertices(msh)) == 8
            test_face_indices(msh)

        end
        @testset "OBJ" begin
            msh = load(joinpath(tf, "test.obj"))
            @test typeof(msh) == GLNormalMesh
            @test length(faces(msh)) == 3954
            @test length(vertices(msh)) == 2248
            @test length(normals(msh)) == 2248
            test_face_indices(msh)

            msh = load(joinpath(tf, "cube.obj")) # quads
            @test length(faces(msh)) == 12
            @test length(vertices(msh)) == 8
            test_face_indices(msh)

            msh = load(joinpath(tf, "polygonal_face.obj"))
            @test length(faces(msh)) == 4
            @test length(vertices(msh)) == 6
            test_face_indices(msh)

            msh = load(joinpath(tf, "test_face_normal.obj"))
            @test length(faces(msh)) == 1
            @test length(vertices(msh)) == 3
            test_face_indices(msh)

        end
        @testset "2DM" begin
            msh = load(joinpath(tf, "test.2dm"))
            @test typeof(msh) == GLNormalMesh
            test_face_indices(msh)
            #@test length(faces(msh)) == 3954
            #@test length(vertices(msh)) == 2248
            #@test length(normals(msh)) == 2248
        end
    end
end

if false
    #using GLVisualize, GLAbstraction, FileIO
    w = glscreen()
    meshes = [visualize(load(joinpath(tf, name))) for name in readdir(tf)];
    meshes = convert(Matrix{Context}, reshape(meshes, (7, 2)));
    view(visualize(meshes))
    renderloop(w)
end
#=

amf1 = mesh(data_path*"pyramid.amf")
@test length(amf1[1].vertices) == 5
@test length(amf1[1].faces) == 4
@test length(amf1[2].vertices) == 5
@test length(amf1[2].faces) == 4

amf1 = mesh(data_path*"pyramid_zip.amf")
@test length(amf1[1].vertices) == 5
@test length(amf1[1].faces) == 4
@test length(amf1[2].vertices) == 5
@test length(amf1[2].faces) == 4
=#