#   Unit "hilbert" of the FourierAnalysis Package for julia language
#
#   MIT License
#   Copyright (c) 2019-2022,
#   Marco Congedo, CNRS, Grenobe, France:
#   https://sites.google.com/site/marcocongedo/home

# ? CONTENTS :
#   This unit implements the Hilbert transform and analytic signal
#   estimations for signals of arbitrary length, using FFTW.

"""
```julia
   (1)
   function analyticsignal( X  :: Union{Vector{T}, Matrix{T}},
                            wl :: Int     = size(X, 1);
                        nonlinear :: Bool    =  false,
                        planner   :: Planner =  getplanner,
                        ⏩       :: Bool    =  true) where T<:Real

   (2)
   function analyticsignal( 𝐗      :: Vector{Matrix{T}},
                            wl     :: Int;
                        nonlinear  ::  Bool      =  false,
                        planner    ::  Planner   =  getplanner,
                        ⏩        ::  Bool      =  true) where T<:Real
```

(1)

Compute the analytic signal(AS) of vector `X` or of
all column vectors of matrix `X` via the FFT and iFFT procedure,
as explained in Marple(1999).
If `wl`=size(X, 1) (default), use the standard method passing to the FFT
and iFFT all samples in `X` altogether, whereas if `wl`<size(X, 1) a
sliding-windows method is used (see below).

Return the analytic signal `𝑌`, a complex vector if `X` is a vector
or a complex matrix holding in its columns the analytic signal of the
columns of `X` if `X` is a matrix. `𝑌` has the same number of samples (rows)
as `X`.

Contrarely to what is done in the [DSP](https://github.com/JuliaDSP/DSP.jl)
package, the DC level of the signal is removed, therefore,
if the input signal features a non-zero DC level,
the real part of the AS will be equal to the input signal with the
DC level removed. The imaginary part of `𝑌` is the Hilbert transform
of such no-DC `X`.

The sliding-windows AS allows an efficient estimation of the AS for vectors and
matrices when they are very large; it proceeds
computing the AS on 50% sliding overlapping windows and forming the AS
by retaining the central half of each window. The number of points effectively
used to obtain the final estimation is ``wl``÷2 (integer division).
`wl` must be even for using this estimation. This procedure
produces edge effects, thus the first and last ``wl÷4`` samples of the AS
estimation should be discarded. In practice, one requires the AS of a
larger data segment and trims at least ``wl÷4`` samples at the beginning and
end of the estimation. This is done automatically
by the [`TFanalyticsignal`](@ref) function.

Also, the sliding-windows AS method creates small discontinuities at sample
``wl``÷4 and then every ``wl``÷2 samples,
therefore ``wl`` should be chosen as large as possible.

!!! note "Nota Bene"

    In order to avoid FFT computation of very long epochs,
    if `wl` > 2^14, then `wl` is set to 2^10. Below this limit, as long as
    the computations are feasable, use the standard method. If you absolutely
    need to use the sliding-windows method, see [window length in FFTW](@ref)
    for setting efficiently argument `wl`.

    The input signal should be previously band-pass or high-pass filtered
    so as not to contain frequency components below the first discrete
    Fourier frequency obtained with windows of `wl` samples, that is,
    below sr/wl Hz, where sr is the sampling rate.

**Optional Keyword Arguments**:

`nonlinear`, if true, the analytic signal is normalized so that its amplitude
is ``1.0`` at all points. This allow non-linear univariate and bivariate estimations
(see [timefrequencyuni.jl](@ref) and [timefrequencybi.jl](@ref)).

`planner` is an instance of the [`Planner`](@ref) object, holding the forward
and backward FFTW plans used to compute the FFTs and the iFFTs.
By default the planner is computed, but it can be passed as an
argumet here if it is pre-computed. This is interesting if the
`analyticsignal` function is to be invoked repeatedly.

if `⏩` is true, the method is run in multi-threaded mode across the series
in `X` if the number of series is at least twice the number of threads Julia
is instructed to use. See [Threads](@ref).

(2)

Compute the analytic signal for all ``k``
multivariate data matrices given as a vector of matrices `𝐗`.
Return a vector of matrices hodling the corresponding analytic signals
as in method (1). The FFT and iFFT plans are computed only once.
The ``k`` matrices in `𝐗` may have different number of columns (i.e., different
number of series) and different number of rows (samples).
However, the number of rows must be larger than `wl` for all of them.

If `⏩` is true, this method run in multi-threaded mode across the
matrices in `𝐗` if the number of matrices is at least twice
the number of threads Julia is instructed to use, otherwise it
tries to run each analytic signal estimation in multi-threaded mode
as per method (1). See [Threads](@ref).

This function is called by the following functions operating
on time-frequency reprsentations: [`TFanalyticsignal`](@ref),
[`TFamplitude`](@ref), [`TFphase`](@ref), [`meanAmplitude`](@ref),
[`concentration`](@ref), [`meanDirection`](@ref), [`comodulation`](@ref),
[`coherence`](@ref).

**References**
Marple S.L. (1999)
Computing the Discrete-Time Analytic Signal via FFT.
IEEE Transactions on Signal Processing 47(9), 2600-3.

**Examples**:
```julia
using FourierAnalysis, FFTW, LinearAlgebra, Statistics, Plots, DSP
t=128; lab=["x", "real(y)", "imag(y)"]

# Analytic signal of one vector
x=sinusoidal(10, 2, 128, t, π/2; DC=10) # cosine
y=analyticsignal(x)
# note that real(y) is x without the DC level, i.e., x=real(y)+DC
plot([x, real(y), imag(y)]; labels=lab)

# make a check
s=sinusoidal(10, 2, 128, t, 0) # sine
norm(s-imag(y)) # should be zero

# Analytic Signal by DSP.jl
y2=hilbert(x)
norm(s-imag(y2)) # should be zero
# DSP.jl does not remove the DC level
# thus x=real(y2) in this case
plot([x, real(y2), imag(y2)]; labels=lab)

# Analytic signal of multiple vectors
x=hcat(x, sinusoidal(10, 3, 128, t, π/2; DC=10))
y=analyticsignal(x)

# sliding-windows analytic signal of one vector
# (note edge effects)
x=sinusoidal(10, 2, 128, t*4, π/2; DC=0)
y=analyticsignal(x, t)
plot([x, real(y), imag(y)]; labels=lab)

# sliding-windows analytic signal of multiple vectors
x=hcat(x, sinusoidal(10, 3, 128, t*4, π/2; DC=0))
y=analyticsignal(x, t)
```
"""
function analyticsignal( X  :: Union{Vector{T}, Matrix{T}},
                         wl :: Int     = size(X, 1);
                  nonlinear :: Bool    =  false,
                  planner   :: Planner =  getplanner,
                  ⏩       :: Bool    =  true) where T<:Real

   # get all needed paramters (see below the _paramHT! function)
   X_, 𝚙, i𝚙, e, tₐₗₗ, n, wl½, two_wl⁻¹, cT, f, g, ζ = _paramHT!(X, wl, planner)

   𝑌=zeros(cT, tₐₗₗ, n) # Preallocate memory for the Analytic Signal of X

   # Add to `𝑌` the analytic signal of a vector `X_` or of the column vectors of
   # matrix `X_`, computed on the epoch(s) of duration `t` samples starting at
   # sample `at`. It does not return anything.
   # ARGUMENTS:
   # `n` (Int) is the number of columns in `X_`; it must be 1 if `X_` is a vector.
   # `wl½` = wl÷2
   # `two_wl⁻¹` = 2/wl
   # `e` (Int) the number of sliding windows
   # `𝚙` is the forward FFTW plan performing the FFT.
   # `i𝚙` is the backward FFTW plan performing the iFFT (Hilbert transform).
   # `ζ`, =zeros(cT, wl-wl½_), a zero-vector appended to the iFFT vectors, since the second half of the FFT is not computed
   # `f`, the lower limit (in samples) of the central half of each Hilbert transform to cumulate with respect to the FFT window
   # `g`, the upper limit (in samples) of the central half of each Hilbert transform to cumulate with respect to the FFT window
   function _analyticsignal!(at, 𝚗)
      #if 𝚗==1 println(at:at+wl-1, " ", f, " ", g, " ", at+f-1:at+g-1) end ###
      y = 𝚙*X_[at:at+wl-1, 𝚗]        # FFT
      @inbounds begin                # transform
         y[1]=0.
         for i=2:wl½ y[i] *= two_wl⁻¹ end
      end
      # iFFT and cumulate
      e==1 ? 𝑌[:, 𝚗]+=i𝚙*vcat(y, ζ) : 𝑌[at+f-1:at+g-1, 𝚗]+=(i𝚙*vcat(y, ζ))[f:g]
   end

   # add the Analytic Signal
   as!(𝚎, 𝚗)=_analyticsignal!((𝚎*wl½)+1, 𝚗)
   _thread(⏩, n) ? (for 𝚎=0:e-1 @threads for 𝚗=1:n as!(𝚎, 𝚗) end end) : (for 𝚗=1:n, 𝚎=0:e-1 as!(𝚎, 𝚗) end)

   # return a vector if `X` is a vector, a matrix if `X` is a matrix.
   # for the sliding-windows version eliminate the first and last
   # t½ samples that have been previously padded.
   𝑌_ = n==1 ? (e>1 ? 𝑌[:][wl½+1:end-wl½] : 𝑌[:]) : (e>1 ? 𝑌[wl½+1:end-wl½, 1:n] : 𝑌)
   if nonlinear @inbounds for i in eachindex(𝑌_) 𝑌_[i]/=abs(𝑌_[i]) end end
   return 𝑌_
end

function analyticsignal( 𝐗      :: Vector{Matrix{T}},
                         wl     :: Int;
                     nonlinear  ::  Bool      =  false,
                     planner    ::  Planner   =  getplanner,
                     ⏩        ::  Bool      =  true) where T<:Real

   cT, k=typeof(Complex(T(0))), length(𝐗)

   planner ≠ getplanner ? plan=planner : plan=Planner(plan_estimate, -1.0, wl, T, true)

   𝒀=Vector{Matrix{cT}}(undef, k)

   # function to add the analytic signal
   as(i, ⏩)=analyticsignal(𝐗[i], wl; nonlinear=nonlinear, planner=plan, ⏩=⏩)
   _thread(⏩, k) ? (@threads for i=1:k 𝒀[i]=as(i, false) end) : (for i=1:k 𝒀[i]=as(i, ⏩) end)

   return 𝒀
end

# Internal function: prepare all parameters that are needed for
# Analytic Signal and sliding-windows Analytic Signal estimations
#
# ARGUMENTS:
# `X` is the input data vector or matrix (n time-series)
# `wl` (Int) is the epoch length for FFT
# `planner` a pre-computed FFT `Planner` or `getPlanner` to compute it
#
# RETURN
# `X_`: if a standard anaytic signal is requested (t==size(X, 1)) then X_=X
#       else X_ is the data in `X` with wl½ zeros prepended and appended.
# `𝚙` the forward FFTW plan. It is computed if argument `planner`=`getplanner`.
# `i𝚙` the backward FFTW plan. It is computed if argument `planner`=`getplanner`.
# `e` (Int) the number of sliding windows, 1 for the standard AS method.
# `tₐₗₗ` (Int) number of samples total. For sliding-windows anaytic signal,
#     this is not size(X, 1) as zeros are padded.
# `n` (Int) 1 if `X` is a Vector, the number of columns of `X` otherwise.
# `wl½` `t`÷2 computed by the right aritmetic bit-wise operation `wl`>>1.
# `two_wl⁻¹` = 2/wl.
# `cT` the complex type corresponding to type `T` in function definition
#     (e.g., ComplexF64 if T=Float64).
# `f`, the lower limit (in samples) of the central half of each
#     Hilbert transform to cumulate with respect to the FFT window.
# `g`, the upper limit (in samples) of the central half of each
#     Hilbert transform to cumulate with respect to the FFT window.
# `ζ`, =zeros(cT, wl-wl½_), a zero-vector append to the iFFT vectors,
#     since the second half of the FFT is not computed
# NB if wl > 2^14 then t is set to 2^10. This affects the default
#     behavior of all AS functions.
function _paramHT!( X       :: AbstractArray{T},
                    wl      :: Int,
                    planner :: Planner) where T<:Real

    size(X, 1) < wl && @error 📌*", the number of samples in input matrix is smaller than the desired window length."
    wl > 2^14 ? wl = 2^10 : nothing
    n, wl½, two_wl⁻¹ = size(X, 2), (wl>>1), T(2/wl)
    if size(X, 1)>wl
        X_=[zeros(T, wl½, n); X; zeros(T, wl½, n)] else X_=X
        isodd(wl) && @error 📌*", for Welch-like Analytic Signal estimation `wl` must be even."
    end
    tₐₗₗ = size(X_, 1) # for sliding-windows anaytic signal, this is not size(X, 1)
    cT = typeof(Complex(T(0)))
    if planner ≠ getplanner
       𝚙=planner.p
       i𝚙=planner.ip
    else
       𝚙=plan_rfft(zeros(T, wl), flags=plan_estimate, timelimit=-1.0)
       i𝚙=plan_bfft(zeros(cT, wl), flags=plan_estimate, timelimit=-1.0)
    end
    f=wl½÷2+1 # lower limit of central region to copy
    g=f+wl½-1 # upper limit of central region to copy
    e = (tₐₗₗ-wl)÷wl½+1 # number of 50% overlapping epochs or 1 if tₐₗₗ=wl
    ζ=zeros(cT, wl-wl½-1)

    return X_, 𝚙, i𝚙, e, tₐₗₗ, n, wl½, two_wl⁻¹, cT, f, g, ζ
end


using Klara

plogtarget(p::Int, v::Vector) = p*log(v[1])-log(factorial(p))

p = BasicDiscUnvParameter(:p, logtarget=plogtarget, nkeys=2)

model = likelihood_model([Constant(:λ), p], isindexed=false)

psetproposal(i::Int) = (i == 0) ? Binary(0, 1) : Binary(i-1, i+1)

sampler = MH(psetproposal, symmetric=false)

mcrange = BasicMCRange(nsteps=10000, burnin=1000)

v0 = Dict(:λ=>6, :p=>2)

job = BasicMCJob(model, sampler, mcrange, v0)

run(job)

chain = output(job)

mean(chain)

acceptance(chain, diagnostics=false)


export AbstractAffinePatch,
    nequations,
    state,
    onpatch!,
    setup!,
    changepatch!

"""
    AbstractAffinePatch

An affine patch is a hyperplane defined by ``v⋅x-1=0``.
"""
abstract type AbstractAffinePatch end

"""
    AbstractAffinePatchState{N}

This holds the actual patch information. `N` indicates the number of products of projective spaces.
"""
abstract type AbstractAffinePatchState{N} end

"""
    nequations(::AbstractAffinePatchState)

Number of equations an affine patch adds.
"""
nequations(::AbstractAffinePatchState{N}) where N = N

"""
    state(::AbstractAffinePatch, x)::AbstractAffinePatchState

Construct the state of the path from `x`.
"""
state(p::AbstractAffinePatch, x) = throw(MethodError(state, (p,x)))

"""
    onpatch!(x::AbstractVector, ::AbstractAffinePatchState)

Scale a vector `x` such that it is on the affine patch.
"""
function onpatch! end

"""
    setup!(::AbstractAffinePatchState, x::AbstractVector)

Setup the affine patch depending on `x` and modify `x` if necessary.
This is only called once at the beginning of a tracked path.
"""
setup!(state::AbstractAffinePatchState, x::AbstractVector) = onpatch!(x, state)

"""
    changepatch!(::AbstractAffinePatch, x::AbstractVector)

The same as [`setup!`](@ref) but only called during the path tracking.
"""
changepatch!(::AbstractAffinePatchState, x::AbstractVector) = nothing

include("affine_patches/common.jl")
include("affine_patches/orthogonal_patch.jl")
include("affine_patches/embedding_patch.jl")
include("affine_patches/random_patch.jl")
include("affine_patches/fixed_patch.jl")


using ForwardDiff

#thresh = 1e-12

@testset "Derivative checks" begin

    ###################################
    # Test Jacobians
    ###################################

    @testset "Jacobian checks" begin

        # Quaternion to rotation matrix
        @testset "Jacobian (Quat -> RotMatrix)" begin
            for i = 1:10    # do some repeats
                q = rand(Quat)  # a random quaternion

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(RotMatrix, q)
                FD_jac = ForwardDiff.jacobian(x -> SVector{9}(Quat(x[1], x[2], x[3], x[4])),
                                              SVector(q.w, q.x, q.y, q.z))

                # compare
                @test FD_jac ≈ R_jac
            end
        end

        # SPQuat to Quat
        @testset "Jacobian (SPQuat -> Quat)" begin
            for i = 1:10    # do some repeats
                spq = rand(SPQuat)  # a random SPQUat
                # TODO: should test the full domain of SPQuats, not just the one with ||.|| < 1

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(Quat, spq)
                FD_jac = ForwardDiff.jacobian(x -> (q = Quat(SPQuat(x[1],x[2],x[3]));
                                                    SVector(q.w, q.x, q.y, q.z)),
                                              SVector(spq.x, spq.y, spq.z))

                # compare
                @test FD_jac ≈ R_jac
            end
        end

        # SPQuat to Quat
        @testset "Jacobian (Quat -> SPQuat)" begin
            for i = 1:10    # do some repeats
                q = rand(Quat)  # a random Quat

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(SPQuat, q)
                FD_jac = ForwardDiff.jacobian(x -> (spq = SPQuat(Quat(x[1],x[2],x[3],x[4]));
                                                    SVector(spq.x, spq.y, spq.z)),
                                              SVector(q.w, q.x, q.y, q.z))

                # compare
                @test FD_jac ≈ R_jac
            end
        end

        # SPQuat to rotation matrix
        @testset "Jacobian (SPQuat -> RotMatrix)" begin
            for i = 1:10    # do some repeats
                spq = rand(SPQuat)  # a random quaternion

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(RotMatrix, spq)
                FD_jac = ForwardDiff.jacobian(x -> SVector{9}(SPQuat(x[1], x[2], x[3])),
                                              SVector(spq.x, spq.y, spq.z))

                # compare
                @test FD_jac ≈ R_jac
            end
        end

        ##############################
        # Jacobians for rotating stuff
        ##############################
#=
        # Quaternion multiplication
        @testset "Jacobian (Quaternion muliplication w.r.t. the right quaternion)" begin

            for i = 1:10    # do some repeats

                ql = quatrand()    # a random quaternion (should work for non-unit quaternions)
                qr = quatrand()    # a random quaternion (should work for non-unit quaternions)

                # transformation function
                J(X) = vec(ql * Quaternion(X...))

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(ql, qr)
                FD_jac = ForwardDiff.jacobian(J, vec(qr))

                # compare
                @test all(abs(Matrix(R_jac) - FD_jac) .< thresh)

            end
        end

        @testset "Jacobian (Quaternion muliplication w.r.t. the left quaternion)" begin

            for i = 1:10    # do some repeats

                ql = quatrand()    # a random quaternion (should work for non-unit quaternions)
                qr = quatrand()    # a random quaternion (should work for non-unit quaternions)

                # transformation function
                J(X) = vec(Quaternion(X...) * qr)

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(ql, qr, Val{false})
                FD_jac = ForwardDiff.jacobian(J, vec(ql))

                # compare
                @test all(abs(Matrix(R_jac) - FD_jac) .< thresh)

            end
        end =#

        # rotate a point by a RotMatrix
        @testset "Jacobian (RotMatrix rotation)" begin
            for i = 1:10    # do some repeats
                r = rand(RotMatrix{Float64})    # a random quaternion
                v = randn(SVector{3,Float64})

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(r, v)
                FD_jac = ForwardDiff.jacobian(x -> RotMatrix(x)*v,
                                              SVector(r))

                # compare
                @test FD_jac ≈ R_jac
            end
        end

        # rotate a point by a quaternion
        @testset "Jacobian (Quat rotation)" begin
            for i = 1:10    # do some repeats
                q = rand(Quat)    # a random quaternion
                v = randn(SVector{3,Float64})

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(q, v)
                FD_jac = ForwardDiff.jacobian(x -> Quat(x[1], x[2], x[3], x[4])*v,
                                              SVector(q.w, q.x, q.y, q.z))

                # compare
                @test FD_jac ≈ R_jac
            end
        end

        # rotate a point by a SPQuat
        @testset "Jacobian (SPQuat rotation)" begin
            for i = 1:10    # do some repeats
                spq = rand(SPQuat)    # a random quaternion
                v = randn(SVector{3,Float64})

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(spq, v)
                FD_jac = ForwardDiff.jacobian(x -> SPQuat(x[1], x[2], x[3])*v,
                                              SVector(spq.x, spq.y, spq.z))

                # compare
                @test FD_jac ≈ R_jac
            end
        end
#=
        # rotate a point by an SpQuat
        @testset "Jacobian (SpQuat rotation)" begin

            for i = 1:10    # do some repeats

                spq = SpQuat(nquatrand())    # a random quaternion
                X = randn(Vec{3,Float64})

                # transformation function
                J(spq) = Vector(rotate(SpQuat(spq...), X))

                # test jacobian to a Rotation matrix
                R_jac = Rotations.jacobian(spq, X)
                FD_jac = ForwardDiff.jacobian(J, vec(spq))

                # compare
                @test all(abs(Matrix(R_jac) - FD_jac) .< thresh)

            end
        end
=#

    end

    ###################################
    # Test Hessians
    ###################################
    #=
    @testset "Hessian checks" begin

        # SpQuat to Quaternion
        @testset "Hessian (SpQuat -> Quaternion)" begin

            for i = 1:10    # do some repeats

                q = nquatrand()                           # a random quaternion
                spq = Rotations.quat_to_spquat_naive(q)   # I want to test the full domain of SpQuats, not just the one with ||.|| < 1

                # test jacobian to a Rotation matrix
                R_hess = Rotations.hessian(Quaternion, spq)
                for d = 1:4
                    H(X) = vec(Quaternion(SpQuat(X...)))[d]   # transformation function
                    FD_hess = ForwardDiff.hessian(H, vec(spq))

                    # compare
                    @test all(abs(Matrix(R_hess[d]) - FD_hess) .< thresh)
                end
            end
        end

        # Quaternion to SpQuat
        @testset "Hessian (Quaternion -> SpQuat)" begin

            for i = 1:10    # do some repeats

                q = nquatrand()                           # a random quaternion

                # test jacobian to a Rotation matrix
                R_hess = Rotations.hessian(SpQuat, q)
                for d = 1:3
                    H(X) = vec(SpQuat(Quaternion(X...)))[d]   # transformation function
                    FD_hess = ForwardDiff.hessian(H, vec(q))

                    # compare
                    @test all(abs(Matrix(R_hess[d]) - FD_hess) .< thresh)
                end
            end
        end

        @testset "Hessian (Quaternion rotation)" begin

            for i = 1:10   # do some repeats

                q = nquatrand()    # a random quaternion
                X = randn(Vec{3,Float64})

                # transformation function
                R_hess = Rotations.hessian(q, X)

                for d = 1:3
                    H(q) = Vector(rotate(Quaternion(q...), X))[d]

                    # test
                    FD_hess = ForwardDiff.hessian(H, vec(q))

                    # compare
                    @test all(abs(Matrix(R_hess[d]) - FD_hess) .< thresh)
                end
            end
        end

        @testset "Hessian (SpQuat rotation)" begin

            for i = 1:10    # do some repeats

                q = nquatrand()    # a random quaternion
                spq = SpQuat(q)
                X = randn(Vec{3,Float64})

                # transformation function
                R_hess = Rotations.hessian(spq, X)

                for d = 1:3
                    H(spq) = Vector(rotate(SpQuat(spq...), X))[d]

                    # test
                    FD_hess = ForwardDiff.hessian(H, vec(spq))

                    # compare
                    @test all(abs(Matrix(R_hess[d]) - FD_hess) .< thresh)
                end
            end
        end
    end =#
end


@testset "Rational elliptic curves" begin

  curves_to_test_tor_short = [
  ([1, 2], 4),
  ([2, 3], 2),
  ([0, 1], 6),
  ([1, 0], 2),
  ([1, 1], 1)
  ]

  curves_to_test_tor_struc =
  [
  ([0, -1, 1, -7820, -263580], [1]),
  ([1, 0, 1, -2731, -55146], [2]),
  ([0, 1, 1, -9, -15], [3]),
  ([1, 1, 1, -80, 242], [4]),
  ([0, -1, 1, -10, -20], [5]),
  ([1, 0, 1, -36, -70], [6]),
  ([1, -1, 1, -3, 3], [7]),
  ([1, 1, 1, 35, -28], [8]),
  ([1, -1, 1, -14, 29], [9]),
  ([1, 0, 0, -45, 81], [10]),
  ([1, -1, 1, -122, 1721], [12]),
  ([1, 1, 1, -135, -660], [2, 2]),
  ([1, 1, 1, -10, -10], [2, 4]),
  ([1, 0, 1, -19, 26], [2, 6]),
  ([1, 0, 0, -1070, 7812], [2, 8]),
  ]

  @testset "Point order computation" begin
    E = EllipticCurve([0, -1, 1, -7820, -263580])
    @test 1 == @inferred order(infinity(E))

    E = EllipticCurve([1, 0, 1, -2731, -55146])
    @test 2 == @inferred order(E([fmpq(-121, 4), fmpq(117, 8)]))

    E = EllipticCurve([0, 1, 1, -9, -15])
    @test 3 == @inferred order(E([5, 9]))

    E = EllipticCurve([1, 1, 1, -80, 242])
    @test 4 == @inferred order(E([5, -2]))

    E = EllipticCurve([0, -1, 1, -10, -20])
    @test 5 == @inferred order(E([5, 5]))

    E = EllipticCurve([1, 0, 1, -36, -70])
    @test 6 == @inferred order(E([-4, 5]))

    E = EllipticCurve([1, -1, 1, -3, 3])
    @test 7 == @inferred order(E([1, 0]))

    E = EllipticCurve([1, 1, 1, 35, -28])
    @test 8 == @inferred order(E([2, 6]))

    E = EllipticCurve([1, -1, 1, -14, 29])
    @test 9 == @inferred order(E([-3, 7]))

    E = EllipticCurve([1, 0, 0, -45, 81])
    @test 10 == @inferred order(E([0, 9]))

    E = EllipticCurve([1, -1, 1, -122, 1721])
    @test 12 == @inferred order(E([-9, 49]))
  end

  @testset "Torsion test" begin
    E = EllipticCurve([1, -1, 1, -19353, 958713])
    @test @inferred istorsion_point(E([103,172]))
    @test @inferred !istorsion_point(E([-121,1292]))
  end

  @testset "Torsion points (Lutz-Nagell)" begin
    for c in curves_to_test_tor_short
      E = EllipticCurve(c[1])
      T = @inferred torsion_points_lutz_nagell(E)
      @test c[2] == length(T)
    end
  end

  @testset "Torsion points (division polynomials" begin
    for c in curves_to_test_tor_short
      E = EllipticCurve(c[1])
      T = @inferred torsion_points_division_poly(E)
      @test c[2] == length(T)
    end

    for c in curves_to_test_tor_struc
      E = EllipticCurve(c[1])
      T = @inferred torsion_points_division_poly(E)
      @test prod(c[2]) == length(T)
    end
  end

  @testset "Torsion points" begin
    for c in curves_to_test_tor_short
      E = EllipticCurve(c[1])
      T = @inferred torsion_points(E)
      @test c[2] == length(T)
      # test the caching
      T = @inferred torsion_points(E)
      @test c[2] == length(T)
    end

    for c in curves_to_test_tor_struc
      E = EllipticCurve(c[1])
      T = @inferred torsion_points(E)
      @test prod(c[2]) == length(T)
      # test the caching
      T = @inferred torsion_points(E)
      @test prod(c[2]) == length(T)
    end
  end

  @testset "Torsion point structure" begin
    for c in curves_to_test_tor_struc
      E = EllipticCurve(c[1])
      T = @inferred torsion_structure(E)
      @test T[1] == c[2]
      @test order(T[2][1]) == T[1][1]
      if length(T[1]) == 2
        @test order(T[2][2]) == T[1][2]
      end
      # test the caching
      T = @inferred torsion_structure(E)
      @test T[1] == c[2]
      @test order(T[2][1]) == T[1][1]
      if length(T[1]) == 2
        @test order(T[2][2]) == T[1][2]
      end
    end
  end

  @testset "Minimal model (Laska-Kraus-Connell" begin
    E = EllipticCurve([1,2,3,4,5])
    EE = @inferred laska_kraus_connell(E)
    @test EE.coeff == [1, -1, 0, 4, 3]

    E = EllipticCurve([625, -15625, 19531250, -2929687500, -34332275390625])
    EE = @inferred laska_kraus_connell(E)
    @test EE.coeff == [1, -1, 0, 4, 3]
  end

  @testset "Tates algorithm" begin
    E = EllipticCurve([625, -15625, 19531250, -2929687500, -34332275390625])
    EE = @inferred tates_algorithm_global(E)
    @test EE.coeff == [1, -1, 0, 4, 3]

    E = EllipticCurve([1,2,3,4,5])
    EE = @inferred tates_algorithm_global(E)
    @test EE.coeff == [1, -1, 0, 4, 3]

    #  25350.a1
    E = EllipticCurve([1, 1, 0, 40050, 7557750])
    Ep, K, f, c = tates_algorithm_local(E, 2)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "I1"
    @test f == 1
    @test c == 1

    Ep, K, f, c = tates_algorithm_local(E, 3)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "I2"
    @test f == 1
    @test c == 2

    Ep, K, f, c = tates_algorithm_local(E, 5)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "III*"
    @test f == 2
    @test c == 2

    Ep, K, f, c = tates_algorithm_local(E, 13)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "IV*"
    @test f == 2
    @test c == 1

    # 150.a1
    E = EllipticCurve([1, 1, 0, -20700, 1134000])
    Ep, K, f, c = tates_algorithm_local(E, 2)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "I5"
    @test f == 1
    @test c == 1

    Ep, K, f, c = tates_algorithm_local(E, 3)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "I10"
    @test f == 1
    @test c == 2

    Ep, K, f, c = tates_algorithm_local(E, 5)
    @test tidy_model(Ep).coeff == E.coeff
    @test K == "III*"
    @test f == 2
    @test c == 2
  end
end


struct Regularize{N,F}

  "x values of points, normalized to grid index space"
  x :: Vector{Float64}

  "y values of points, normalized to grid index space"
  y :: Vector{Float64}

  "1/dV factor"
  overdv :: Float64

  "weights for each point (e.g. arclengths), divided by dV"
  wgt :: Vector{Float64}

  "Discrete Delta function"
  ddf :: AbstractDDF

  "Radius of the DDF"
  ddf_radius :: Float64

  "Symmetry flag"
  _issymmetric :: Bool

end

"""
    Regularize(x,y,dx,[ddftype=Yang3],[graddir=0],[I0=(1,1)], [weights=1.0], [filter=false],
                       [issymmetric=false])

Constructor to set up an operator for regularizing and interpolating data from/to
points immersed in the grid to/from fields on the grid itself. The supplied
`x` and `y` represent physical coordinates of the immersed points, and `dx`
denotes a uniform physical cell size of the grid. The separate arguments `x` and
`y` can be replaced by a single argument `X` of type `VectorData` holding the
coordinates.

The operations of regularization and interpolation are carried out with a discrete
delta function (ddf), which defaults to the type `Yang3`. Others are also possible,
such as `Roma`, `Goza` or `M3`. The optional argument `graddir`, if set to 1 or 2, will
generate an interpolation operator that evaluates the negative of the
respective component of the gradient of a grid field at the immersed points. The
default value of this argument is 0, which simply interpolates. Note that the
regularization form of this gradient type is also possible.

The optional tuple
`I0` represents the indices of the primary node that coincides with `(x,y) = (0,0)`.
This defaults to `(1,1)`, which leaves one layer of ghost (dual) cells and sets
the physical origin in the lower left corner of the grid of interior dual cells.

Another optional parameter, `weights`, sets the weight of each point in the
regularization. This would generally be set with, say, the differential arc
length for regularization of data on a curve. It can be a vector (of the same length
as x and y) or a scalar if uniform. It defaults to 1.0.

The optional Boolean parameter `filter` can be set to `true` if it is desired to
apply filtering (see Goza et al, J Comput Phys 2016) to the grid data before
interpolating. This is generally only used in the context of preconditioning
the solution for forces on the immersed points.

If the optional Boolean parameter `issymmetric` is set to `true`, then the
regularization and interpolation are constructed to be transposes of each other.
Note that this option overrides any supplied weights. The default of this
parameter is `false`.

The resulting operator can be used in either direction, regularization and
interpolation, with the first argument representing the *target* (the entity
to regularize/interpolate to), and the second argument
the *source* (the entity to regularize/interpolate from). The regularization
does not use the filtering option.

# Example

In the example below, we set up a 12 x 12 grid. Using the default value for `I0`
and setting `dx = 0.1`, the physical dimensions of the non-ghost part of the grid
are 1.0 x 1.0. Three points are set up in the interior, and a vector field is assigned
to them, with the x component of each of them set to 1.0. These data are regularized
to a field of primal edges on the grid, using the Roma DDF kernel.

```jldoctest
julia> x = [0.25,0.75,0.25]; y = [0.75,0.25,0.25];

julia> X = VectorData(x,y);

julia> q = Edges(Primal,(12,12));

julia> dx = 0.1;

julia> H = Regularize(x,y,dx;ddftype=Roma)
Regularization/interpolation operator with non-filtered interpolation
  DDF type CartesianGrids.Roma
  3 points in grid with cell area 0.01

julia> f = VectorData(X);

julia> fill!(f.u,1.0);

julia> H(q,f)
Edges{Primal,12,12,Float64} data
u (in grid orientation)
11×12 Array{Float64,2}:
 0.0  0.0  0.0       0.0     0.0      …  0.0       0.0     0.0      0.0  0.0
 0.0  0.0  0.0       0.0     0.0         0.0       0.0     0.0      0.0  0.0
 0.0  0.0  8.33333  33.3333  8.33333     0.0       0.0     0.0      0.0  0.0
 0.0  0.0  8.33333  33.3333  8.33333     0.0       0.0     0.0      0.0  0.0
 0.0  0.0  0.0       0.0     0.0         0.0       0.0     0.0      0.0  0.0
 0.0  0.0  0.0       0.0     0.0      …  0.0       0.0     0.0      0.0  0.0
 0.0  0.0  0.0       0.0     0.0         0.0       0.0     0.0      0.0  0.0
 0.0  0.0  8.33333  33.3333  8.33333     8.33333  33.3333  8.33333  0.0  0.0
 0.0  0.0  8.33333  33.3333  8.33333     8.33333  33.3333  8.33333  0.0  0.0
 0.0  0.0  0.0       0.0     0.0         0.0       0.0     0.0      0.0  0.0
 0.0  0.0  0.0       0.0     0.0      …  0.0       0.0     0.0      0.0  0.0
v (in grid orientation)
12×11 Array{Float64,2}:
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
```
"""
function Regularize(x::AbstractVector{T},y::AbstractVector{T},dx::T;
                    ddftype=Yang3,graddir::Int=0,
                    I0::Tuple{Int,Int}=(1,1),
                    weights::Union{T,Vector{T}}=1.0,
                    filter::Bool = false,
                    issymmetric::Bool = false) where {T<:Real}

  _issymmetric = (filter ? false : issymmetric)

  n = length(x)
  @assert length(y)==n
  if !_issymmetric
    if typeof(weights) == T
      wtvec = similar(x)
      fill!(wtvec,weights/(dx*dx))
    else
      @assert length(weights)==n
      wtvec = deepcopy(weights)./(dx*dx)
    end
  else
    # if the regularization and interpolation are symmetric, then the
    # weights are automatically set to be the cell area in order to cancel it
    # in the denominator of the regularization operator.
    wtvec = similar(x)
    fill!(wtvec,1.0)
  end

  baseddf = DDF(ddftype=ddftype,dx=1.0)
  if graddir == 0
    ddf = baseddf
  else
    ddf = GradDDF(graddir,ddftype=ddftype,dx=1.0)
  end

  Regularize{length(x),filter}(x/dx.+I0[1],y/dx.+I0[2],1.0/(dx*dx),
                      wtvec,ddf,_get_regularization_radius(baseddf),_issymmetric)
end

Regularize(x::T,y::T,a...;b...) where {T<:Real} = Regularize([x],[y],a...;b...)

Regularize(x::VectorData,a...;b...) = Regularize(x.u,x.v,a...;b...)

function Base.show(io::IO, H::Regularize{N,F}) where {N,F}
    filter = F ? "filtered" : "non-filtered"
    ddftype,_ = typeof(H.ddf).parameters
    op = H._issymmetric ? "Symmetric regularization/interpolation" : "Regularization/interpolation"
    println(io, "$op operator with $filter interpolation")
    println(io, "  DDF type $ddftype")
    println(io, "  $N points in grid with cell area $(sprint(show,1.0/H.overdv;context=:compact => true))")
end

function _get_regularization_radius(ddf::DDF)
  v = 1.0
  r = 0.0
  dr = 0.01
  while (v = abs(ddf(r))) > eps()
    r += dr
  end
  return r
end

"""
    RegularizationMatrix(H::Regularize,f::PointData,u::CellData) -> Hmat

Construct and store a matrix representation of regularization associated with `H`
for data of type `f` to data of type `u`. The resulting matrix `Hmat` can then be
used to apply on point data of type `f` to regularize it to grid data of type `u`,
using `mul!(u,Hmat,f)`. It can also be used as just `Hmat*f`.

If `H` is a symmetric regularization and interpolation operator, then this
actually returns a tuple `Hmat, Emat`, where `Emat` is the interpolation matrix.
"""
struct RegularizationMatrix{TU,TF} <: AbstractMatrix{Float64}
  M :: SparseMatrixCSC{Float64,Int64}
end

"""
    InterpolationMatrix(H::Regularize,u::CellData,f::PointData) -> Emat

Construct and store a matrix representation of interpolation associated with `H`
for data of type `u` to data of type `f`. The resulting matrix `Emat` can then be
used to apply on grid data of type `u` to interpolate it to point data of type `f`,
using `mul!(f,Emat,u)`. It can also be used as just `Emat*u`.
"""
struct InterpolationMatrix{TU,TF} <: AbstractMatrix{Float64}
  M :: SparseMatrixCSC{Float64,Int64}
end

@wraparray RegularizationMatrix M 2
@wraparray InterpolationMatrix M 2

# ======  Regularization and interpolation operators of scalar types ======== #

pointtype = :ScalarData
for (gridtype,ctype,dnx,dny,shiftx,shifty) in @generate_scalarlist(SCALARLIST)

# Regularization
  @eval function (H::Regularize{N,F})(target::$gridtype{$ctype,NX,NY,T,DDT},source::$pointtype{N,S,DT}) where {N,F,NX,NY,S,T,DT,DDT}
        radius = H.ddf_radius
        fill!(target,0.0)
        xmin, xmax = -radius+$shiftx, radius+$shiftx
        ymin, ymax = -radius+$shifty, radius+$shifty
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                target[x,y] += source[pt]*H.wgt[pt]*H.ddf(x-$shiftx-H.x[pt],y-$shifty-H.y[pt])
            end
        end
        target
  end

# Interpolation
  @eval function (H::Regularize{N,false})(target::$pointtype{N,S,DT},source::$gridtype{$ctype,NX,NY,T,DDT}) where {N,NX,NY,S,T,DT,DDT}
        radius = H.ddf_radius
        fill!(target,0.0)
        xmin, xmax = -radius+$shiftx, radius+$shiftx
        ymin, ymax = -radius+$shifty, radius+$shifty
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                target[pt] += source[x,y]*H.ddf(x-$shiftx-H.x[pt],y-$shifty-H.y[pt])
            end
        end
        target
  end

# Interpolation with filtering
  @eval function (H::Regularize{N,true})(target::$pointtype{N,S,DT},source::$gridtype{$ctype,NX,NY,T,DDT}) where {N,NX,NY,S,T,DT,DDT}
        tmp = typeof(source)()
        radius = H.ddf_radius
        fill!(target,0.0)
        xmin, xmax = -radius+$shiftx, radius+$shiftx
        ymin, ymax = -radius+$shifty, radius+$shifty
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                tmp[x,y] += H.wgt[pt]*H.ddf(x-$shiftx-H.x[pt],y-$shifty-H.y[pt])
            end
        end
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dny,floor(Int,H.y[pt]+ymax))
            @inbounds for y in prangey, x in prangex
                w = tmp[x,y]
                w = w > eps() ? source[x,y]/w : 0.0
                target[pt]  += w*H.ddf(x-$shiftx-H.x[pt],y-$shifty-H.y[pt])
            end
        end
        target
  end

  # Construct regularization matrix
  @eval function RegularizationMatrix(H::Regularize{N,F},
    f::$pointtype{N,S,DT},
    u::$gridtype{$ctype,NX,NY,T,DDT}) where {N,F,NX,NY,S,T,DT,DDT}

    Hmat = spzeros(length(u),length(f))
    g = deepcopy(f)
    v = deepcopy(u)
    fill!(g,0.0)
    for i = 1:N
      g[i] = 1.0
      Hmat[:,i] = sparsevec(H(v,g))
      g[i] = 0.0
    end
    if H._issymmetric
      # In symmetric case, these matrices are identical. (Interpolation is stored
      # as its transpose.)
      return RegularizationMatrix{typeof(u),typeof(f)}(Hmat),InterpolationMatrix{typeof(u),typeof(f)}(Hmat)
    else
      return RegularizationMatrix{typeof(u),typeof(f)}(Hmat)
    end
  end

  # Construct interpolation matrix
  @eval function InterpolationMatrix(H::Regularize{N,false},
    u::$gridtype{$ctype,NX,NY,T,DDT},
    f::$pointtype{N,S,DT}) where {N,NX,NY,S,T,DT,DDT}

    Emat = spzeros(length(u),length(f))
    g = deepcopy(f)
    v = deepcopy(u)
    fill!(g,0.0)
    for i = 1:N
      g[i] = 1.0/H.wgt[i]  # unscale for interpolation
      Emat[:,i] = sparsevec(H(v,g))
      g[i] = 0.0
    end
    InterpolationMatrix{typeof(u),typeof(f)}(Emat)
  end

  # Construct interpolation matrix with filtering. Because we construct
  # the matrix by supplying unit vectors from the points, it is easier
  # to carry out the filtering here rather than call the filtered operator
  # we defined above
  @eval function InterpolationMatrix(H::Regularize{N,true},
    u::$gridtype{$ctype,NX,NY,T,DDT},
    f::$pointtype{N,S,DT}) where {N,NX,NY,S,T,DT,DDT}

    Emat = spzeros(length(u),length(f))
    g = deepcopy(f)
    v = deepcopy(u)
    fill!(g,1.0)
    wt = sparsevec(H(v,g))
    wt.nzval .= 1 ./ wt.nzval
    fill!(g,0.0)
    for i = 1:N
      g[i] = 1.0/H.wgt[i]  # unscale for interpolation
      Emat[:,i] = wt.*sparsevec(H(v,g))
      g[i] = 0.0
    end
    InterpolationMatrix{typeof(u),typeof(f)}(Emat)
  end

end

# ===== Regularization and interpolation operators of vector data to edges ===== #
pointtype = :VectorData
for (gridtype,ctype,dunx,duny,dvnx,dvny,shiftux,shiftuy,shiftvx,shiftvy) in @generate_collectionlist(VECTORLIST)

# Regularization
  @eval function (H::Regularize{N,F})(target::$gridtype{$(ctype...),NX,NY,T,DDT},source::$pointtype{N,S,DT}) where {N,F,NX,NY,S,T,DT,DDT}
        H(target.u,source.u)
        H(target.v,source.v)
        target
  end

# Interpolation
  @eval function (H::Regularize{N,F})(target::$pointtype{N,S,DT},source::$gridtype{$(ctype...),NX,NY,T,DDT}) where {N,F,NX,NY,S,T,DT,DDT}
        H(target.u,source.u)
        H(target.v,source.v)
        target
  end

  # Construct regularization matrix
  @eval function RegularizationMatrix(H::Regularize{N,F},src::$pointtype{N,S,DT},target::$gridtype{$(ctype...),NX,NY,T,DDT}) where {N,F,NX,NY,S,T,DT,DDT}

    lenu = length(target.u)
    lenv = length(target.v)
    Hmat = spzeros(lenu+lenv,2N)

    if H._issymmetric
      # In symmetric case, these matrices are identical. (Interpolation is stored
      # as its transpose.)
      Hmat[1:lenu,          1:N]    = RegularizationMatrix(H,src.u,target.u)[1].M
      Hmat[lenu+1:lenu+lenv,N+1:2N] = RegularizationMatrix(H,src.v,target.v)[1].M
      return RegularizationMatrix{typeof(target),typeof(src)}(Hmat),InterpolationMatrix{typeof(target),typeof(src)}(Hmat)
    else
      Hmat[1:lenu,          1:N]    = RegularizationMatrix(H,src.u,target.u).M
      Hmat[lenu+1:lenu+lenv,N+1:2N] = RegularizationMatrix(H,src.v,target.v).M
      return RegularizationMatrix{typeof(target),typeof(src)}(Hmat)
    end
  end

  # Construct interpolation matrix
  @eval function InterpolationMatrix(H::Regularize{N,F},src::$gridtype{$(ctype...),NX,NY,T,DDT},target::$pointtype{N,S,DT}) where {N,F,NX,NY,S,T,DT,DDT}

    # note that we store interpolation matrices in the same shape as regularization matrices
    lenu = length(src.u)
    lenv = length(src.v)
    Emat = spzeros(lenu+lenv,2N)
    Emat[1:lenu,          1:N]    = InterpolationMatrix(H,src.u,target.u).M
    Emat[lenu+1:lenu+lenv,N+1:2N] = InterpolationMatrix(H,src.v,target.v).M
    InterpolationMatrix{typeof(src),typeof(target)}(Emat)
  end

end

# ======  Regularization and interpolation operators of tensor data to edge gradients ==== #
# Here, u describes both diagonal components and v the off diagonal
# We do not use the scalar-wise operations here because there is double use of
# the same ddf evaluation, so this saves us quite a bit of time.

pointtype = :TensorData
for (gridtype,ctype,dunx,duny,dvnx,dvny,shiftux,shiftuy,shiftvx,shiftvy) in @generate_collectionlist(TENSORLIST)

# Regularization
  @eval function (H::Regularize{N,F})(target::$gridtype{$(ctype...),NX,NY,T,DDT},source::$pointtype{N,S,DT}) where {N,F,NX,NY,S,T,DT,DDT}
        radius = H.ddf_radius
        fill!(target.dudx,0.0)
        fill!(target.dvdy,0.0)
        xmin, xmax = -radius+$shiftux, radius+$shiftux
        ymin, ymax = -radius+$shiftuy, radius+$shiftuy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dunx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$duny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                d = H.wgt[pt]*H.ddf(x-$shiftux-H.x[pt],y-$shiftuy-H.y[pt])
                target.dudx[x,y] += source.dudx[pt]*d
                target.dvdy[x,y] += source.dvdy[pt]*d
            end
        end
        fill!(target.dudy,0.0)
        fill!(target.dvdx,0.0)
        xmin, xmax = -radius+$shiftvx, radius+$shiftvx
        ymin, ymax = -radius+$shiftvy, radius+$shiftvy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dvnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dvny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                d = H.wgt[pt]*H.ddf(x-$shiftvx-H.x[pt],y-$shiftvy-H.y[pt])
                target.dudy[x,y] += source.dudy[pt]*d
                target.dvdx[x,y] += source.dvdx[pt]*d
            end
        end
        target
  end

  # @eval function (H::Regularize{N,F})(target::$ctype,source::$ftype) where {N,F,NX,NY,S,T}
  #   H(target.dudx,source.dudx)
  #   H(target.dudy,source.dudy)
  #   H(target.dvdx,source.dvdx)
  #   H(target.dvdy,source.dvdy)
  #   target
  # end

# Interpolation
  @eval function (H::Regularize{N,false})(target::$pointtype{N,S,DT},source::$gridtype{$(ctype...),NX,NY,T,DDT}) where {N,NX,NY,S,T,DT,DDT}
        radius = H.ddf_radius
        fill!(target.dudx,0.0)
        fill!(target.dvdy,0.0)
        xmin, xmax = -radius+$shiftux, radius+$shiftux
        ymin, ymax = -radius+$shiftuy, radius+$shiftuy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dunx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$duny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                d = H.ddf(x-$shiftux-H.x[pt],y-$shiftuy-H.y[pt])
                target.dudx[pt] += source.dudx[x,y]*d
                target.dvdy[pt] += source.dvdy[x,y]*d
            end
        end
        fill!(target.dudy,0.0)
        fill!(target.dvdx,0.0)
        xmin, xmax = -radius+$shiftvx, radius+$shiftvx
        ymin, ymax = -radius+$shiftvy, radius+$shiftvy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dvnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dvny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                d = H.ddf(x-$shiftvx-H.x[pt],y-$shiftvy-H.y[pt])
                target.dudy[pt] += source.dudy[x,y]*d
                target.dvdx[pt] += source.dvdx[x,y]*d
            end
        end
        target
  end

  # @eval function (H::Regularize{N,F})(target::$ftype,source::$ctype) where {N,F,NX,NY,S,T}
  #   H(target.dudx,source.dudx)
  #   H(target.dudy,source.dudy)
  #   H(target.dvdx,source.dvdx)
  #   H(target.dvdy,source.dvdy)
  #   target
  # end

# Interpolation with filtering -- need to speed up
  @eval function (H::Regularize{N,true})(target::$pointtype{N,S,DT},source::$gridtype{$(ctype...),NX,NY,T,DDT}) where {N,NX,NY,S,T,DT,DDT}
        tmp = typeof(source)()
        radius = H.ddf_radius
        fill!(target.dudx,0.0)
        fill!(target.dvdy,0.0)
        xmin, xmax = -radius+$shiftux, radius+$shiftux
        ymin, ymax = -radius+$shiftuy, radius+$shiftuy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dunx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$duny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                tmp.dudx[x,y] += H.wgt[pt]*H.ddf(x-$shiftux-H.x[pt],y-$shiftuy-H.y[pt])
            end
        end
        tmp.dvdy .= tmp.dudx
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dunx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$duny,floor(Int,H.y[pt]+ymax))
            @inbounds for y in prangey, x in prangex
                w = tmp.dudx[x,y]
                w = w > eps() ? source.dudx[x,y]/w : 0.0
                target.dudx[pt]  += w*H.ddf(x-$shiftux-H.x[pt],y-$shiftuy-H.y[pt])
                w = tmp.dvdy[x,y]
                w = w > eps() ? source.dvdy[x,y]/w : 0.0
                target.dvdy[pt]  += w*H.ddf(x-$shiftux-H.x[pt],y-$shiftuy-H.y[pt])
            end
        end
        fill!(target.dudy,0.0)
        fill!(target.dvdx,0.0)
        xmin, xmax = -radius+$shiftvx, radius+$shiftvx
        ymin, ymax = -radius+$shiftvy, radius+$shiftvy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dvnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dvny,floor(Int,H.y[pt]+ymax))
            for y in prangey, x in prangex
                tmp.dudy[x,y] += H.wgt[pt]*H.ddf(x-$shiftvx-H.x[pt],y-$shiftvy-H.y[pt])
            end
        end
        tmp.dvdx .= tmp.dudy
        @inbounds for pt in 1:N
            prangex = max(1,ceil(Int,H.x[pt]+xmin)):min(NX-$dvnx,floor(Int,H.x[pt]+xmax))
            prangey = max(1,ceil(Int,H.y[pt]+ymin)):min(NY-$dvny,floor(Int,H.y[pt]+ymax))
            @inbounds for y in prangey, x in prangex
                w = tmp.dudy[x,y]
                w = w > eps() ? source.dudy[x,y]/w : 0.0
                target.dudy[pt] += w*H.ddf(x-$shiftvx-H.x[pt],y-$shiftvy-H.y[pt])
                w = tmp.dvdx[x,y]
                w = w > eps() ? source.dvdx[x,y]/w : 0.0
                target.dvdx[pt] += w*H.ddf(x-$shiftvx-H.x[pt],y-$shiftvy-H.y[pt])
            end
        end
        target
  end

  # Construct regularization matrix
  @eval function RegularizationMatrix(H::Regularize{N,F},src::$pointtype{N,S,DT},target::$gridtype{$(ctype...),NX,NY,T,DDT}) where {N,F,NX,NY,S,T,DT,DDT}

    # note that we only need to compute two distinct matrices, since there are
    # only two types of cell data in this tensor

    #Hmat = (spzeros(length(target.u),length(src.u)),spzeros(length(target.v),length(src.v)))
    lenu = length(target.dudx)
    lenv = length(target.dudy)
    Hmat = spzeros(2lenu+2lenv,4N)
    g = deepcopy(src)
    v = deepcopy(target)
    g.dudx .= g.dudy .= g.dvdx .= g.dvdy .= zeros(Float64,N)
    for i = 1:N
      g.dudx[i] = 1.0  # these two are sufficient
      g.dudy[i] = 1.0
      H(v,g)
      Hmat[1:lenu,i]           = sparsevec(v.dudx)
      Hmat[lenu+1:lenu+lenv,i+N] = sparsevec(v.dudy)
      Hmat[lenu+lenv+1:lenu+2lenv,i+2N] = sparsevec(v.dudy)
      Hmat[lenu+2lenv+1:2lenu+2lenv,i+3N] = sparsevec(v.dudx)
      g.dudx[i] = 0.0
      g.dudy[i] = 0.0
    end
    if H._issymmetric
      # In symmetric case, these matrices are identical. (Interpolation is stored
      # as its transpose.)
      return RegularizationMatrix{typeof(target),typeof(src)}(Hmat),InterpolationMatrix{typeof(target),typeof(src)}(Hmat)
    else
      return RegularizationMatrix{typeof(target),typeof(src)}(Hmat)
    end
  end

  # Construct interpolation matrix
  @eval function InterpolationMatrix(H::Regularize{N,false},src::$gridtype{$(ctype...),NX,NY,T,DDT},target::$pointtype{N,S,DT}) where {N,NX,NY,S,T,DT,DDT}

    # note that we store interpolation matrices in the same shape as regularization matrices
    #Emat = (spzeros(length(src.u),length(target.u)),spzeros(length(src.v),length(target.v)))
    lenu = length(src.dudx)
    lenv = length(src.dudy)
    Emat = spzeros(2lenu+2lenv,4N)
    g = deepcopy(target)
    v = deepcopy(src)
    g.dudx .= g.dudy .= g.dvdx .= g.dvdy .= zeros(Float64,N)
    for i = 1:N
      g.dudx[i] = 1.0/H.wgt[i]  # unscale for interpolation
      g.dudy[i] = 1.0/H.wgt[i]  # unscale for interpolation
      H(v,g)
      Emat[1:lenu,i]           = sparsevec(v.dudx)
      Emat[lenu+1:lenu+lenv,i+N] = sparsevec(v.dudy)
      Emat[lenu+lenv+1:lenu+2lenv,i+2N] = sparsevec(v.dudy)
      Emat[lenu+2lenv+1:2lenu+2lenv,i+3N] = sparsevec(v.dudx)
      g.dudx[i] = 0.0
      g.dudy[i] = 0.0
    end
    InterpolationMatrix{typeof(src),typeof(target)}(Emat)
  end

  # Construct interpolation matrix with filtering
  @eval function InterpolationMatrix(H::Regularize{N,true},src::$gridtype{$(ctype...),NX,NY,T,DDT},target::$pointtype{N,S,DT}) where {N,NX,NY,S,T,DT,DDT}

    # note that we store interpolation matrices in the same shape as regularization matrices
    #Emat = (spzeros(length(src.u),length(target.u)),spzeros(length(src.v),length(target.v)))
    lenu = length(src.dudx)
    lenv = length(src.dudy)
    Emat = spzeros(2lenu+2lenv,4N)
    g = deepcopy(target)
    v = deepcopy(src)
    fill!(g,1.0)
    H(v,g)
    wtu = sparsevec(v.dudx)
    wtu.nzval .= 1 ./ wtu.nzval
    wtv = sparsevec(v.dudy)
    wtv.nzval .= 1 ./ wtv.nzval
    fill!(g,0.0)
    for i = 1:N
      g.dudx[i] = 1.0/H.wgt[i]  # unscale for interpolation
      g.dudy[i] = 1.0/H.wgt[i]  # unscale for interpolation
      H(v,g)
      Emat[1:lenu,i]             = wtu.*sparsevec(v.dudx)
      Emat[lenu+1:lenu+lenv,i+N] = wtv.*sparsevec(v.dudy)
      Emat[lenu+lenv+1:lenu+2lenv,i+2N] = wtv.*sparsevec(v.dudy)
      Emat[lenu+2lenv+1:2lenu+2lenv,i+3N] = wtu.*sparsevec(v.dudx)
      g.dudx[i] = 0.0
      g.dudy[i] = 0.0
    end
    InterpolationMatrix{typeof(src),typeof(target)}(Emat)
  end

end

###### Matrix multiplication extended to these Regularization/Interpolation matrices ########

# Regularize
function mul!(u::C,Hmat::RegularizationMatrix{C,F},f::F) where {C<:GridData,F<:PointData}
  fill!(u,0.0)
  nzv = Hmat.M.nzval
  rv = Hmat.M.rowval
  @inbounds for col = 1:Hmat.M.n
    fj = f[col]
    for j = Hmat.M.colptr[col]:(Hmat.M.colptr[col + 1] - 1)
        u[rv[j]] += nzv[j]*fj
    end
  end
  u
end

# Interpolate
function mul!(f::F,Emat::InterpolationMatrix{C,F},u::C) where {C<:GridData,F<:PointData}
  fill!(f,0.0)
  nzv = Emat.M.nzval
  rv = Emat.M.rowval
  @inbounds for col = 1:Emat.M.n
      tmp = zero(eltype(f))
      for j = Emat.M.colptr[col]:(Emat.M.colptr[col + 1] - 1)
          tmp += transpose(nzv[j])*u[rv[j]]
      end
      f[col] += tmp
  end
  f
end

# Interpolation of regularization, used for developing the filtering matrix
function mul!(C::Array{Float64},Emat::InterpolationMatrix{G,F},
                              Hmat::RegularizationMatrix{G,F}) where {G<:GridData,F<:PointData}
  fill!(C,0.0)
  Enzv = Emat.M.nzval
  Erv = Emat.M.rowval
  @inbounds for row = 1:Emat.M.n, col = 1:Hmat.M.n
      tmp = zero(eltype(C))
      for j = Emat.M.colptr[row]:(Emat.M.colptr[row + 1] - 1)
          tmp += transpose(Enzv[j])*Hmat[Erv[j],col]
      end
      C[row,col] += tmp
  end
  return C
end

(*)(Hmat::RegularizationMatrix{TU,TF},src::TF) where {TU,TF<:PointData} =
        mul!(TU(),Hmat,src)

(*)(Emat::InterpolationMatrix{TU,TF},src::TU) where {TU<:Union{Nodes,Edges,EdgeGradient,NodePair},TF<:PointData} =
                mul!(TF(),Emat,src)

(*)(Emat::InterpolationMatrix,Hmat::RegularizationMatrix) =
        mul!(Array{eltype(Emat),2}(undef,Emat.M.n,Hmat.M.n),Emat,Hmat)

function Base.summary(io::IO, H::RegularizationMatrix{TU,TF}) where {TU,TF}
    print(io, "Regularization matrix acting on type $TF and returning type $TU")
end

function Base.summary(io::IO, H::InterpolationMatrix{TU,TF}) where {TU,TF}
    print(io, "Interpolation matrix acting on type $TU and returning type $TF")
end


abstract type ClassificationModel end

"""
    KNN( X, Y; DistanceType::String )

DistanceType can be "euclidean", "manhattan". `Y` Must be one hot encoded.

Returns a KNN classification model.
"""
struct KNN <: ClassificationModel
    X::Array{Float64,2}
    Y::Array{Float64,2}
    DistanceType::String
end

"""
    ( model::KNN )( Z; K = 1 )

Returns a 1 hot encoded inference from `X` with `K` Nearest Neighbors, using a KNN object.
"""
function ( model::KNN )( Z; K = 1 )
    MostCommon = 0
    Obs = size( Z )[ 1 ]
    Classes = size(model.Y)[2]
    DistMat = zeros( size( model.X )[ 1 ], Obs )
    #Predictions = zeros( size( Z )[ 1 ] )
    Predictions = zeros( Obs, Classes )
    #Apply Distance Fn
    if model.DistanceType == "euclidean"
        DistMat = SquareEuclideanDistance(model.X, Z)
    elseif model.DistanceType == "manhattan"
        DistMat = ManhattanDistance(model.X, Z)
    end
    #Find nearest neighbors and majority vote
    for obs in 1 : Obs
        Preds = sortperm( DistMat[:, obs] )[ 1 : K ]
        if K == 1
            lbls = argmax(model.Y[ Preds,: ], dims = 2 )
            MostCommon = argmax( StatsBase.countmap( lbls ) )[2]
        else
            lbls = argmax(model.Y[ Preds,: ], dims = 1 )
            MostCommon = argmax( StatsBase.countmap( lbls ) )[2]
        end
        Predictions[ obs, MostCommon ] =  1
    end
    return Predictions
end

"""
    ProbabilisticNeuralNetwork( X, Y )

Stores data for a PNN. `Y` Must be one hot encoded.

Returns a PNN classification model.
"""
struct ProbabilisticNeuralNetwork{ a, b }
    X::a
    Y::b
end

"""
    (PNN::ProbabilisticNeuralNetwork)(X; sigma = 0.1)

Returns a 1 hot encoded inference from `X` with a probabilistic neural network.
"""
function (PNN::ProbabilisticNeuralNetwork)(X; sigma = 0.1)
    (TrainObs, Classes) = size(PNN.Y)
    Score = zeros(size(X)[1], Classes)
    rbf = GaussianKernel(X, PNN.X, sigma)
    for class in 1 : Classes
        classinds = findall( PNN.Y[ : , class ] .== 1.0 )
        Score[:, class] = sum( rbf[ classinds , : ], dims = 1 ) / length( classinds )
    end
    return Score
end

struct GaussianDiscriminant
    Basis::Union{PCA, LDA}
    ClassSize::Array
    pi::Array
    ProjectedClassMeans::Array{Float64,2}
    ProjectedClassCovariances::Array
end

"""
    GaussianDiscriminant(M, X, Y; Factors = nothing)

Returns a GaussianDiscriminant classification model on basis object `M` (PCA, LDA) and one hot encoded `Y`.
"""
function GaussianDiscriminant(M, X, Y; Factors = nothing)
    (Obs, ClassNumber) = size( Y )
    Variables = size( X )[ 2 ]
    ClassSize = zeros( ClassNumber )
    Factors = isa(Factors, Nothing) ? length(M.Values) : Factors
    Projected = M(X; Factors = Factors)
    #Calculate the probability density functions for each class
    YPred = zeros(Obs, ClassNumber)
    ProjClassMeans = zeros( ClassNumber, Factors)
    ClassCovariances = []
    for class in 1 : ClassNumber
        Members = Y[ :, class ] .== 1
        ClassSize[class] = sum(Members)
        ProjClassMeans[class, :] = StatsBase.mean(Projected[Members,:], dims = 1)
        MeanCentered = Projected[Members,:] .- ProjClassMeans[class,:]'
        push!(ClassCovariances, (1.0 / (ClassSize[class] - 1.0 )) .* ( MeanCentered' * MeanCentered  ) )
    end
    return GaussianDiscriminant(M, ClassSize, ClassSize ./ Obs, ProjClassMeans, ClassCovariances )
end

"""
    ( model::GaussianDiscriminant )( Z; Factors = size(model.ProjectedClassMeans)[2] )

Returns a 1 hot encoded inference from `Z` using a GaussianDiscriminant object.
This function enforces positive definiteness in the class covariance matrices.
"""
function ( model::GaussianDiscriminant )( Z; Factors = size(model.ProjectedClassMeans)[2] )
    MaximumLatentFactors = size(model.ProjectedClassMeans)[2]
    @assert Factors <= MaximumLatentFactors
    ClassNumber = length(model.ClassSize)
    YHat = zeros( size(Z)[1] , ClassNumber )
    Projected = model.Basis(Z; Factors = Factors)
    PDF = 0.0
    scalar = 0.0
    warn = false
    for class in 1 : ClassNumber
        MeanCentered = Projected .- model.ProjectedClassMeans[class, 1:Factors]'
        ProjClassCov = model.ProjectedClassCovariances[ class ][1:Factors, 1:Factors]
        try#catch singular exception/semipositive definiteness
            scalar = 1.0 / sqrt( (( 2.0 * pi )^Factors) * LinearAlgebra.det( ProjClassCov ) )
            scalar = (scalar == Inf) ? 1e32 : scalar
            if any(LinearAlgebra.eigen(ProjClassCov).values .< 1e-32)#Grr...
                error("Covariance is not positive definite.")
            end
            for obs in 1 : size(Z)[1]
                PDF = scalar * exp(-0.5 * (MeanCentered[obs,:]' * Base.inv( ProjClassCov ) * MeanCentered[obs,:] ) )
                YHat[obs, class] = model.pi[class] .* PDF
            end
        catch
            ProjEVD = eigen(ProjClassCov)
            ProjEVD.values[ real(ProjEVD.values) .< 1e-12 ] .= 1e-12
            Recon = ProjEVD.vectors * LinearAlgebra.Diagonal(ProjEVD.values) * Base.inv(ProjEVD.vectors)
            scalar = 1.0 / sqrt( (( 2.0 * pi )^Factors) * LinearAlgebra.det( Recon ) )
            scalar = (scalar == Inf) ? 1e32 : scalar
            for obs in 1 : size(Z)[1]
                PDF = scalar * exp(-0.5 * (MeanCentered[obs,:]' * Base.inv( Recon ) * MeanCentered[obs,:] ) )
                YHat[obs, class] = model.pi[class] .* PDF
            end
            warn = true
        end

    end
    if warn == true
        println("Warning: some class covariance matrices were not positive definite. \n This usually means a few classes have low rank. The Null Space was removed.")
    end
    return YHat
end

"""
    ConfidenceEllipse(cov, mean, confidence, axis = [1,2]; pointestimate = 180 )

Returns a 2-D array whose columns are X & Y coordinates of a confidence ellipse. The ellipse is
generated by the covariance matrix, mean vector, and the number of points to include in the plot.
"""
function ConfidenceEllipse(cov, mean, confidence, axis = [1,2]; pointestimate = 180 )
    coordtuples = (axis[1], axis[1]), (axis[1], axis[2]), (axis[2], axis[1]), (axis[2], axis[2])
    coords = [CartesianIndex( cord ) for cord in coordtuples]
    cov = reshape(cov[ coords ], 2,2)
    eig = LinearAlgebra.eigen(cov)
    ordering = sortperm(eig.values, rev = true);
    eig.vectors .= eig.vectors[:,ordering];
    eig.values .= eig.values[ordering];
    theta = atan(eig.vectors[1,:][2], eig.vectors[1,:][1])
    #theta = atan(eig.vectors[axis[2],:][1], eig.vectors[axis[1],:][1])
    extent = 0.5 * (quantile(Normal(), 0.5 + confidence/2 ) - quantile(Normal(), 0.5 - confidence/2 ) ) * sqrt.(eig.values)
    Coords = zeros( pointestimate, 2 )
    time = (1 : pointestimate) .* ( ( 360.0 / pointestimate ) * ( pi / 180.0 ) )
    Coords[ :, 1 ] .= extent[1] .* cos.( time )
    Coords[ :, 2 ] .= extent[2] .* sin.( time )
    Rotated = ( [ [ cos(theta), -sin(theta) ] [ sin(theta), cos(theta) ] ] * Coords' )';
    Translate = Rotated .+ mean[axis]'
    return Translate
end

struct LogisticRegression
    Coefficients
    Biases
    CostPerIteration
end

function softmax(x)
  exponent = exp.(x)
  return exponent ./ sum(exponent)
end

"""
    MultinomialSoftmaxRegression(X, Y; LearnRate = 1e-3, maxiters = 1000, L2 = 0.0)

Returns a LogisticRegression classification model made by Stochastic Gradient Descent.
"""
function MultinomialSoftmaxRegression(X, Y; LearnRate = 1e-3, maxiters = 1000, L2 = 0.0)
    (Obs, ClassNumber) = size(Y)
    Vars = size(X)[2]
    Output = zeros( size( Y ) )
    #initialize weight vector(W) as zero. Most people call this theta, not W.
    W = randn(Vars, ClassNumber)
    B = zeros(1, 1)
    #For observing training error
    CostPerIt = zeros( maxiters )
    for it in 1 : maxiters
        for o in 1:Obs
            Output[o,:] = softmax( ( X[o,:]' * W ) .+ B )
            #Calculate the cost function - Not MSE. We are doing cross entropy loss
            CostPerIt[it] += -sum( Y[o,:] .* log.( Output[o,:] .+ 1e-6 ) ) + (L2 * sum(W .^ 2))
            residualsOutput = Y[o,:] .- Output[o,:]
            #calculate changes to be applied to the weights by these gradients and update them...
            B .+= LearnRate * [1.0] * sum(residualsOutput)
            W .+= LearnRate .* (X[o,:] * residualsOutput')
            if L2 > 0.0
                W .+= LearnRate .* ( (2.0 / L2) .* W)
            end
        end
        CostPerIt[it] /= Obs
    end
    return LogisticRegression(W, B, CostPerIt)
end

"""
    ( model::LogisticRegression )( X )

Returns a 1 hot encoded inference from `X` using a LogisticRegression object.
"""
( model::LogisticRegression )( X ) = softmax( (X * model.Coefficients) .+ model.Biases )

struct GaussianNaiveBayes
    TotalSamples::Int
    classcount::Int
    Priors
    Means::Array{Float64,2}
    Vars::Array{Float64,2}
end

"""
    GaussianNaiveBayes(X,Y)

Returns a GaussianNaiveBayes classification model object from `X` and one hot encoded `Y`.
"""
function GaussianNaiveBayes(X,Y)
    (obs, vars) = size(X)
    classes = size(Y)[2]
    ClasswisePrior = sum(Y, dims = 1) ./ obs
    ClasswiseMeans = zeros(classes, vars)
    ClasswiseVars = zeros(classes, vars)
    for c in 1 : classes
        ClassIndices = Y[:,c] .== 1
        ClasswiseMeans[c,:] = mean(X[ClassIndices,:], dims = 1)
        ClasswiseVars[c,:] = Statistics.var(X[ClassIndices,:], dims = 1)
    end
    return GaussianNaiveBayes(obs, classes, ClasswisePrior, ClasswiseMeans, ClasswiseVars)
end

Likelihood( x, mean, var ) = ( 1.0 ./ sqrt.( 2.0 * pi * var ) ) .* exp.( -0.5 .* ( ( (x .- mean) .^ 2.0 ) ./ var ) )

"""
    (gnb::GaussianNaiveBayes)(X)

Returns a 1 hot encoded inference from `X` using a GaussianNaiveBayes object.
"""
function (gnb::GaussianNaiveBayes)(X)
    (obs, vars) = size(X)
    Predictions = zeros(obs, gnb.classcount)
    for o in 1 : obs, c in 1 : gnb.classcount
        Predictions[o,c] = gnb.Priors[c] * prod( Likelihood( X[o,:], gnb.Means[c,:], gnb.Vars[c,:] ) )
    end
    return Predictions
end

struct linearperceptron{ a, b }
    W::a
    Loss::b
end

"""
    LinearPerceptron(X, Y; LearningRate = 1e-3, MaxIters = 5000)

Returns a batch trained LinearPerceptron classification model object from `X` and one hot encoded `Y`.
"""
function LinearPerceptronBatch(X, Y; LearningRate = 1e-3, MaxIters = 5000)
    W = zeros( size( X )[ 2 ], size( Y )[ 2 ] )
    Loss = zeros( MaxIters )
    for iter in 1 : MaxIters
        YHat = map(x -> (x > 0.0) ? 1.0 : 0.0, X * W)
        Err = Y .- YHat
        for obs in 1:size(X)[1]
            if argmax(YHat[obs,:]) == argmax(Y[obs,:])
                Err[obs,:] .= 0.0
            end
        end
        Loss[iter] = sum(Err .^ 2)
        W .+= LearningRate .* ( X' * Err )
    end
    return linearperceptron( W, Loss )
end

"""
    LinearPerceptronsgd(X, Y; LearningRate = 1e-3, MaxIters = 5000)

Returns a SGD trained LinearPerceptron classification model object from `X` and one hot encoded `Y`.
"""
function LinearPerceptronSGD(X, Y; LearningRate = 1e-3, MaxIters = 5000)
    W = zeros( size( X )[ 2 ], size( Y )[ 2 ] )
    Loss = zeros( MaxIters )
    Err = zeros(1, size(Y)[2])
    for iter in 1 : MaxIters, obs in 1:size(X)[1]
        #YHat = X[obs,:]' * W
        YHat = map(x -> (x > 0.0) ? 1.0 : 0.0, X[obs,:]' * W)
        Err = Y[obs,:]' .- YHat
        if argmax(YHat) == argmax(Y[obs,:])
            Err .= 0.0
        end
        Loss[iter] += sum(Err .^ 2)
        W .+= LearningRate .* ( X[obs,:] * Err )
    end
    return ChemometricsTools.linearperceptron( W, Loss )
end

"""
    (L::linearperceptron)(X)

Returns a 1 hot encoded inference from `X` using a LinearPerceptron object.
"""
(L::linearperceptron)(X) = X * L.W

struct SIMCA
    PCAs::Array{ Union{ PCA, LDA }, 1 }
    Factors::Array{Int,1}
    Preprocessers::Array
    Classes::Int
    ClassSizes::Array{Int,1}
    ClassLimits::Array{Float64,1}
end

"""
    SIMCA( X, Y;
            VarianceExplained = repeat([0.95], size(Y)[2]),
            Quantile = repeat([0.95], size(Y)[2]) )

Returns a Soft-Independent Modelling of Class Analogies(SIMCA) classification model
 object from `X` and one hot encoded `Y`.
"""
function SIMCA( X, Y;
                VarianceExplained = repeat([0.95], size(Y)[2]),
                Quantile = repeat([0.95], size(Y)[2]) )
    if length( VarianceExplained ) == 1
        VarianceExplained = repeat( [ VarianceExplained ], size( Y )[ 2 ] )
    end
    if length( Quantile ) == 1
        Quantile = repeat( [ Quantile ], size( Y )[ 2 ] )
    end
    (Obs, ClassNumber) = size( Y )
    Vars = size(X)[2]
    ClassSizes = Int.(zeros(ClassNumber))
    ClassLimits = zeros(ClassNumber)
    Factors = Int.(zeros(ClassNumber))
    residual_variance = zeros(ClassNumber)
    PCAs = []
    Preprocessers = []
    for class in 1 : ClassNumber
        Members = Y[ :, class ] .== 1
        ClassSizes[class] = sum(Members)
        @assert(ClassSizes[class] > 1, "Classes must have more than 1 member to create a PCA basis...")
        CS = Center(X[Members,:])
        Xcs = CS( X[Members,:] )
        pca = PCA( Xcs )
        factors = findfirst( cumsum(ExplainedVariance(pca)) .> VarianceExplained[class] )
        residual_variance = Xcs .- (pca(Xcs; Factors = factors) * pca.Loadings[1:factors,:])
        DOF = ( ClassSizes[class] - factors - 1 ) * ( Vars - factors )
        S0 = sum(residual_variance .^ 2) / DOF
        ClassLimits[class] = S0 * sqrt( quantile( Distributions.FDist( ( Vars - factors ) , DOF ), Quantile[class]) )
        Factors[class] = factors
        residual_variance[class] = S0
        push!( Preprocessers, CS )
        push!( PCAs, pca )
    end
    return SIMCA(PCAs, Factors, Preprocessers, ClassNumber, ClassSizes, ClassLimits )
end

"""
    (s::SIMCA)( X )

Returns a 1 hot encoded inference from `X` from a SIMCA model.
"""
function ( s::SIMCA )( X )
    Obs, Vars = size( X )
    Y = zeros( Obs, s.Classes )
    for class in 1 : s.Classes
        Xp = s.Preprocessers[ class ]( X )
        F = s.Factors[ class ]
        XProject = s.PCAs[ class ]( Xp ; Factors = F )
        residuals = Xp .- ( XProject * s.PCAs[ class ].Loadings[1:F,:] )
        Y[:, class] = sqrt.( sum( residuals .^ 2, dims = 2 ) ./ ( Vars - F ) )
        #Not in traditional method but convenient for modern classification formalism
        Y[ : , class ] .= (s.ClassLimits[class] .- Y[ : , class ])
        #Remove outliers by Q stat...
        Y[ Y[:, class] .< 0.0 , class ] .= 0.0
    end
    return Y
end


using Plots

dt = 0.01
g = 9.80665
s = 2
a = 1.2

function lander(x_0, v_0)
    x = x_0
    v = v_0
    t = 0
    xs = []
    ts = []
    while (x > 0)
        f = t >= s ? a : 0
        v += (g-f*g)*dt
        x -= v*dt
        t += dt
        push!(xs, x)
        push!(ts, t)
    end

    (xs, ts)
end
xs, ts = lander(100, 0)

plot(ts, xs, label="lander",xlabel="t",ylabel="x", color=:cornflowerblue)
savefig("lander.png")


include("splinesParallel.jl")

knots = [0,0,0,0,1,2,3,4,4,4,4]
ncontrols = 8
degree = 2
basis = pbsplineBasis(degree, knots, ncontrols)
point = [0.7,0.2,3.0]
basis(point)


using Test
using LogisticRegression: predict

W = ones(1, 1)
b = zeros(1)
y1 = predict(W, b, [-1.0])
y2 = predict(W, b, [ 1.0])
@test y1 == [0]
@test y2 == [1]


"""
    Tensor{k,label}

Abstract base for any tensor of rank `k`.
"""
abstract type Tensor{k,label} end

(::Type{T})(k::Int) where {T<:Tensor} = T{k}()

LinearAlgebra.rank(::Tensor{k}) where k = k

"""
    system(::Tensor)

A general tensor acts on the full system, i.e. all coordinates.
"""
system(::Type{<:Tensor}) = FullSystem()

couples(a::SpinOrbital{<:Orbital}, ::Type{T}, b::SpinOrbital{<:Orbital}) where {T<:Tensor} =
    isequal(other_quantum_numbers(system(T), a, b)...)

couples(a::SpinOrbital{<:RelativisticOrbital}, ::Type{T}, b::SpinOrbital{<:RelativisticOrbital}) where {T<:Tensor} =
    isequal(first.(other_quantum_numbers(system(T), a, b))...)

# This is only true for SO(3)
components(::Tensor{k}) where k = -k:k

function Base.show(io::IO, ::Tensor{k,label}) where {k,label}
    write(io,to_boldface(label)*"̂")
    write(io,"⁽",to_superscript(k),"⁾")
end

# * Tensor components

"""
    TensorComponent(tensor, q)

Represents the `q`th component of a `tensor`; `abs(q) ≤ rank(tensor)`.
"""
struct TensorComponent{T<:Tensor}
    tensor::T
    q::Int
    function TensorComponent(tensor::T, q::Int) where {T<:Tensor}
        k = rank(tensor)
        q ∈ components(tensor) ||
            throw(ArgumentError("Tensor component $(q) not possible for tensor $(tensor) of rank $k"))
        new{T}(tensor, q)
    end
end

function Base.show(io::IO, Tq::TensorComponent)
    show(io, Tq.tensor)
    write(io, to_subscript(Tq.q))
end

Base.parent(Tᵏq::TensorComponent) = Tᵏq.tensor
component(Tᵏq::TensorComponent) = Tᵏq.q

system(::TensorComponent{T}) where T = system(T)

Base.hash(Tq::TensorComponent, h::UInt) = hash(Tq.tensor, hash(Tq.q, h))

# * Tensor products

@doc raw"""
    TensorProduct{K}(T, U)

A tensor of rank `K` formed from the product of the tensors `T` and
`U`, according to

```math
\begin{equation}
\tag{V3.1.20}
\tensor{X}^{(K)}_Q \equiv
\{\tensor{T}^{(k_1)}\tensor{U}^{(k_2)}\}^{(K)}_Q \defd
\tensor{T}^{(k_1)}_{q_1}
\tensor{U}^{(k_2)}_{q_2}
C_{k_1q_1k_2q_2}^{KQ}
\end{equation}
```
"""
struct TensorProduct{K,A<:Tensor,B<:Tensor} <: Tensor{K,'X'}
    T::A
    U::B
end
TensorProduct(K::Int, T::A, U::B) where {A<:Tensor, B<:Tensor} =
    TensorProduct{K,A,B}(T, U)

function Base.show(io::IO, X::TensorProduct{K}) where K
    write(io,"{")
    show(io, X.T)
    show(io, X.U)
    write(io,"}⁽",to_superscript(K),"⁾")
end

system(::TensorProduct{<:Any,A,B}) where {A,B} = (system(A), system(B))

# ** Scalar product

@doc raw"""
    TensorScalarProduct(T, U)

A tensor of rank 0 (and thus implicitly 0 projection) formed from the
product of the tensors `T` and `U` (which have to have the same rank),
according to

```math
\begin{equation}
\tag{V3.1.30,35}
(\tensor{T}^{(k)} \cdot
\tensor{U}^{(k)}) \defd
(-)^k\angroot{k}
\{\tensor{T}^{(k)}
\tensor{U}^{(k)}\}^{(0)}_0 \equiv
(-)^q
\tensor{T}^{(k)}_{q}
\tensor{U}^{(k)}_{-q}
\end{equation}
```
"""
struct TensorScalarProduct{A<:Tensor,B<:Tensor} <: Tensor{0,'X'}
    T::A
    U::B
    function TensorScalarProduct(T::A, U::B) where {A<:Tensor,B<:Tensor}
        rank(T) == rank(U) ||
            throw(ArgumentError("Cannot form scalar product of tensors of differing ranks"))
        new{A,B}(T,U)
    end
end

function Base.show(io::IO, X::TensorScalarProduct)
    write(io,"(")
    show(io, X.T)
    write(io,"⋅")
    show(io, X.U)
    write(io,")")
end

"""
    dot(T::Tensor, U::Tensor)

Form the scalar product of the two tensors `T` and `U`, which need to
have the same rank.

# Examples

```jldoctest
julia> SphericalTensor(4)⋅SphericalTensor(4)
(𝐂̂⁽⁴⁾⋅𝐂̂⁽⁴⁾)
```
"""
LinearAlgebra.dot(T::Tensor, U::Tensor) =
    TensorScalarProduct(T, U)

system(::TensorScalarProduct{A,B}) where {A,B} = (system(A), system(B))

# * Linear combination of tensors

"""
    LinearCombinationTensor

Represents a linear combination of tensor components.
"""
const LinearCombinationTensor{T<:Tensor,N<:Number} = LinearCombination{<:TensorComponent{<:T},N}

@linearly_combinable TensorComponent

export Tensor, TensorComponent,
    TensorProduct, TensorScalarProduct,
    system


"""
DFS
---
    compute depth first search distances and returns the distance (d), the discover (dt),
    the finish time(ft), and the predecessor array (pred) in the tuple (d,dt,ft, pred).\n
    pred[i] = 0 if vertex i is in a component not reachable from u and i != u.

Functions
---------
- (d,dt,ft,pred) = dfs(A::MatrixNetwork,u::Int64,full::Int64,target::Int64)
- (d,dt,ft,pred) = dfs(A::MatrixNetwork,u::Int64)
- (d,dt,ft,pred) = dfs{T}(A::SparseMatrixCSC{T,Int64},u::Int64,full::Int64,target::Int64)
- (d,dt,ft,pred) = dfs{T}(A::SparseMatrixCSC{T,Int64},u::Int64)
- (d,dt,ft,pred) = dfs(ei::Vector{Int64},ej::Vector{Int64},u::Int64,full::Int64,target::Int64)
- (d,dt,ft,pred) = dfs(ei::Vector{Int64},ej::Vector{Int64},u::Int64)

Example
-------
~~~
A = load_matrix_network("dfs_example")
(d,dt,ft,pred)  = dfs(A,1)
~~~
"""

function dfs(A::MatrixNetwork,u::Int64,full::Int64,target::Int64)

    (rp,ci) = (A.rp,A.ci)
    n=length(rp)-1
    d=-1*ones(Int64,n)
    dt=-1*ones(Int64,n)
    ft=-1*ones(Int64,n)
    pred=zeros(Int64,n)
    rs=zeros(Int64,2*n)
    rss=0 # recursion stack holds two nums (v,ri)

    #start dfs at u
    t=0
    targethit=0
    for i=1:n
        if i==1
            v=u
        else
            v=mod(u+i-1,n)+1
            if d[v]>0
                continue
            end
        end
        d[v]=0
        dt[v]=t
        t=t+1
        ri=rp[v]
        rss=rss+1
        rs[2*rss-1]=v
        rs[2*rss]=ri # add v to the stack

        while rss>0
            v=rs[2*rss-1]
            ri=rs[2*rss]
            rss=rss-1 # pop v from the stack
            if v==target || targethit == 1
                ri=rp[v+1]
                targethit=1 # end the algorithm if v is the target
            end
            while ri<rp[v+1] #Traversing thru all its adjancencies
                w=ci[ri]
                ri=ri+1
                if d[w]<0
                    d[w]=d[v]+1
                    pred[w]=v
                    rss=rss+1
                    rs[2*rss-1]=v
                    rs[2*rss]=ri # add v to the stack
                    v=w
                    ri=rp[w]
                    dt[v]=t
                    t=t+1
                    continue #discover a new vertex!
                end
            end
            ft[v]=t
            t=t+1 # finish with v
        end
        if full == 0
            break
        end
    end
    return (d,dt,ft,pred)
end

############################
### Additional functions ###
############################

function dfs(A::MatrixNetwork,u::Int64)
    return dfs(A,u,0,0);
end

## CSC sparse matrices:
function dfs{T}(A::SparseMatrixCSC{T,Int64},u::Int64)
    return dfs(MatrixNetwork(A),u)
end

function dfs{T}(A::SparseMatrixCSC{T,Int64},u::Int64,full::Int64,target::Int64)
    return dfs(MatrixNetwork(A),u,full,target)
end

## Triplet Format:
function dfs(ei::Vector{Int64},ej::Vector{Int64},u::Int64)
    return dfs(MatrixNetwork(ei,ej),u)
end

function dfs(ei::Vector{Int64},ej::Vector{Int64},u::Int64,full::Int64,target::Int64)
    return dfs(MatrixNetwork(ei,ej),u,full,target)
end

## CSR sparse matrices:
function dfs{T}(rp::Vector{Int64},ci::Vector{Int64},vals::Vector{T},n::Int64,u::Int64)
    return dfs(MatrixNetwork(n,rp,ci,vals),u)
end

function dfs{T}(rp::Vector{Int64},ci::Vector{Int64},vals::Vector{T},n::Int64,u::Int64,full::Int64,target::Int64)
    return dfs(MatrixNetwork(n,rp,ci,vals),u,full,target)
end


function diag_davidson( pw::PWGrid, Vpot, X0;
                        tol=1e-5, tol_avg=1e-7, NiterMax=200, verbose=false )

    # get size info
    Ncols = size(X0)[2]
    Ngwx  = size(X0)[1]

    if Ncols <= 0
        @printf("diag_davidson requires at least one initial wave function!\n");
        return
    end
    # orthonormalize the initial wave functions.
    X = ortho_gram_schmidt(X0)  # normalize (again)?

    evals    = zeros(Float64, Ncols)
    R        = zeros(ComplexF64, Ngwx, Ncols)
    Hred     = zeros(ComplexF64, 2*Ncols, 2*Ncols)
    Sred     = zeros(ComplexF64, 2*Ncols, 2*Ncols)
    res      = zeros(Float64, Ncols)
    res_norm = zeros(Float64, Ncols)

    HX = op_H( pw, Vpot, X )

    # Initial eigenvalues
    for ic = 1:Ncols
        for ig = 1:Ngwx
            evals[ic] = evals[ic] + real( conj(X[ig,ic]) * HX[ig,ic] )
        end
    end

    # Calculate residuals
    for ic = 1:Ncols
        for ig = 1:Ngwx
            R[ig,ic] = evals[ic]*X[ig,ic] - HX[ig,ic]
        end
    end

    for ic = 1:Ncols
        res[ic] = 0.0
        for ig = 1:Ngwx
            res[ic] = res[ic] + real( R[ig,ic] * conj(R[ig,ic]) )
        end
        res[ic] = sqrt( res[ic] )
    end

    EPS = eps()

    set1 = 1:Ncols
    set2 = Ncols+1:2*Ncols

    sum_evals = sum(evals)
    sum_evals_old = sum_evals

    for iter = 1:NiterMax

        res_norm[:] .= 1.0

        for ic = 1:Ncols
            if EPS < res[ic]
                res_norm[ic] = 1.0/res[ic]
            end
        end

        for ic = 1:Ncols
            for ig = 1:Ngwx
                R[ig,ic] = res_norm[ic] * R[ig,ic]
            end
        end

        R = Kprec(pw, R)

        HR = op_H( pw, Vpot, R )

        # FIXME: Pull this outside the loop?
        if iter == 1
            Hred[set1,set1] = X' * HX
        else
            Hred[1:Ncols,1:Ncols] .= 0.0 + im*0.0
            for ic = 1:Ncols
                Hred[ic,ic] = evals[ic] + im*0.0 # use diagm ?
            end
            #Hred[set1,set1] = diagm(evals[:])
        end

        Hred[set1,set2] = X' * HR
        Hred[set2,set2] = R' * HR
        Hred[set2,set1] = Hred[set1,set2]'

        Sred[set1,set1] = Matrix(Diagonal(ones(ComplexF64,Ncols)))
        Sred[set1,set2] = X' * R
        Sred[set2,set2] = R' * R
        Sred[set2,set1] = Sred[set1,set2]'

        Hred = (Hred + Hred')/2.0
        Sred = (Sred + Sred')/2.0

        λ_red, X_red = eigen( Hermitian(Hred), Hermitian(Sred) )

        evals = λ_red[1:Ncols]

        #for ic = 1:2*Ncols
        #  print(ic); print(" "); println(λ_red[ic])
        #end

        X  = X  * X_red[set1,set1] + R  * X_red[set2,set1]
        HX = HX * X_red[set1,set1] + HR * X_red[set2,set1]

        # Calculate residuals
        for ic = 1:Ncols
            for ig = 1:Ngwx
                R[ig,ic] = evals[ic]*X[ig,ic] - HX[ig,ic]
            end
        end

        for ic = 1:Ncols
            res[ic] = 0.0
            for ig = 1:Ngwx
                res[ic] = res[ic] + real( R[ig,ic] * conj(R[ig,ic]) )
            end
            res[ic] = sqrt( res[ic] )
            @printf("%4d %18.10f %18.10f\n", ic, evals[ic], res[ic] )
        end

        sum_evals = sum(evals)
        diffSum = abs(sum_evals - sum_evals_old)
        @printf("iter %d tol_avg = %18.10e\n", iter, abs(diffSum))
        if diffSum < tol_avg
            if verbose
                @printf("Davidson convergence: tol_avg\n")
            end
            break
        end
        sum_evals_old = sum_evals
    end

    return evals, X

end


"""
    Exo1Binion{T <: Real} = Parabolic{Binion{T}}

A 1-exo-binion is a parabolic Cayley-Dickson construct with binions.
"""
const Exo1Binion{T <: Real} = Parabolic{Binion{T}}

function Exo1Binion(z::Binion{T}) where T <: Real
    Exo1Binion{T}(z, zero(Binion{T}))
end

function Exo1Binion(a::T, b::T, c::T, d::T) where T <: Real
    Exo1Binion{T}(Binion{T}(a, b), Binion{T}(c, d))
end

function Exo1Binion(a::T, b::T, c::T) where T <: Real
    Exo1Binion{T}(Binion{T}(a, b), Binion{T}(c, zero(T)))
end

function Exo1Binion(a::T, b::T) where T <: Real
    Exo1Binion{T}(Binion{T}(a, b), zero(Binion{T}))
end

function Exo1Binion(a::T) where T <: Real
    Exo1Binion{T}(Binion{T}(a, zero(T)), zero(Binion{T}))
end

function Exo1Binion(a::Real, b::Real)
    Exo1Binion(promote(a, b)...)
end

function Exo1Binion(a::Real, b::Real, c::Real)
    Exo1Binion(promote(a, b, c)...)
end

function Exo1Binion(a::Real, b::Real, c::Real, d::Real)
    Exo1Binion(promote(a, b, c, d)...)
end

function Exo1Binion(z::Exo1Real{T}) where T <: Real
    Exo1Binion(z.l, zero(T), z.r, zero(T))
end

function selfstar(z::Exo1Binion)
    (z + star(z)) / 2
end

function anti_selfstar(z::Exo1Binion)
    (z - star(z)) / 2
end

"""
    unreal(z::Exo1Binion)

The unreal part of a 1-exo-binion is a 3-vector.
"""
function unreal(z::Exo1Binion)
    vcat(unreal(z.l), array(z.r))
end

function show(io::IO, z::Exo1Binion)
    print(io, "[1: ")
    print(io, z.l.l)
    print(io, ", i: ")
    print(io, z.l.r)
    print(io, ", a: ")
    print(io, z.r.l)
    print(io, ", ia: ")
    print(io, z.r.r)
    print(io, "]")
end

function random(::Type{Exo1Binion{T}}) where T <: Real
    Exo1Binion{T}(random(Binion{T}), random(Binion{T}))
end


# Initialize environment in current directory
@info("Ensuring example environment instantiated...")
import Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()

@info("Loading Zygote...")
using Zygote, LinearAlgebra

# This example will showcase how we do a simple linear fit with Zygote, making
# use of complex datastructures, a home-grown stochastic gradient descent
# optimizer, and some good old-fashioned math.  We start with the problem
# statement:  We wish to learn the mapping `f(X) -> Y`, where `X` is a matrix
# of vector observations, `f()` is a linear mapping function and `Y` is a
# vector of scalar observations.

# Because we like complex objects, we will define our linear regression as the
# following object:
mutable struct LinearRegression
    # These values will be implicitly learned
    weights::Matrix
    bias::Float64

    # These values will not be learned
    name::String
end
LinearRegression(nparams, name) = LinearRegression(randn(1, nparams), 0.0, name)

# Our linear prediction looks very familiar; w*X + b
function predict(model::LinearRegression, X)
    return model.weights * X .+ model.bias
end

# Our "loss" that must be minimized is the l2 norm  between our current
# prediction and our ground-truth Y
function loss(model::LinearRegression, X, Y)
    return norm(predict(model, X) .- Y, 2)
end

# Our "ground truth" values (that we will learn, to prove that this works)
weights_gt = [1.0, 2.7, 0.3, 1.2]'
bias_gt = 0.4

# Generate a dataset of many observations
X = randn(length(weights_gt), 10000)
Y = weights_gt * X .+ bias_gt

# Add a little bit of noise to `X` so that we do not have an exact solution,
# but must instead do a least-squares fit:
X .+= 0.001.*randn(size(X))

# Now we begin our "training loop", where we take examples from `X`,
# calculate loss with respect to the corresponding entry in `Y`, find the
# gradient upon our model, update the model, and continue.  Before we jump
# in, let's look at what `Zygote.gradient()` gives us:
@info("Building model...")
model = LinearRegression(size(X, 1), "Example")

# Calculate gradient upon `model` for the first example in our training set
@info("Calculating gradient (the first time can take a while to compile...)")
grads = Zygote.gradient(model) do m
    return loss(m, X[:,1], Y[1])
end

# The `grads` object is a Tuple containing one element per argument to
# `gradient()`, so we take the first one to get the gradient upon `model`:
grads = grads[1]

# Because our LinearRegression object is mutable, the gradient holds a
# reference to it, which we peel via `grads[]`:
grads = grads[]

# We now get a `NamedTuple` so we can now do things like `grads.weight`. Let's
# print it out, just to see what it looks like.  Note that while `weights` and
# `bias` have gradients, `name` just naturally has a  gradient of `nothing`,
# because it was not involved in the calculation of the output loss.
@info grads

# Let's define an update rule that will allow us to modify the weights
# of our model a tad bit according to the gradients
function sgd_update!(model::LinearRegression, grads, η = 0.001)
    model.weights .-= η .* grads.weights
    model.bias -= η * grads.bias
end

# Now let's do that for each example in our training set:
@info("Running train loop for $(size(X,2)) iterations")
for idx in 1:size(X, 2)
    grads = Zygote.gradient(m -> loss(m, X[:, idx], Y[idx]), model)[1][]
    sgd_update!(model, grads)
end

# Now let's look at how well we've approximated the ground truth weights/bias:
@info("Ground truth weights: $(weights_gt)")
@info("Learned weights: $(round.(model.weights; digits=3))")
@info("Ground truth bias: $(bias_gt)")
@info("Learned bias: $(round(model.bias; digits=3))")


"""
    to_distribution(x)

Convert to a `Uniform` distribution from an `Interval`.
"""
to_distribution(x::Interval) = Uniform(x.lo, x.hi)
to_distribution(x::Distribution) = x
to_distribution(x::ReachInterval) = to_distribution(x.dat)
to_distribution(x) = x

"""
    to_interval(x)

Convert to a `Interval` from a `Distribution`.
"""
to_interval(x::Interval) = x
to_interval(x::Distribution) = Interval(_minimum(x), _maximum(x))
to_interval(x::ReachInterval) = x.dat
to_interval(x) = x

"""
    to_reach_interval(x)

Convert to a `ReachInterval` from an `Interval` or `Distribution`.
"""
to_reach_interval(x::Interval) = ReachInterval(x)
to_reach_interval(x::Distribution) = to_reach_interval(to_interval(x))
to_reach_interval(x::ReachInterval) = x
to_reach_interval(x) = RA.Singleton([x])
to_reach_interval(x::Integer) = to_reach_interval(float(x))

"""
    recursive_convert(x)

Recursively convert a variable `x` by applying a conversion function `f`.
Mostly to be used  with `to_distribution` and `to_interval`.
"""
recursive_convert(f, x) = f(x)
recursive_convert(f, x::NamedTuple) = map(x->recursive_convert(f, x), x)
recursive_convert(f, x::AbstractArray) = recursive_convert.(f, x)

# Internal maximum and minimum to work across all uncertainty types
_minimum(x) = minimum(x)
_minimum(x::Interval) = inf(x)
_minimum(x::ReachInterval) = min(x)

_maximum(x) = maximum(x)
_maximum(x::Interval) = sup(x)
_maximum(x::ReachInterval) = max(x)


function parse_line(line)
    id, x, y, w, h = parse.(Int, match(r"#(\d+) @ (\d+),(\d+): (\d+)x(\d+)", line).captures)
    (id, x .+ (1:w), y .+ (1:h))
end

input = parse_line.(readlines("03.input"))

cloth = zeros(Int, 1000, 1000)
for (id, xrange, yrange) in input
    cloth[xrange, yrange] .+= 1
end

# part 1: number of squares with >1 claim
@show sum(c > 1 for c in cloth)

# part 2: find only claim that has no intersection
# (re-use the "cloth" marked up for the first star to make this O(n))

@btime [id
        for (id, xrange, yrange)
        in input
        if all(isequal(1), view(cloth, xrange, yrange))]
#   452.265 μs (5281 allocations: 185.72 KiB)
# 1-element Array{Int64,1}:
#  724

@btime first(id
             for (id, xrange, yrange)
             in input
             if all(isequal(1), view(cloth, xrange, yrange)))
#   244.011 μs (2899 allocations: 101.86 KiB)
# 724


# ------------------------------------------------------------------------------------------
# # Julia 很快
#
# 基准数值测试程序（benchmarks）通常被用来比较编程语言的性能。
# 这些 benchmarks 可能引起很多的讨论，首先是作为基准测试的到底是什么，再有就是究竟是什么导致不同语言间的差异。
# 这些简单的问题有时比你乍想之下复杂得多。
#
# 这个 notebook 目的在于让你实现一个简单的基准测试。
# 通过这个 notebook 可以看到作者的配有四核 Macbook Pro 表现如何，或者你可以自己跑跑看。
#
# （材料来自于 MIT 的 Steven Johnson 的一场演讲：[18S096/Boxes-and-registers.ipynb at master ·
# mitmath/18S096](https://github.com/mitmath/18S096/blob/master/lectures/lecture1/Boxes-and-
# registers.ipynb)）
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# ## 目录
#
# - [`sum` 函数的定义](#sum:-一个简单易懂的函数)
# - [不同语言中的各种基准测试](#不同语言中的各种基准测试)
#     1. [C (手写)](#1.-C-语言)
#     2. [C (手写 + `-ffast-math`)](#2.-使用--ffast-math-的-C-语言)
#     3. [python (内置)](#3.-Python-的内置-sum)
#     4. [python (numpy)](#4.-Python:-numpy)
#     5. [python (手写)](#5.-Python（手写）)
#     6. [Julia (内置)](#6.-Julia（内置）)
#     7. [Julia (手写)](#7.-Julia（手写）)
#     8. [Julia (手写 + SIMD)](#8.-Julia（手写-+--@simd）)
# - [性能测试小结](#性能测试小结)
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# ## `sum`: 一个简单易懂的函数
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# 考虑**求和**函数 `sum(a)`，它是用来计算
# $$
# \mathrm{sum}(a) = \sum_{i=1}^n a_i
# $$
# 其中 $n$ 是 `a` 的长度。
# ------------------------------------------------------------------------------------------

a = rand(10^7)  # 在[0,1)上均匀分布的随机数组成的1维向量

sum(a)

# ------------------------------------------------------------------------------------------
# 预期结果应该在 $0.5 \times 10^7$ 左右，因为所有元素的平均值应该是 0.5 左右
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# ## 不同语言中的各种基准测试
# ------------------------------------------------------------------------------------------

@time sum(a)

@time sum(a)

@time sum(a)

# ------------------------------------------------------------------------------------------
# `@time` 宏会产生受干扰的结果，所以不是做基准测试的最佳选择！
#
# 好在 Julia 有个 `BenchmarkTools.jl` 包来简单并准确地进行基准测试：
# ------------------------------------------------------------------------------------------

using Pkg
Pkg.add("BenchmarkTools")

using BenchmarkTools

# ------------------------------------------------------------------------------------------
# ###  1. C 语言
#
# C 语言对人类不友好但是对机器友好，经常作为一个黄金标准。
# 在 C 语言的两倍以内通常来说就比较理想了。
# 然而，即使仅考虑 C 语言，不同的 C 语言程序员也可能采用或不采用不同的优化方式。
#
# 原作者不会 C 语言，所以他看不懂下面这个代码块中的代码，
# 但是他知道 Julia 中可以编译、运行 C 语言代码。注意 `"""` 所包围的是个多行字符串。
# ------------------------------------------------------------------------------------------

using Libdl
C_code = """
#include <stddef.h>
double c_sum(size_t n, double *X) {
    double s = 0.0;
    for (size_t i = 0; i < n; ++i) {
        s += X[i];
    }
    return s;
}
"""

const Clib = tempname()  # 创建一个临时文件

# 通过将 C_code 导入 gcc 来编译成共享链接库
# (只有装了 gcc 才行):

open(`gcc -fPIC -O3 -msse3 -xc -shared -o $(Clib * "." * Libdl.dlext) -`, "w") do f
    print(f, C_code)
end

# 定义一个调用这个 C 语言函数的 Julia 函数：
c_sum(X::Array{Float64}) = ccall(("c_sum", Clib), Float64, (Csize_t, Ptr{Float64}), length(X), X)

c_sum(a)

c_sum(a) ≈ sum(a)  # 输入\approx后按<TAB>键得到≈符号

c_sum(a) - sum(a)

≈  # `isapprox` 函数的别名

?isapprox

# ------------------------------------------------------------------------------------------
# 现在我们可以在 Julia 中直接对 C 语言代码进行基准测试了：
# ------------------------------------------------------------------------------------------

c_bench = @benchmark c_sum($a)

println("C: 最快 $(minimum(c_bench.times) / 1e6) 毫秒")

d = Dict()  # 字典，也就是关联数组（键值对）
d["C"] = minimum(c_bench.times) / 1e6  # 毫秒
d

using Plots
gr()

using Statistics  # 导入统计包以计算标准差
t = c_bench.times / 1e6  # 毫秒
m, σ = minimum(t), std(t)

histogram(t, bins=500,
    xlim=(m - 0.01, m + σ),
    xlabel="milliseconds", ylabel="count", label="")

# ------------------------------------------------------------------------------------------
# ### 2. 使用 `-ffast-math` 的 C 语言
#
# 如果我们允许 C 重新安排浮点操作，它就会使用 SIMD 指令(single instruction, multiple data)进行矢量化。
# ------------------------------------------------------------------------------------------

const Clib_fastmath = tempname()   # 创建临时文件

# 同上但是增加了 -ffast-math 标志
open(`gcc -fPIC -O3 -msse3 -xc -shared -ffast-math -o $(Clib_fastmath * "." * Libdl.dlext) -`, "w") do f
    print(f, C_code)
end

# 定义一个调用这个 C 语言函数的 Julia 函数：
c_sum_fastmath(X::Array{Float64}) = ccall(("c_sum", Clib_fastmath), Float64, (Csize_t, Ptr{Float64}), length(X), X)

c_fastmath_bench = @benchmark $c_sum_fastmath($a)

d["C -ffast-math"] = minimum(c_fastmath_bench.times) / 1e6  # 毫秒

# ------------------------------------------------------------------------------------------
# ### 3. Python 的内置 `sum`
# ------------------------------------------------------------------------------------------

# ------------------------------------------------------------------------------------------
# `PyCall` 包给 Julia 提供了 Python 接口：
# ------------------------------------------------------------------------------------------

using Pkg; Pkg.add("PyCall")
using PyCall

# 获得 Python 的内置 "sum" 函数：
pysum = pybuiltin("sum")

pysum(a)

pysum(a) ≈ sum(a)

py_list_bench = @benchmark $pysum($a)

d["Python built-in"] = minimum(py_list_bench.times) / 1e6
d

# ------------------------------------------------------------------------------------------
# ### 4. Python: `numpy`
#
# **条件允许的情况下利用硬件 "SIMD"**
#
# `numpy` 是一个 Python 的优化过的 C 语言库。
# 在 Julia 版 Python 中的安装方法为：
# ------------------------------------------------------------------------------------------

using Pkg; Pkg.add("Conda")
using Conda

Conda.add("numpy")

numpy_sum = pyimport("numpy")["sum"]

py_numpy_bench = @benchmark $numpy_sum($a)

numpy_sum(a)

numpy_sum(a) ≈ sum(a)

d["Python numpy"] = minimum(py_numpy_bench.times) / 1e6
d

# ------------------------------------------------------------------------------------------
# ### 5. Python（手写）
# ------------------------------------------------------------------------------------------

py"""
def py_sum(A):
    s = 0.0
    for a in A:
        s += a
    return s
"""

sum_py = py"py_sum"

py_hand = @benchmark $sum_py($a)

sum_py(a)

sum_py(a) ≈ sum(a)

d["Python hand-written"] = minimum(py_hand.times) / 1e6
d

# ------------------------------------------------------------------------------------------
# ### 6. Julia（内置）
#
# **是直接用 Julia 写的，不是 C！**
# ------------------------------------------------------------------------------------------

@which sum(a)

j_bench = @benchmark sum($a)

d["Julia built-in"] = minimum(j_bench.times) / 1e6
d

# ------------------------------------------------------------------------------------------
# ### 7. Julia（手写）
# ------------------------------------------------------------------------------------------

function mysum(A)
    s = 0.0  # s = zero(eltype(a))
    for a in A
        s += a
    end
    s
end

j_bench_hand = @benchmark mysum($a)

d["Julia hand-written"] = minimum(j_bench_hand.times) / 1e6
d

# ------------------------------------------------------------------------------------------
# ### 8. Julia（手写 +  `@simd`）
# ------------------------------------------------------------------------------------------

function mysum_simd(A)
    s = 0.0  # s = zero(eltype(A))
    @simd for a in A
        s += a
    end
    s
end

j_bench_hand_simd = @benchmark mysum_simd($a)

mysum_simd(a)

d["Julia hand-written simd"] = minimum(j_bench_hand_simd.times) / 1e6
d

# ------------------------------------------------------------------------------------------
# ## 性能测试小结
# ------------------------------------------------------------------------------------------

for (key, value) in sort(collect(d), by=last)
    println(rpad(key, 25, "."), lpad(round(value; digits=1), 6, "."))
end