### A Pluto.jl notebook ###
# v0.17.3

using Markdown
using InteractiveUtils

# ╔═╡ 3247c742-65df-11ec-0eed-7d76c7786941
begin
	using Pkg; Pkg.activate(@__DIR__); Pkg.instantiate()
	Pkg.precompile()

	using PlutoUI
	using GLMNet
	using RDatasets
	using MLBase
	using Plots
	using DecisionTree
	using Distances
	using NearestNeighbors
	using Random
	using LinearAlgebra
	using DataStructures
	using LIBSVM
end;

# ╔═╡ ff42960a-f27f-41b8-8eee-14fb5b0454ba
PlutoUI.TableOfContents(aside=true, indent=true, depth=3)

# ╔═╡ b4c7b20b-ef19-4684-933b-7a00bfb69dce
html"""

"""

# ╔═╡ 73a1eed0-af5b-42b3-a4b8-a6f191f49daa
md"""
# 🔧 Classification

Put simply, classification is the task of predicting a label for a given observation. For example: you are given certain physical descriptions of an animal, and your taks is to classify them as either a dog or a cat. Here, we will classify iris flowers.

[YouTube Link](https://www.youtube.com/watch?v=OQRPeIQasdo)

As we will see later, we will use different classifiers and at the end of this notebook, we will compare them. We will define our accuracy function right now to get it out of the way. We will use a simple **accuracy** function that returns the ratio of the number of correctly classified observations (true positives) to the total number of predictions.
"""

# ╔═╡ 68c8634d-a052-4bd0-a101-5111dd9d4f68
findaccuracy(ŷ, y) = sum(ŷ .== y) / length(y)

# ╔═╡ 091524fe-ad86-45a3-ad25-370fe25a7a4e
md"""
## Load data

We will use Iris dataset from [RDatasets.jl](https://github.com/JuliaStats/RDatasets.jl) package. It is already stored as a `DataFrame`.
"""

# ╔═╡ 7a1a6481-eb71-4eb8-b47e-cf65c2c4cafa
iris = dataset("datasets", "iris")

# ╔═╡ 0bca5475-72f2-4189-b3d8-dd06cc951fc8
md"""
## Data wrangling

First, let's define our feature matrix `X`...
"""

# ╔═╡ e7f4e859-5e3c-439d-b069-46f17f2acf02
X = Matrix(iris[:, 1:4])

# ╔═╡ 0c1f6c24-ab74-494a-89d8-ba3fbda7fa24
md"""
And our label vector `labels`...
"""

# ╔═╡ b086ca45-6a60-4d4e-8387-de7e247812ed
labels = iris[:, 5]

# ╔═╡ c495a8f7-17d0-42f7-9adf-440d83d310c1
md"""
Since our dependent variable is categorical, let's encode it using the [MLBase.jl](https://github.com/JuliaStats/MLBase.jl) package...

**Note:** some algorithms require numeric labels rather than string labels.
"""

# ╔═╡ 35fad54f-e861-46e7-ab43-04a205c1991f
# mapping labels
irismap = labelmap(labels)

# ╔═╡ df121f29-a69d-42c9-b6a7-f129c4e94506
# enconding labels vector
y = labelencode(irismap, labels)

# ╔═╡ 1f39a7ed-7320-4f72-b4a0-fac44a71bb75
md"""
## Train/test split

In classification, we often want to use some of the data to fit a model (i.e. _training data_), and the rest of the data to validate (i.e. _testing data_). We will get this data ready now, so that we can easily use it in the rest of this notebook.

We will create the `train_test_split` function to perform train/test split. It will return the `y_train` indexes...

**Note:** the following `train_test_split` function performs a stratified random sampling by class.
"""

# ╔═╡ ecb46a59-1acf-48b9-b3c7-4c319388bc33
function train_test_split(y::Array{Int64,1}, train_ratio::Float64)
	classes = unique(y)
	y_train_idx = []
	for class in classes
		idxs = findall(y .== class)
		selected_idxs = randsubseq(idxs, train_ratio)
		push!(y_train_idx, selected_idxs...)
	end

	return y_train_idx
end

# ╔═╡ 8ca957e5-1bec-4498-aaca-86dccca1a49b
md"""
**Note:** `randsubseq(A, p)` function returns a vector consisting of a random subsequence of the given array `A`, where each element of `A` is included (in order) with independent probability `p`. Technically, this process is known as _Bernoulli sampling_ of `A`.
"""

# ╔═╡ 8868df3e-5933-4670-bbaf-eb8be100382b
# y train indexes
train_idxs = train_test_split(y, 0.7)

# ╔═╡ 27f97d52-c5c2-41a3-9faf-a8271a404b23
# y test indexes
test_idxs = setdiff(1:length(y), train_idxs)

# ╔═╡ 95a6bc1e-7986-4d86-9843-ceac5b668e4c
md"""
We will need one more function, and that is the function that will assign classes based on the predicted values when the predicted values are continuous.

**Example:** suppose that we have $\hat{y}_i = 0.95$. This is a continuous prediction and we may convert it into $1$ (i.e. _setosa_).
"""

# ╔═╡ 137c8d77-d6fd-44eb-937e-5978d201670d
assign_class(ŷᵢ) = argmin(abs.(ŷᵢ .- [1,2,3]))

# ╔═╡ 6317039e-1926-4143-a4d6-61f61b5e83e0
md"""
## Data modeling

We are going to build 7 different classification models:
1. Lasso
2. Ridge
3. Elastic Net
4. Decision Trees
5. Random Forests
6. Nearest Neighbour
7. Support Vector Machine

Then, we will compare model accuracies to figure out which models have the best performance.
"""

# ╔═╡ 1bd96a25-9542-432b-8e54-d6896a9cdade
md"""
### Lasso

**Least Absolute Shrinkage and Selection Operator (Lasso)** is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. It was originally introduced in Geophysics, and later by Robert Tibshirani who coined the term.

Lasso was originally formulated for linear regression models. This simple case reveals a substantial amount about the estimator. These include its relationship to Ridge and best subset selection and the connections between Lasso coefficient estimates and so-called _soft thresholding_. It also reveals that (like standard linear regression) the coefficient estimates do not need to be unique if covariates (independent variables) are collinear.

**Note:** check [here](https://en.wikipedia.org/wiki/Lasso_(statistics)) for more math details.

For Lasso, Ridge and Elastic Net methods, we are going to use the [GLMNet.jl](https://github.com/JuliaStats/GLMNet.jl) package.

We use the `glmnet` function, we will get several different solutions and $\lambda$ values (weighting parameter for the norm factor). Thus, we may use the `glmnetcv` function to get the _best_ $\lambda$ value (i.e. the $\lambda$ value with the lowest mean loss).

**Note:** to perform Lasso, we must set `alpha=1`, which is the default `alpha` value for the `glmnet` function.
"""

# ╔═╡ dd6889f6-18da-4218-bc48-b6589699498a
# Lasso solutions
lasso_sol = glmnet(X[train_idxs, :], y[train_idxs])

# ╔═╡ d03a03d1-fc97-4cb3-a852-5a3bc359a141
# performing cross validation
lasso_cv = glmnetcv(X[train_idxs, :], y[train_idxs])

# ╔═╡ d6008ed9-2a46-446a-b684-1d961c3ae20a
# best λ for Lasso
lasso_λ = lasso_sol.lambda[argmin(lasso_cv.meanloss)]

# ╔═╡ 039b7860-02fb-4f97-a1c2-9d20af989aac
# Lasso final model
lasso_f̂ = glmnet(X[train_idxs, :], y[train_idxs],lambda=[lasso_λ])

# ╔═╡ d0192106-f984-41e8-aa50-897481729dca
md"""
Now, we can predict labels for testing data...
"""

# ╔═╡ c106c3da-6444-4e64-9530-b6102766cd22
lasso_pred = GLMNet.predict(lasso_f̂, X[test_idxs,:])

# ╔═╡ 0da3fe69-f22d-4a29-9c88-c6b8e2e13d08
md"""
Note that predictions above are continuous rather than discrete. Thus, we must use the `assign_class` function to convert continuous predictions into discrete predictions...
"""

# ╔═╡ 1da8374e-aba5-4da3-bfe4-8f450408a20b
lasso_ŷ = assign_class.(lasso_pred)

# ╔═╡ 26ff22e5-e54e-4b85-b6d2-0db838cc1aca
md"""
Finally, we can compute Lasso accuracy using the `findaccuracy` function...
"""

# ╔═╡ 5d31a589-459e-4c11-8ea8-9840eb2cf8c0
lasso_acc = findaccuracy(lasso_ŷ, y[test_idxs])

# ╔═╡ 0040a4e5-13d3-4277-a865-e3f1bd328388
md"""
### Ridge

**Ridge** is a method of estimating the coefficients of multiple-regression models in scenarios where independent variables are highly correlated.

Ridge regression was developed as a possible solution to the imprecision of least square estimators when linear regression models have some multicollinear (highly correlated) independent variables—by creating a ridge regression estimator (RR). This provides a more precise ridge parameters estimate, as its variance and mean square estimator are often smaller than the least square estimators previously derived.

The main difference between Lasso and Ridge is that the former optimizes for L1-norm and Ridge optimizes for L2-norm.

**Note:** check [here](https://en.wikipedia.org/wiki/Ridge_regression) for more math details.

To perform Ridge, we are going to use the same functions. However, we must set `alpha = 0`.
"""

# ╔═╡ face4238-2511-4de5-8df3-2d25a668386e
# Ridge solutions
ridge_sol = glmnet(X[train_idxs, :], y[train_idxs], alpha=0)

# ╔═╡ faf82a4f-5acb-4d45-bf33-ded1dd82733c
# performing cross validation
ridge_cv = glmnetcv(X[train_idxs, :], y[train_idxs], alpha=0)

# ╔═╡ 103b74be-e907-41c5-ba08-3bd3ee4242e1
# best λ for Ridge
ridge_λ = ridge_sol.lambda[argmin(ridge_cv.meanloss)]

# ╔═╡ b16bfae9-5020-48fd-a796-f77d67a53f2a
# Ridge final model
ridge_f̂ = glmnet(X[train_idxs, :], y[train_idxs],lambda=[ridge_λ], alpha=0)

# ╔═╡ 4d175b94-43d1-497c-8022-42f544789e6d
md"""
Now, we can predict labels for testing data...
"""

# ╔═╡ e5e6130f-62de-4966-b164-9323b0b3bfb5
ridge_pred = GLMNet.predict(ridge_f̂, X[test_idxs,:])

# ╔═╡ a5a16631-0cd9-44cf-ae12-3f66b822f74b
md"""
Note that predictions above are also continuous rather than discrete. Thus, we must use the `assign_class` function to convert continuous predictions into discrete predictions...
"""

# ╔═╡ 887520af-9c11-4185-9f97-7da958332f76
ridge_ŷ = assign_class.(ridge_pred)

# ╔═╡ 6e37ae05-527f-4e5d-9207-383c0290f8d9
md"""
Finally, we can compute Ridge accuracy using the `findaccuracy` function...
"""

# ╔═╡ cf702060-5b49-451a-9075-3fb6f544e666
ridge_acc = findaccuracy(ridge_ŷ, y[test_idxs])

# ╔═╡ 94547be0-0467-4323-b5cd-f1efa9b98ac8
md"""
### Elastic Net

In the fitting of linear or logistic regression models, the **Elastic Net** is a regularized regression method that linearly combines the L1 and L2 penalties of the lasso and ridge methods.

**Note:** check [here](https://en.wikipedia.org/wiki/Elastic_net_regularization) for more math details.

To perform Elastic Net, we are going to use the same functions. However, we must set `alpha = 0.5`.
"""

# ╔═╡ ca590a9a-5909-4f1e-b02e-8af5da63439f
# Elastic Net solutions
enet_sol = glmnet(X[train_idxs, :], y[train_idxs], alpha=0.5)

# ╔═╡ d2e707e7-81ef-4b3d-8b8b-5ef65c8fc754
# performing cross validation
enet_cv = glmnetcv(X[train_idxs, :], y[train_idxs], alpha=0.5)

# ╔═╡ 9da0251a-ec4e-4b15-9527-9212af497e61
# best λ for Elastic Net
enet_λ = enet_sol.lambda[argmin(enet_cv.meanloss)]

# ╔═╡ eb72d5c2-15c4-41a9-81a2-c7c9f5f047ec
# Elastic Net final model
enet_f̂ = glmnet(X[train_idxs, :], y[train_idxs], alpha=0.5, lambda=[enet_λ])

# ╔═╡ 7a73633e-87d2-449b-8115-db6bf7e00266
md"""
Now, we can predict labels for testing data...
"""

# ╔═╡ 87906726-319e-454e-9299-77f6647081b8
enet_pred = GLMNet.predict(enet_f̂, X[test_idxs, :])

# ╔═╡ 5cdc7e8c-42e4-4f75-ab10-0a2b65d55809
md"""
Note that predictions above are also continuous rather than discrete. Thus, we must use the `assign_class` function to convert continuous predictions into discrete predictions...
"""

# ╔═╡ 14e764bb-94d9-4b1a-b2ce-1a39b930334a
enet_ŷ = assign_class.(enet_pred)

# ╔═╡ 46e9152e-7ee0-4420-8de6-331cef06b6dd
md"""
Finally, we can compute Elastic Net accuracy using the `findaccuracy` function...
"""

# ╔═╡ de00cd9a-009b-4991-aace-17ea06e41a1a
enet_acc = findaccuracy(enet_ŷ, y[test_idxs])

# ╔═╡ d999d681-d250-401e-87fe-963e625bfa56
md"""
### Decision Trees

**Decision Trees Learning** uses a decision tree (as a predictive model) to go from observations about an item (represented in the branches) to conclusions about the item's target value (represented in the leaves). Tree models where the target variable can take a discrete set of values are called classification trees; in these tree structures, leaves represent class labels and branches represent conjunctions of features that lead to those class labels.

**Note:** check [here](https://en.wikipedia.org/wiki/Decision_tree) for more math details.

We are going to use the [DecisionTree.jl](https://github.com/bensadeghi/DecisionTree.jl) package.
"""

# ╔═╡ 7936c964-03a4-4498-83ad-25a6cc7825a5
# defining Decision Tree model
dt_f̂ = DecisionTreeClassifier(max_depth=2)

# ╔═╡ 00258a2e-3c99-4810-879c-9b15489e52ae
# training algorithm
DecisionTree.fit!(dt_f̂, X[train_idxs,:], y[train_idxs])

# ╔═╡ 4c0e88ae-a7ca-45ca-9a23-fbd4cd752cbe
md"""
Now, we can predict labels for testing data...
"""

# ╔═╡ 02ef2267-6694-4006-b9c2-21241e7ff13a
dt_ŷ = DecisionTree.predict(dt_f̂, X[test_idxs, :])

# ╔═╡ da6959b5-29a0-42c6-8a55-3711b5f41074
md"""
Finally, we can compute Decision Tree accuracy using the `findaccuracy` function...
"""

# ╔═╡ bc55879d-d6b2-4562-a710-988968aa3ae4
dt_acc = findaccuracy(dt_ŷ, y[test_idxs])

# ╔═╡ 50a8aa34-fa23-4dbc-b69f-b897a54ad157
md"""
### Random Forests

**Random Forests** are an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees. Random decision forests correct for decision trees habit of overfitting to their training set.

**Note:** check [here](https://en.wikipedia.org/wiki/Random_forest) for more math details.

The `RandomForestClassifier` function is available through the [DecisionTree.jl](https://github.com/bensadeghi/DecisionTree.jl) package as well.
"""

# ╔═╡ 71672bfa-558e-4ceb-88fb-aa61f615603e
# defining Random Forests model
rf_f̂ = RandomForestClassifier(n_trees=20)

# ╔═╡ 7d3e38ab-009a-445b-9e63-acc8e3ac4d8c
# training algorithm
DecisionTree.fit!(rf_f̂, X[train_idxs,:], y[train_idxs])

# ╔═╡ 18564590-adaa-4f6d-8180-0c4ece6a4398
md"""
Now, we can predict labels for testing data...
"""

# ╔═╡ b19ddb74-a299-48f7-9bec-20d078edafb4
rf_ŷ = DecisionTree.predict(rf_f̂, X[test_idxs,:])

# ╔═╡ a36aad87-8772-4ff6-9e63-34313346f37e
md"""
Finally, we can compute Decision Tree accuracy using the `findaccuracy` function...
"""

# ╔═╡ 04d476ca-07cb-4138-bd1b-679c223d7faf
rf_acc = findaccuracy(rf_ŷ, y[test_idxs])

# ╔═╡ 626d7979-0362-4a54-8f7d-1df787a68195
md"""
### Nearest Neighbour

The **Nearest Neighbour Search** algorithm aims to find the point in the tree that is nearest to a given input point. This search can be done efficiently by using the tree properties to quickly eliminate large portions of the search space.

**Note:** check [here](https://en.wikipedia.org/wiki/K-d_tree) for more math details.

We are going to use the [NearestNeighbors.jl](https://github.com/KristofferC/NearestNeighbors.jl) package. This is not straight forward, since we need to do some preprocessing first. Thus, let's build the KD tree.

**Note:** we need to transpose `X` matrix in order to use NearestNeighbors.jl package.
"""

# ╔═╡ f81dfbd1-b018-4098-a484-2619215abace
# building KD tree
kdtree = KDTree(X[train_idxs,:]')

# ╔═╡ 14c7f4ba-1e06-4fe5-9fb5-b0e9bbf4793d
md"""
Now, we will use the `knn` function, which performs a lookup of the `k` nearest neigbours to the `points` from the data in the `tree`. If `sortres = true`, the result is sorted such that the results are in the order of increasing distance to the point. It returns both `indices` and `distances` of the training data.
"""

# ╔═╡ 0dd179a1-d48d-42d2-85c6-62ba48af0ccb
# getting indices and distances
knn_idxs, knn_dists = knn(kdtree, X[test_idxs,:]', 5, true);

# ╔═╡ 588afb06-4f7b-458f-8037-6ef53b96b60b
# indices
knn_idxs

# ╔═╡ ff2fcea0-3588-4a79-9906-f3effa148e86
md"""
**Note:** this is a nested `Array`, where each element is an 1D `Array` with the **indexes** of the 5 nearest points. Each element represents a test observation.
"""

# ╔═╡ 75d5de77-08db-4bdd-b418-fe9fa9e6353d
# distances
knn_dists

# ╔═╡ 4644848e-c9ae-4742-9ce3-4747a85b1fd2
md"""
**Note:** this is also a nested `Array`, where each element is an 1D `Array` with the **distances** of the 5 nearest points. Each element represents a test observation. Each element is in ascending distance order.
"""

# ╔═╡ fa554c22-2156-43a6-8014-cbd98914f27d
# 5 nearest train observation labels to each test observation
nn_y_train = y[train_idxs][hcat(knn_idxs...)]

# ╔═╡ be1405be-750e-4a74-895a-ed56beb0cd36
# counting the number of labels per test observation
nn_possible_labels = map(i -> counter(nn_y_train[:, i]), 1:size(nn_y_train, 2))

# ╔═╡ 45625714-3ff0-4b46-8087-14ecbe672d08
md"""
Now, we can predict labels for testing data. Thus, we need to select the most frequent label per each test observation.
"""

# ╔═╡ ed93cb77-9eee-491a-835c-2952bd612946
# selecting the most frequent label per each test observation
nn_ŷ = map(i -> parse(Int,
					  string(string(argmax(nn_possible_labels[i])))),
		   1:size(nn_y_train, 2))

# ╔═╡ ec8854ea-542a-4e15-8533-ec6225cafa72
md"""
Finally, we can compute Nearest Neighbour accuracy using the `findaccuracy` function...
"""

# ╔═╡ 1ee8b7ec-3542-40ae-945c-3bf489266985
nn_acc = findaccuracy(nn_ŷ, y[test_idxs])

# ╔═╡ 112a337f-11ab-4074-88bd-17a887562d43
md"""
### Support Vector Machine

**Support Vector Machines (SVM)** are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis.

Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier. SVM maps training examples to points in space so as to maximise the width of the gap between the two categories. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall.

**Note:** check [here](https://en.wikipedia.org/wiki/Support-vector_machine) for more math details.

Here, we will use the [LIBSVM.jl](https://github.com/JuliaML/LIBSVM.jl) package. We need to transpose `X` matrix in order to use this package.
"""

# ╔═╡ 428463aa-0cc7-4008-95c7-8d1b5a28f2de
# training SVM model
svm_f̂ = svmtrain(X[train_idxs,:]', y[train_idxs])

# ╔═╡ fb93f2ce-36d2-48f0-bafd-5ca18afadfd7
md"""
Now, we can predict labels for testing data. At the same time, we will also get an `Array` with decision values...
"""

# ╔═╡ f5bb8271-4239-4c4e-956d-9714a38d2c25
svm_ŷ, decision_values = svmpredict(svm_f̂, X[test_idxs,:]');

# ╔═╡ 4a3d2073-f465-4fe2-9c4d-4ceb810a2285
# SVM predictions
svm_ŷ

# ╔═╡ ead6066e-6257-4659-84a6-98ea79993ea7
md"""
Finally, we can compute Support Vector Machine accuracy using the `findaccuracy` function...
"""

# ╔═╡ 92c46bcc-b3fa-4704-85cf-74782b9e6bef
svm_acc = findaccuracy(svm_ŷ, y[test_idxs])

# ╔═╡ 0f44eabe-c806-4154-b54e-65f8c678ef47
md"""
## Comparing models

Now, we can compare the accuracies of each model...
"""

# ╔═╡ e1fdde22-08ea-4ae4-82a6-b7adbb6e250a
begin
	models = ["Lasso", "Ridge", "Elastic Net", "DT", "RF", "NN", "SVM"]

	accs = [lasso_acc, ridge_acc, enet_acc, dt_acc, rf_acc, nn_acc, svm_acc]

	hcat(models, accs)
end

# ╔═╡ Cell order:
# ╟─3247c742-65df-11ec-0eed-7d76c7786941
# ╟─ff42960a-f27f-41b8-8eee-14fb5b0454ba
# ╟─b4c7b20b-ef19-4684-933b-7a00bfb69dce
# ╟─73a1eed0-af5b-42b3-a4b8-a6f191f49daa
# ╠═68c8634d-a052-4bd0-a101-5111dd9d4f68
# ╟─091524fe-ad86-45a3-ad25-370fe25a7a4e
# ╠═7a1a6481-eb71-4eb8-b47e-cf65c2c4cafa
# ╟─0bca5475-72f2-4189-b3d8-dd06cc951fc8
# ╠═e7f4e859-5e3c-439d-b069-46f17f2acf02
# ╟─0c1f6c24-ab74-494a-89d8-ba3fbda7fa24
# ╠═b086ca45-6a60-4d4e-8387-de7e247812ed
# ╟─c495a8f7-17d0-42f7-9adf-440d83d310c1
# ╠═35fad54f-e861-46e7-ab43-04a205c1991f
# ╠═df121f29-a69d-42c9-b6a7-f129c4e94506
# ╟─1f39a7ed-7320-4f72-b4a0-fac44a71bb75
# ╠═ecb46a59-1acf-48b9-b3c7-4c319388bc33
# ╟─8ca957e5-1bec-4498-aaca-86dccca1a49b
# ╠═8868df3e-5933-4670-bbaf-eb8be100382b
# ╠═27f97d52-c5c2-41a3-9faf-a8271a404b23
# ╟─95a6bc1e-7986-4d86-9843-ceac5b668e4c
# ╠═137c8d77-d6fd-44eb-937e-5978d201670d
# ╟─6317039e-1926-4143-a4d6-61f61b5e83e0
# ╟─1bd96a25-9542-432b-8e54-d6896a9cdade
# ╠═dd6889f6-18da-4218-bc48-b6589699498a
# ╠═d03a03d1-fc97-4cb3-a852-5a3bc359a141
# ╠═d6008ed9-2a46-446a-b684-1d961c3ae20a
# ╠═039b7860-02fb-4f97-a1c2-9d20af989aac
# ╟─d0192106-f984-41e8-aa50-897481729dca
# ╠═c106c3da-6444-4e64-9530-b6102766cd22
# ╟─0da3fe69-f22d-4a29-9c88-c6b8e2e13d08
# ╠═1da8374e-aba5-4da3-bfe4-8f450408a20b
# ╟─26ff22e5-e54e-4b85-b6d2-0db838cc1aca
# ╠═5d31a589-459e-4c11-8ea8-9840eb2cf8c0
# ╟─0040a4e5-13d3-4277-a865-e3f1bd328388
# ╠═face4238-2511-4de5-8df3-2d25a668386e
# ╠═faf82a4f-5acb-4d45-bf33-ded1dd82733c
# ╠═103b74be-e907-41c5-ba08-3bd3ee4242e1
# ╠═b16bfae9-5020-48fd-a796-f77d67a53f2a
# ╟─4d175b94-43d1-497c-8022-42f544789e6d
# ╠═e5e6130f-62de-4966-b164-9323b0b3bfb5
# ╟─a5a16631-0cd9-44cf-ae12-3f66b822f74b
# ╠═887520af-9c11-4185-9f97-7da958332f76
# ╟─6e37ae05-527f-4e5d-9207-383c0290f8d9
# ╠═cf702060-5b49-451a-9075-3fb6f544e666
# ╟─94547be0-0467-4323-b5cd-f1efa9b98ac8
# ╠═ca590a9a-5909-4f1e-b02e-8af5da63439f
# ╠═d2e707e7-81ef-4b3d-8b8b-5ef65c8fc754
# ╠═9da0251a-ec4e-4b15-9527-9212af497e61
# ╠═eb72d5c2-15c4-41a9-81a2-c7c9f5f047ec
# ╟─7a73633e-87d2-449b-8115-db6bf7e00266
# ╠═87906726-319e-454e-9299-77f6647081b8
# ╟─5cdc7e8c-42e4-4f75-ab10-0a2b65d55809
# ╠═14e764bb-94d9-4b1a-b2ce-1a39b930334a
# ╟─46e9152e-7ee0-4420-8de6-331cef06b6dd
# ╠═de00cd9a-009b-4991-aace-17ea06e41a1a
# ╟─d999d681-d250-401e-87fe-963e625bfa56
# ╠═7936c964-03a4-4498-83ad-25a6cc7825a5
# ╠═00258a2e-3c99-4810-879c-9b15489e52ae
# ╟─4c0e88ae-a7ca-45ca-9a23-fbd4cd752cbe
# ╠═02ef2267-6694-4006-b9c2-21241e7ff13a
# ╟─da6959b5-29a0-42c6-8a55-3711b5f41074
# ╠═bc55879d-d6b2-4562-a710-988968aa3ae4
# ╟─50a8aa34-fa23-4dbc-b69f-b897a54ad157
# ╠═71672bfa-558e-4ceb-88fb-aa61f615603e
# ╠═7d3e38ab-009a-445b-9e63-acc8e3ac4d8c
# ╟─18564590-adaa-4f6d-8180-0c4ece6a4398
# ╠═b19ddb74-a299-48f7-9bec-20d078edafb4
# ╟─a36aad87-8772-4ff6-9e63-34313346f37e
# ╠═04d476ca-07cb-4138-bd1b-679c223d7faf
# ╟─626d7979-0362-4a54-8f7d-1df787a68195
# ╠═f81dfbd1-b018-4098-a484-2619215abace
# ╟─14c7f4ba-1e06-4fe5-9fb5-b0e9bbf4793d
# ╠═0dd179a1-d48d-42d2-85c6-62ba48af0ccb
# ╠═588afb06-4f7b-458f-8037-6ef53b96b60b
# ╟─ff2fcea0-3588-4a79-9906-f3effa148e86
# ╠═75d5de77-08db-4bdd-b418-fe9fa9e6353d
# ╟─4644848e-c9ae-4742-9ce3-4747a85b1fd2
# ╠═fa554c22-2156-43a6-8014-cbd98914f27d
# ╠═be1405be-750e-4a74-895a-ed56beb0cd36
# ╟─45625714-3ff0-4b46-8087-14ecbe672d08
# ╠═ed93cb77-9eee-491a-835c-2952bd612946
# ╟─ec8854ea-542a-4e15-8533-ec6225cafa72
# ╠═1ee8b7ec-3542-40ae-945c-3bf489266985
# ╟─112a337f-11ab-4074-88bd-17a887562d43
# ╠═428463aa-0cc7-4008-95c7-8d1b5a28f2de
# ╟─fb93f2ce-36d2-48f0-bafd-5ca18afadfd7
# ╠═f5bb8271-4239-4c4e-956d-9714a38d2c25
# ╠═4a3d2073-f465-4fe2-9c4d-4ceb810a2285
# ╟─ead6066e-6257-4659-84a6-98ea79993ea7
# ╠═92c46bcc-b3fa-4704-85cf-74782b9e6bef
# ╟─0f44eabe-c806-4154-b54e-65f8c678ef47
# ╠═e1fdde22-08ea-4ae4-82a6-b7adbb6e250a


function procrustes(X::Shape, Y::Shape)
    @assert size(X) == size(Y)
    n, m = size(X)
    mu_x = mean(X, 1)
    mu_y = mean(Y, 1)

    X0 = X .- mu_x
    Y0 = Y .- mu_y

    ss_x = sum(X0.^2)
    ss_y = sum(Y0.^2)

    norm_x = sqrt(ss_x)
    norm_y = sqrt(ss_y)

    X0 ./= norm_x
    Y0 ./= norm_y

    A = X0'Y0
    U, s, Vt = svd(A)
    V = Vt'
    T = V*U'

    traceTA = sum(s)

    b = traceTA * norm_x / norm_y
    d = 1 - traceTA ^ 2
    Z = norm_x * traceTA * (Y0 * T) .+ mu_x

    c = mu_x - b * (mu_y * T)

    tform = @compat(Dict("rotation" => T, "scale" => b, "translation" => c))
    return d, Z, tform

end

function align_shapes(shapes::Vector{Shape}; n_iter=100)
    shapes = copy(shapes)
    # mean_shape = mean(shapes)
    mean_shape = shapes[1]
    ref_shape = mean_shape
    for it=1:n_iter
        for i=1:length(shapes)
            _, shapes[i], _ = procrustes(mean_shape, shapes[i])
        end
        new_mean_shape = mean(shapes)
        _, Y, _ = procrustes(ref_shape, new_mean_shape)
        new_mean_shape = Y
        mean_shape = new_mean_shape
    end
    mean_shape = mean(shapes)
    return mean_shape, shapes
end


using Bijectors, LinearAlgebra, Optim, Plots, Random, StatsFuns, Stheno, WarpedRegressors

# Specify a Warped-GP.
f = GP(kernel(EQ(); l=2.0, s=0.5), GPC())
σ² = 0.0001
ϕ = Bijectors.Exp()
g = warp(f, ϕ)

# Generate some training data.
rng = MersenneTwister(123456)
Ntr = 35
xtr = 3 * randn(rng, Ntr)
g_xtr = g(xtr, σ²)
ytr = rand(rng, g_xtr)

# Specify parameter handler.
unpack(θ) = (
    σ² = log1pexp(θ[1]) + 1e-6,
    l  = log1pexp(θ[2]) + 1e-6,
    s  = log1pexp(θ[3]) + 1e-6,
)

# Infer σ², l, and s.
function nlml(θ::AbstractVector{<:Real})
    θ = unpack(θ)
    f = GP(kernel(EQ(); l=θ.l, s=θ.s), GPC())
    g = warp(f, ϕ)
    return -logpdf(g(xtr, θ.σ²), ytr)
end
θ0 = randn(3)
results = Optim.optimize(nlml, θ0, NelderMead())
θ = unpack(results.minimizer)

# Construct posterior Warped GP
f_learned = GP(kernel(EQ(); l=θ.l, s=θ.s), GPC())
g_learned = warp(f_learned, ϕ)
g_posterior = posterior(g_learned(xtr, θ.σ²), ytr)

# Visualise the posterior.
let
    xpr = range(-10.0, 10.0; length=250)
    Ypr = hcat([rand(rng, g_posterior(xpr, θ.σ²)) for _ in 1:1000]...)
    median_pr = map(ys->quantile(ys, 0.5), eachrow(Ypr));
    q95_pr = map(ys->quantile(ys, 0.95), eachrow(Ypr));
    q05_pr = map(ys->quantile(ys, 0.05), eachrow(Ypr));

    plotly()
    plt = plot()
    plot!(plt, xpr, Ypr[:, 1:10]; linecolor=:blue, linewidth=0.5, linealpha=0.2, label="")
    plot!(plt, xpr, median_pr; linecolor=:blue, linewidth=1.0, linealpha=1.0, label="")
    plot!(plt, xpr, [median_pr, median_pr];
        linewidth=0.0,
        fillrange=[q05_pr, q95_pr],
        fillalpha=0.3,
        fillcolor=:blue,
        label="",
    )
    scatter!(plt, xtr, ytr; label="", markersize=2)
    display(plt)
end


"""
Computes the Bayesian score component for the given target variable index and
    Dirichlet prior counts given in alpha

INPUT:
    i       - index of the target variable
    parents - list of indeces of parent variables (should not contain self)
    r       - list of instantiation counts accessed by variable index
              r[1] gives number of discrete states variable 1 can take on
    data - matrix of sufficient statistics / counts
              d[j,k] gives the number of times the target variable took on its kth instantiation
              given the jth parental instantiation

OUTPUT:
    the Bayesian score, Float64
"""
function bayesian_score_component(
    i::Int,
    parents::AbstractVector{I},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    alpha::AbstractMatrix{Float64}, # ncategories[i]×prod(ncategories[parents])
) where {I<:Integer}

    (n, m) = size(data)
    if !isempty(parents)
        Np = length(parents)
        stridevec = fill(1, Np)
        for k in 2:Np
            stridevec[k] = stridevec[k-1] * ncategories[parents[k-1]]
        end
        js = (data[parents,:] - 1)' * stridevec + 1
    else
        js = fill(1, m)
    end

    N = sparse(vec(data[i,:]), vec(js), 1, size(alpha)...) # note: duplicates are added together
    sum(SpecialFunctions.loggamma.(alpha + N)) - sum(SpecialFunctions.loggamma.(alpha)) + sum(SpecialFunctions.loggamma.(sum(alpha,dims=1))) - sum(SpecialFunctions.loggamma.(sum(alpha,dims=1) + sum(N,dims=1)))::Float64
end
function bayesian_score_component_uniform(
    i::Int,
    parents::AbstractVector{I},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    prior::DirichletPrior,
) where {I<:Integer}

    (n, m) = size(data)
    if !isempty(parents)
        Np = length(parents)
        stridevec = fill(1, Np)
        for k in 2:Np
            stridevec[k] = stridevec[k-1] * ncategories[parents[k-1]]
        end
        js = (data[parents,:] .- 1)' * stridevec .+ 1
    else
        js = fill(1, m)
    end

    n = prod(ncategories[parents])
    N = sparse(data[i,:], js, 1, ncategories[i], n...) # note: duplicates are added together

    u = prior.α
    p = SpecialFunctions.loggamma(u)

    # Given a sparse N, we can be clever in our calculation and not waste time
    # computing the same SpecialFunctions.loggamma values by exploiting the sparse structure.
    sum0 = sum(SpecialFunctions.loggamma.(nonzeros(N) .+ u)) .+ p * (ncategories[i] * n - nnz(N))
    sum1 = n * ncategories[i] * p
    sum2 = n * SpecialFunctions.loggamma(ncategories[i] * u)
    cc = ncategories[i] * u

    @static if Base.VERSION.major == 0 && Base.VERSION.minor < 5
        sN = sparsevec(sum(N[1:ncategories[i],:], 1)) # Slower, but should be supported by Julia 0.4
    else
        sN = sum(N[i,:] for i=1:ncategories[i])
    end
    sum3 = sum(SpecialFunctions.loggamma.(nonzeros(sN) .+ cc)) .+ (size(N, 2) - nnz(sN)) * SpecialFunctions.loggamma(cc)
    sum0 - sum1 + sum2 - sum3::Float64
end
function bayesian_score_component(
    i::Int,
    parents::AbstractVector{I},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    prior::DirichletPrior,
) where {I<:Integer}

    if typeof(prior) == UniformPrior
        return bayesian_score_component_uniform(i, parents, ncategories, data, prior)
    end
    alpha = get(prior, i, ncategories, parents)
    bayesian_score_component(i, parents, ncategories, data, alpha)
end

function bayesian_score(
    parent_list::Vector{Vector{Int}},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    prior::DirichletPrior,
    )

    tot = 0.0
    for (i, p) in enumerate(parent_list)
        tot += bayesian_score_component(i, p, ncategories, data, prior)
    end
    tot
end
function bayesian_score(bn::DiscreteBayesNet, data::DataFrame, prior::DirichletPrior=UniformPrior())

    n = length(bn)
    parent_list = Array{Vector{Int}}(undef, n)
    ncategories = Array{Int}(undef, n)
    datamat = Matrix{Int}(data)'

    for (i,cpd) in enumerate(bn.cpds)
        parent_list[i] = inneighbors(bn.dag, i)
        ncategories[i] = infer_number_of_instantiations(convert(Vector{Int}, data[!,i]))
    end

    bayesian_score(parent_list, ncategories, datamat, prior)
end

function bayesian_score_component(
    i::Int,
    parents::AbstractVector{Int},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    prior::DirichletPrior,
    cache::ScoreComponentCache,
    )

    if !haskey(cache[i], parents)
        (cache[i][parents] = bayesian_score_component(i, parents, ncategories, data, prior))
    end

    cache[i][parents]
end
function bayesian_score_components(
    parent_list::Vector{Vector{Int}},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    prior::DirichletPrior,
    )

    score_components = Array{Float64}(undef, length(parent_list))
    for (i,p) in enumerate(parent_list)
        score_components[i] = bayesian_score_component(i, p, ncategories, data, prior)
    end
    score_components
end
function bayesian_score_components(
    parent_list::Vector{Vector{Int}},
    ncategories::AbstractVector{Int},
    data::AbstractMatrix{Int},
    prior::DirichletPrior,
    cache::ScoreComponentCache,
    )

    score_components = Array{Float64}(undef, length(parent_list))
    for (i,p) in enumerate(parent_list)
        score_components[i] = bayesian_score_component(i, p, ncategories, data, prior, cache)
    end
    score_components
end
function bayesian_score_components(bn::DiscreteBayesNet, data::DataFrame, prior::DirichletPrior=UniformPrior())

    n = length(bn)
    parent_list = Array{Vector{Int}}(undef, n)
    ncategories = Array{Int}(undef, n)
    datamat = Matrix{Int}(data)'

    for (i,cpd) in enumerate(bn.cpds)
        parent_list[i] = inneighbors(bn.dag, i)
        ncategories[i] = infer_number_of_instantiations(convert(Vector{Int}, data[!,i]))
    end

    bayesian_score_components(parent_list, ncategories, datamat, prior)
end

"""
    bayesian_score(G::DAG, names::Vector{Symbol}, data::DataFrame[, ncategories::Vector{Int}[, prior::DirichletPrior]])

Compute the bayesian score for graph structure `g`, with the data in `data`. `names` containes a symbol corresponding to each vertex in `g` that is the name of a column in `data`. `ncategories` is a vector of the number of values that each variable in the Bayesian network can take.

Note that every entry in data must be an integer greater than 0
"""
function bayesian_score(G::DAG,
                        names::Vector{Symbol},
                        data::DataFrame,
                        ncategories::Vector{Int}=Int[infer_number_of_instantiations(convert(Vector{Int}, data[!,n])) for n in names],
                        prior::DirichletPrior=UniformPrior())
    datamat = Array{Int}(undef, ncol(data), nrow(data))
    for i in 1:nv(G)
        datamat[i,:] = data[!,names[i]]
    end

    # NOTE: this is badj(G) prior to v0.6 and inneighbors(G) in v0.6
    backwards_adjacency = [inneighbors(G, i) for i in 1 : nv(G)]
    return bayesian_score(backwards_adjacency, ncategories, datamat, prior)
end


y_true = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
y_pred = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]

# absolute_error
@test absolute_error(y_true, y_pred) == [9, 7, 5, 3, 1, 1, 3, 5, 7, 9]

# percent_error
@test 1 == 1

# log_error
@test 1 == 1

# squared_error
@test squared_error(y_true, y_pred) == [81, 49, 25, 9, 1, 1, 9, 25, 49, 81]

# squared_log_error
@test 1 == 1

# absolute_percent_error
@test 1 == 1

# mean_error
@test mean_error(y_true, y_pred) == 0.0

# mean_absolute_error
@test mean_absolute_error(y_true, y_pred) == 5.0

# median_absolute_error
@test median_absolute_error(y_true, y_pred) == 5.0

# mean_percent_error
@test 1 == 1

# median_percent_error
@test 1 == 1

# mean_squared_error
@test mean_squared_error(y_true, y_pred) == 33.0

# median_squared_error
@test median_squared_error(y_true, y_pred) == 25.0

# sum_squared_error
@test sum_squared_error(y_true, y_pred) == 330

# mean_squared_log_error
@test 1 == 1

# mean_absolute_percent_error
@test 1 == 1

# median_absolute_percent_error
@test 1 == 1

# symmetric_mean_absolute_percent_error
@test 1 == 1

# symmetric_median_absolute_percent_error
@test 1 == 1

# mean_absolute_scaled_error
@test mean_absolute_scaled_error(y_true, y_pred) == 1.125

# total_variance_score
@test total_variance_score(y_true, y_pred) == 82.5

# explained_variance_score
@test explained_variance_score(y_true, y_pred) == 82.5

# unexplained_variance_score
@test unexplained_variance_score(y_true, y_pred) == 330

# r2_score
@test r2_score(y_true, y_pred) == 1.0


using Test, SimplePolynomials, Polynomials, Mods, Multisets

a = SimplePolynomial(1, 1, 1)
b = SimplePolynomial(1, 2, 3)
c = SimplePolynomial(0, 1, -3, 4, 1)
x = getx()
p = (1 + x + x^3 - x^2)
q = (1 + 3x) * (1 - x) + 5

@testset "Polynomial Constructors" begin
    @test p == SimplePolynomial(1, 1, -1, 1)
    @test p == SimplePolynomial([1, 1, -1, 1])
    @test c == SimplePolynomial(c)
    @test c == SimplePolynomial(c.data)
    @test x + Mod{13}(2) == SimplePolynomial([Mod{13}(2), Mod{13}(1)])
end

@testset "Rational Function Constructors" begin
    f = p / q
    @test SimpleRationalFunction(p, q) == f
    @test f(1) == p(1) // q(1)
    f = inv(p)
    @test f(1) * p(1) == 1
end

@testset "String" begin
    p = getx()+1
    @test string(p) == "1 + x"
end

@testset "Polynomial Arithmetic" begin
    @test a + a == 2a
    @test (a + a + a) / 3 == a
    @test a - a == 0
    @test a * a == SimplePolynomial(1, 2, 3, 2, 1)
    @test a * a * a == a^3
    @test eltype(b) == BigInt
end

@testset "Rational Function Arithmetic" begin
    f = p / q
    g = a / b
    @test f * g == (p * a) / (b * q)
    @test f + g == (p * b + a * q) // (b * q)
    @test f - g == (-g) + f
    @test f / g == (p * b) / (a * q)
    @test inv(f) == q / p
end

@testset "Complex numbers" begin
    z = a + im
    @test z(-5) == a(-5) + im
    f = (im + x) / (2 * im * x)
    @test 2f(1) == 1 - im
    z = (x-im)*(x+im)
    @test z == x^2 + 1
end

@testset "Mod numbers" begin
    z = SimplePolynomial(Mod{13}.([1,2,3]))
    @test z(1) == Mod{13}(6)
    f = inv(z)
    @test f(1) == inv(Mod{13}(6))
end

@testset "Polynomial GCD" begin
    p = (x-1)^2 * (x-2)
    q = (x-1) * (x-3)
    @test gcd(p,q) == x-1

    p = (x-1)^3 * (x-3)
    @test gcd(p,p') == (x-1)^2

    c = Mod{13}.(coeffs(p))
    p = SimplePolynomial(c)
    @test gcd(p,p') == (x-1)^2
end

# @testset "Size change" begin
#     # B = big(b)
#     @test B == b
#     @test eltype(B) == BigInt
#     @test eltype(small(B)) == Int

#     r = a/b
#     @test eltype(numerator(r)) == Rational{BigInt}
#     # r = big(r)
#     # @test eltype(denominator(r)) == Rational{BigInt}
# end

@testset "PolyCalculus" begin
    p = 3x // 2 - 5x^2 + (2 - im) * x^3
    @test derivative(integral(p)) == p
end

@testset "Mod Polys" begin
    p = SimplePolynomial(Mod{13}.(0:4))
    @test p(0) == Mod{13}(0)
    @test 2p == p + p
    @test 3 * (p / 3) + 1 == Mod{13}(1) + p
end

@testset "Rational Basics" begin
    f = a / b
    g = b / a
    @test f * g == 1
    @test 2f == (a + a) / b
    @test (f - g)^2 == f^2 - 2f * g + g^2
    @test (f / g)^2 == (f^2) // (g^2)
end

@testset "Rational derivative" begin
    f = inv(1 + x)
    g = derivative(f)
    @test g == -1 // (1 + x)^2
end

@testset "Rational Mod" begin
    x = getx(Mod{13})
    p = Mod{13}(1) + x
    q = Mod{13}(2) + 3x
    f = p / q
    @test f(0) == Mod{13}(1 // 2)
end

@testset "Functionize" begin
    x = getx()
    p = 3 + 5x + 8x^2
    P = make_function(p)
    @test p(10) == P(10)

    f = p / (1-x)
    F = make_function(f)
    @test F(10) == f(10)
    A = [ 2 3 ; 0 -1]
    p = -2 - x + x^2
    @test all(p(A).==0)
end

@testset "Rational Roots" begin
    x = getx()
    p = (x^2+1)*(2x-1)^2*(3x+5)
    R = rational_roots(p)
    @test R == Multiset([1//2, -5//3, 1//2])
end

@testset "Numerical Roots" begin
    x = getx()
    p = (x^2+1)*(2x-1)^2*(3x+5)*(x^2+2)
    R = roots(p)
    Z = p.(R)
    @test sum(abs(z) for z in Z) < 10^-10

    R = newton_roots(p)
    Z = p.(R)
    @test sum(abs(z) for z in Z) < 10^-50

end

nothing


"""
    Parities

Implementation of parity queries using Fast Hadamard Walsh Transform and Gosper
iteration.

Releasing privacy prserving marginals for contingency table.
The measurments in this case are not the marginals, but the fourier coefficients of the contingency table,
that are sufficient for recovering the marginals.

Details can be found in:
"B. Barak, K. Chaudhuri, C. Dwork, S. Kale,F. McSherry, and K. Talwar.
Privacy, accuracy, and consistency too: a holistic solution to contingency table release. In PODS, 2007."
"""
struct Parities <: Queries
    dimension::Int
    order::Int
    idx::Array{Int, 1}
end

"""
    hadamard_basis_vector(index, dimension)

Returns hadamard basis vector given by `index`.
"""
function hadamard_basis_vector(index::Int, dimension::Int)
    hadamard = zeros(Float64, 1 << dimension)
    hadamard[1] = 1.0
    for i = 0:dimension-1
        sign = (index & (1 << i)) > 0 ? -1.0 : 1.0
        @simd for j = 1:(1 << i)
            @inbounds hadamard[j + (1 << i)] = sign * hadamard[j]
        end
    end
    hadamard
end

"""
return the downward closure of all the incides of the k-way marginals, of dimension d.

# Example:
    k = 2, d = 4
    k-way indices= [0011, 0101, 0110, 1001, 1010, 1100]
    downward_closure(0011) = [0000, 0001, 0010, 0011]
"""
function complete_way_marginals_indices(k::Int64, d::Int64)

    idx = Set(zeros(Int64, 0))

    for s = gosper(d, k)
        for alpha in BinaryItr(d, s)
            temp = alpha + 1
            push!(idx, temp)
        end
    end

    sort(collect(idx))
end

function Parities(dimension, order)
    Parities(dimension, order, complete_way_marginals_indices(order, dimension))
end

function get(queries::Parities, i::Int)
    HistogramQuery(hadamard_basis_vector(queries.idx[i]-1, queries.dimension))
end

"""
evaluate the measurements of the linear queries defined by
queries.idx (the indices needed for calculating the marginals of the contingency table)
"""
function evaluate(queries::Parities, h::Histogram)
    2^queries.dimension * fwht_natural(h.weights)[queries.idx]
end

function fourierCoefficients(queries::Parities, h::Histogram)
    2^queries.dimension * fwht_natural(h.weights)
end

"""
Calculate the marginal C_beta : R^(2^d) -> R^(2^(norm1(beta))), for beta in {0,1}^d,
of the fourier basis vector in the location alpha
"""
function fourierMarginal(dimension::Int64, beta::Int64, alpha::Int64)

    @assert beta & alpha == alpha

    beta_norm1, beta_indices = norm_1_with_indices(beta)
    alpha_norm1, alpha_indices = norm_1_with_indices(alpha)

    marginal = zeros(Int64, 2^beta_norm1)

    inverse_beta = setdiff(collect(1:dimension), beta_indices)

    for (gamma_index, gamma) in enumerate(BinaryItr(dimension, beta))
        marginal[gamma_index] = 2^length(inverse_beta)*((-1)^(norm_1(alpha&gamma)))
    end

    marginal
end

"""
Calculate the marginal C_beta : R^(2^d) -> R^(2^(norm1(beta))), for beta in {0,1}^d,
of the contingencyTable (represent as histogram).
"""
function calculateMarginal(queries::Parities, h::Histogram, beta::Int64)

    beta_norm1, beta_indices = norm_1_with_indices(beta)

    marginal = zeros(Int64, 2^beta_norm1)

    coefficients = fourierCoefficients(queries, h)

    for alpha in BinaryItr(queries.dimension, beta)
        marginal += fourierMarginal(queries.dimension, beta, alpha)*coefficients[alpha+1]
    end

    marginal/(2^queries.dimension)
end

function Marginals(queries::Parities, h::Histogram)

    marginals = []

    for beta in queries.idx
        push!(marginals, calculateMarginal(queries, h, beta))
    end

    marginals
end

function initialize(queries::Parities, data::Tabular, ps::MWParameters)
    initialize(queries, Histogram(data), ps)
end

struct FactorParity <: FactorHistogramQuery
    attributes::Array{Int, 1}
end

"""
    FactorParities

Implementation of parity queries over factored histograms.
"""
struct FactorParities <: FactorHistogramQueries
    queries::Array{FactorParity, 1}
end

attributes(q::FactorParity) = q.attributes
default_value(q::FactorParity) = 0.0
get(qs::FactorParities, i::QueryIndex) = qs.queries[i]

function restrict(q::FactorParity, attributes::Array{Int, 1})
    idx = 0
    d = length(attributes)
    for a in q.attributes
        i = findfirst(x -> x==a, attributes)
        idx += 2^(d-i)
    end
    HistogramQuery(hadamard_basis_vector(idx, d))
end

function FactorParities(dimension, order)
    qs = FactorParity[]
    for k = 1:order
        append!(qs, [FactorParity(s) for s = subsets(collect(1:dimension), k)])
    end
    FactorParities(qs)
end

function parity(x::Array{Float64, 1}, attributes::Array{Int64, 1})
    (-1.0)^sum(x[attributes])
end

function evaluate(q::FactorParity, table::Tabular)
    s = 0.0
    n = size(table.data)[2]
    a = attributes(q)
    @simd for i = 1:n
        @inbounds s += parity(table.data[:, i], a)
    end
    s/n
end

function evaluate(qs::FactorParities, table::Tabular)
    evals = zeros(Float64, length(qs.queries))
    @simd for i in 1:length(qs.queries)
        @inbounds evals[i] = evaluate(qs.queries[i], table)
    end
    evals
end

function evaluate(qs::FactorParities, d::FactorHistogram)
    [evaluate(q, d) for q in qs.queries]
end


# Function for calculating the sample autocovariance of a vector for lags from
# 0 to maxlag, returning a vector of length maxlag+1
function autocov(sample::Vector{Float64}, maxLag::Int)
  n = size(sample, 1)

  if n <= maxLag
    throw("The length of the input vector must be at least maxlag+1")
  end

  sample = sample-mean(sample)

  acv = Array(Float64, maxLag+1)

  for i = 0:maxLag
     acv[i+1] = dot(sample[1:n-i], sample[1+i:n])
  end

  acv = acv/n

  return acv
end

# Function for calculating the standard Normal cumulative density function more
# accurately for small values of its argument
function logOfNormalCdf(x::Union(Float64, Vector{Float64}))

  c::Float64 = -6.5
  a = [1, -1, 3, -15, 105, -945, 10395, -135135, 2027025, -34459425, 654729075]

  n = size(x, 1)
  y = Array(Float64, n)

  for i in 1:n
    if x[i]>c
      y[i] = log(cdf(Normal(0., 1.), x[i]))
    else
      y[i] = log(-dot(a, x[i].^(-2*(1:11)+1)))-x[i]^2/2-log(2*pi)/2
    end
  end

  return y
end

# Function for estimating the variance of a single MCMC chain using the initial
# monotone sequence estimator of Geyer, see
# Practical Markov Chain Monte Carlo, Charles J. Geyer, Statistical Science,
# Vol. 7, No. 4. (1992), pp. 473-483, doi:10.2307/2246094
function imse(sample::Union(Array{Float64, 1}, Array{Float64, 2}), maxLag::Int)
  nData = size(sample, 1)
  nPars = size(sample, 2)

  k = convert(Int, floor((maxLag-1)/2))

  # Preallocate memory
  acv = Array(Float64, maxLag+1, nPars)
  g = Array(Float64, k+1, nPars)
  m = (k+1)*ones(nPars)
  variance = Array(Float64, nPars)

  # Calculate empirical autocovariance
  for i = 1:nPars
    acv[:, i] = autocov(sample[:, i], maxLag)
  end

  # Calculate \hat{G}_{n, m} from the autocovariances, see pp. 477 in Geyer
  for i = 1:nPars
    for j = 0:k
      g[j+1, i] = acv[2*j+1, i]+acv[2*j+2, i]
      if g[j+1, i] <= 0
        m[i] = j
        break
      end
    end
  end

  # Create the monotone sequence of g
  for i = 1:nPars
    if m[i] > 1
      for j = 2:m[i]
        if g[j, i] > g[j-1, i]
          g[j, i] = g[j-1, i]
        end
      end
    end
  end

  # Calculate the initial monotone sequence estimator
  for i = 1:nPars
    variance[i] = (-acv[1, i]+2*sum(g[1:m[i], i]))/nData
  end

  return variance
end


import Interpolations:
        CubicSplineInterpolation,
        Line, OnGrid, Throw

"""
Cubic spline interpolation of `x` and `y` on a regular grid `inpt_grid_x`. Returns the
interpolated `y` values.
"""
function cubicspline_interpolation(x::AbstractRange, y, inpt_grid_x;
        bc = Line(OnGrid()),
        extrapolation_bc = NaN,
        sort = true)

    # Knots must be sorted in ascending order.
    if sort
        sort_inds = sortperm(x)
        # Interpolation object
        itp = CubicSplineInterpolation(x[sort_inds], y[sort_inds],
                bc = bc,
                extrapolation_bc = extrapolation_bc)
    else
        itp = CubicSplineInterpolation(x, y,
                bc = bc,
                extrapolation_bc = extrapolation_bc)
    end

    # Evaluate the interpolation object at the x-axis grid we provide. These will
    # be our interpolated y-axis values
    itp(inpt_grid_x)
end

export cubicspline_interpolation


#############################################################################
#
# Functions for computing the primary radius of the bounding ellipsoid of the
# oscillatory part of the Riemann theta function.
#
# The two subroutines solve for the radius using,
#
# * Theorem 3 of [CRTF] (no derivatives)
# * A generalization of Theorems 5 and 7 of [CRTF] for N derivatives
#
# Original Authors
# -------
# * Chris Swierczewski (@cswiercz) - September 2012, July 2016
# * Grady Williams (@gradyrw) - October 2012
# * Jeremy Upsal (@jupsal) - July 2016
#
# References
# ----------
#
# .. [CRTF] B. Deconinck, M.  Heil, A. Bobenko, M. van Hoeij and M. Schmies,
#    Computing Riemann Theta Functions, Mathematics of Computation, 73, (2004),
#    1417-1442.
#
################################################################################

"""
    radius(ϵ::Float64,
           T::Matrix{Float64},
           derivs::Vector{Vector{Float64}} = Vector{Float64}[],
           accuracy_radius::Float64 = 5.)

Returns the primary radius of the bounding ellipsoid for computing the
Riemann theta function up to accuracy `ϵ`.

The derivative oscillatory part of the Riemann theta function has linear
growth in :math:`z` along the directions of the columns of the Riemann
matrix. `accuracy_radius` is used to determine a radius resulting in an
accurate Riemann theta for all

.. math ::

    ||z|| < \text{accuracy_radius}.

Parameters
----------
- ϵ : Requested accuracy.
- T : A gxg matrix representing the Cholesky decomposition of the imaginary
    part of a Riemann matrix.
- derivs : (Default: []) A list of directional derivatives. The number of
    directional derivatives is the order, N, of the derivative we wish to
    compute.
- accuracy_radius : (Default: 5) Radius for guaranteed region of accuracy. See above.

Returns
-------
- radius : The initial radius of the bounding ellipsoid used to truncate the
    Riemann theta function to desired accuracy.

"""
function radius(ϵ::Float64,
                T::AbstractMatrix{Float64},
                derivs::Vector{Vector{ComplexF64}} = Vector{ComplexF64}[],
                accuracy_radius::Float64 = 5.)
    g = size(T,1)
    # compute the LLL-reduction of T
    U = lll([ T[:,i] for i in 1:size(T,2) ])
    ρ = minimum(norm, U)

    if length(derivs) == 0
        radius0(ϵ, ρ, g)
    else
        radiusN(ϵ, ρ, T, derivs, accuracy_radius)
    end
end

"""
    radius0(eps::Float64, r::Float64, g::Int)::Float64

Compute the radius with no derivatives.
"""
function radius0(ϵ::Float64, ρ::Float64, g::Int)::Float64
    tmp = 2ϵ / g * (ρ / 2.)^g
    tmp = gammainvccdf(g / 2., 1.0, tmp)
    R = sqrt(tmp) + ρ / 2.
    max(R, ( sqrt(2g) + ρ ) / 2)
end

"""
   radiusN(eps, r, T, derivs, accuracy_radius=5)::Float64

Compute the radius with N derivatives.

Parameters
----------
- eps : Requested accuracy.
- r : The length of the shortest lattice vector in the LLL reduction of the
    Cholesky decomposition of the imaginary part of the Riemann matrix.
- T : A gxg matrix representing the Cholesky decomposition of the imaginary
    part of a Riemann matrix.
- derivs : A list of directional derivatives. The number of directional
    derivatives is the order, N, of the derivative we wish to compute.
- accuracy_radius : Radius for guaranteed region of accuracy. See :func:`radius`.

Returns
-------
- radius : The initial radius of the bounding ellipsoid used to truncate the
    Riemann theta function to desired accuracy.

"""
function radiusN(ϵ::Float64, ρ::Float64,
                 T::AbstractMatrix{Float64},
                 derivs::Vector{Vector{ComplexF64}},
                 accuracy_radius::Float64 = 5.)::Float64

    N, g = length(derivs), size(T,1)
    prodnormderiv = prod(norm, derivs)
    normTinv = norm(inv(T))

    lhs = ϵ * ρ ^ g * 2 ^ (1. - g - N) /
          ( π ^ (N / 2) * g * normTinv ^ N * prodnormderiv )

    # define lower bound (guess) and attempt to solve for the radius
    lbnd = (sqrt(g + 2N + sqrt(g^2 + 8N)) + ρ) / 2.
    function rhs(ins::Float64)
        ai(k) = binomial(N,k) * π^(k/2.) *
                (accuracy_radius * normTinv)^k *
                gamma((g+N-k)/2.) *
                gammaccdf((g+N-k)/2., 1.0, ins)
        sum(ai, 0:N) - lhs
    end
    inszero = fzero(rhs, lbnd, 1e5)

    max(sqrt(inszero) + ρ / 2., lbnd)
end


module TestReturnInf

using Test
using TreeParzen

function objective(params)
    if params[:x] < 0
        return (params[:x] - 3)^2
    end

    return Inf
end
best = fmin(objective, Dict(:x => HP.Uniform(:x, -5.0, 5.0)), 50)
@test best[:x] < 0

end
true


@testset "binning.jl" begin
    @testset "EqualSize" begin
        # default constructor
        @test EqualSize().n == 10

        # exception
        @test_throws ErrorException EqualSize(; n=0)
        @test_throws ErrorException EqualSize(; n=-1)

        # binning
        nbins = 11
        b = ReliabilityDiagrams.bins(rand(100), rand(100), EqualSize(; n=nbins))
        @test b isa Histogram{Int}
        @test length(b.weights) == nbins
        @test length(b.edges) == 1
        @test first(b.edges) == range(0; stop=1, length=nbins + 1)

        @test ReliabilityDiagrams.binindex(b, prevfloat(0.0)) == 0
        @test ReliabilityDiagrams.binindex(b, 0) == 1
        @test ReliabilityDiagrams.binindex(b, 0.05) == 1
        @test ReliabilityDiagrams.binindex(b, 0.1) == 2
        @test ReliabilityDiagrams.binindex(b, 1) == nbins
        @test ReliabilityDiagrams.binindex(b, nextfloat(1.0)) == nbins + 1

        @test ReliabilityDiagrams.max_nonempty_bins(b, 1) == 1
        @test ReliabilityDiagrams.max_nonempty_bins(b, nbins - 1) == nbins - 1
        @test ReliabilityDiagrams.max_nonempty_bins(b, nbins) == nbins
        @test ReliabilityDiagrams.max_nonempty_bins(b, nbins + 1) == nbins
    end

    @testset "EqualMass" begin
        # default constructor
        @test EqualMass().n == 10

        # exception
        @test_throws ErrorException EqualMass(; n=0)
        @test_throws ErrorException EqualMass(; n=-1)

        # binning
        x = rand(100)
        nbins = 17
        b = ReliabilityDiagrams.bins(x, rand(100), EqualMass(; n=nbins))
        @test b isa Histogram{Int}
        @test length(b.weights) == nbins
        @test length(b.edges) == 1
        @test first(b.edges) ≈ quantile(x, range(0; stop=1, length=nbins + 1))

        @test ReliabilityDiagrams.binindex(b, prevfloat(minimum(x))) == 0
        @test ReliabilityDiagrams.binindex(b, minimum(x)) == 1
        @test ReliabilityDiagrams.binindex(b, quantile(x, 0.05)) == 1
        @test ReliabilityDiagrams.binindex(b, quantile(x, 0.1)) == 2
        @test ReliabilityDiagrams.binindex(b, maximum(x)) == nbins
        @test ReliabilityDiagrams.binindex(b, nextfloat(maximum(x))) == nbins + 1

        @test ReliabilityDiagrams.max_nonempty_bins(b, 1) == 1
        @test ReliabilityDiagrams.max_nonempty_bins(b, nbins - 1) == nbins - 1
        @test ReliabilityDiagrams.max_nonempty_bins(b, nbins) == nbins
        @test ReliabilityDiagrams.max_nonempty_bins(b, nbins + 1) == nbins
    end
end


using HDF5

TRAIN_DATA_PATH="datasets/train_catvnoncat.h5"
TEST_DATA_PATH="datasets/test_catvnoncat.h5"

train_data_x_orig = h5read(TRAIN_DATA_PATH, "train_set_x")
train_data_y_orig = h5read(TRAIN_DATA_PATH, "train_set_y")

test_data_x_orig = h5read(TEST_DATA_PATH, "test_set_x")
test_data_y_orig = h5read(TEST_DATA_PATH, "test_set_y")

classes = h5read(TEST_DATA_PATH, "list_classes")

train_data_y_orig = reshape(train_data_y_orig, (1, size(train_data_y_orig)[1]))

test_data_y_orig = reshape(test_data_y_orig, (1, size(test_data_y_orig)[1]))

function reshape_image_array(image_array::Array{UInt8,4})
    return reshape(image_array, :, size(image_array)[4])
end
function reshape_image_array_reverse(image_array::Array{UInt8,2})
    return reshape(image_array, 64, 64, 3, :)
end

test_data_x = reshape_image_array(test_data_x_orig) ./ 255
test_data_x = Array{Float32, 2}(test_data_x)

train_data_x = reshape_image_array(train_data_x_orig) ./ 255
train_data_x = Array{Float32, 2}(train_data_x)

using Statistics, BenchmarkTools

include("../../NeuralNetwork.jl")

layer_dims = [12288, 7, 1]

# @btime neural_network_dense($train_data_x, $train_data_y_orig, $layer_dims, 100, 0.0075)

parameters, activations = neural_network_dense(train_data_x, train_data_y_orig, layer_dims, 1000, 0.0075)

predicts, accuracy = predict(train_data_x, train_data_y_orig, parameters, activations)

predicts, accuracy = predict(test_data_x, test_data_y_orig, parameters, activations)


using OpenQuantumTools, OrdinaryDiffEq, Plots
# define the Hamiltonian
H = DenseHamiltonian([(s)->1.0], [-σz], unit=:ħ)

# define the initial state by PauliVec[k][j],
# which is the jth eigenvector of the
# Pauli matrix σₖ
u0 = PauliVec[1][1]
# define total evolution time in (ns)
tf = 10
# combine H and u0 into an Annealing object
annealing = Annealing(H, u0)

sol_tsit = solve_schrodinger(annealing, tf, alg=Tsit5(), abstol=1e-6, reltol=1e-6);
sol_trbdf = solve_schrodinger(annealing, tf, alg=TRBDF2(), abstol=1e-6, reltol=1e-6);
# LinearExponential is a fixed-stepsize method, the user needs to specify the time steps using the keyword argument `tstops`.
sol_linexp = solve_schrodinger(annealing, tf, alg=LinearExponential(), abstol=1e-6, reltol=1e-6, tstops=range(0,tf,length=100));
# Even though Exprb is an adaptive method, it tends to skip a lot of middle points. So if you want an accurate solution in the middle,
# it is better to manually add more points.
sol_exprb32 = solve_schrodinger(annealing, tf, alg=Exprb32(), tstops=range(0,tf,length=100));

# this code block shows how to plot the expectation value of X
t_list = range(0,tf,length=100)
tsit = []
trbdf = []
linexp = []
exprb32 = []
for s in t_list
    # sol_tsit(s)'*σx*sol_tsit(s) calculates the
    # expectation value <ψ(s)|X|ψ(s)>
    push!(tsit, real(sol_tsit(s)'*σx*sol_tsit(s)))
    push!(trbdf, real(sol_trbdf(s)'*σx*sol_trbdf(s)))
    push!(linexp, real(sol_linexp(s)'*σx*sol_linexp(s)))
    push!(exprb32, real(sol_exprb32(s)'*σx*sol_exprb32(s)))
end
scatter(t_list[1:3:end], tsit[1:3:end], label="Tsit", marker=:+, markersize=8)
scatter!(t_list[2:3:end], trbdf[2:3:end], label="TRBDF")
scatter!(t_list[3:3:end], linexp[3:3:end], label="LinExp", marker=:d)
plot!(t_list, exprb32, label="Exprb", linestyle=:dash)
xlabel!("t (ns)")
ylabel!("<X>")
title!("Free Evolution")

annealing = Annealing(H, u0)
sol_tsit = solve_von_neumann(annealing, tf, alg=Tsit5(), abstol=1e-6, reltol=1e-6)

sol_tsit(0.5)

sol_bdf = solve_von_neumann(annealing, tf, alg=TRBDF2(), reltol=1e-6, vectorize=true)
sol_bdf(0.5 * tf)

sol_linexp = solve_von_neumann(annealing, tf, alg=LinearExponential(), tstops=range(0,tf,length=100));
sol_linexp = solve_von_neumann(annealing, tf, alg=LinearExponential(), tstops=range(0,tf,length=100), vectorize=true);

sol_tsit = solve_von_neumann(annealing, tf, alg=Tsit5(), reltol=1e-6);
sol_trbdf = solve_von_neumann(annealing, tf, alg=TRBDF2(), reltol=1e-6);

sol_linexp = solve_von_neumann(annealing, tf, alg=LinearExponential(), tstops=range(0,tf,length=100), vectorize=true);

sol_exprb32 = solve_von_neumann(annealing, tf, alg=Exprb32(), tstops=range(0,tf,length=100), vectorize=true);

t_list = range(0,tf,length=100)
tsit = []
trbdf = []
linexp = []
exprb32 = []
for s in t_list
    push!(tsit, real(tr(sol_tsit(s)*σx)))
    push!(trbdf, real(tr(sol_trbdf(s)*σx)))
    push!(linexp, real(tr(σx*reshape(sol_linexp(s),2,2))))
    push!(exprb32, real(tr(σx*reshape(sol_exprb32(s),2,2))))
end
scatter(t_list[1:3:end], tsit[1:3:end], label="Tsit", marker=:+, markersize=8)
scatter!(t_list[2:3:end], trbdf[2:3:end], label="TRBDF")
scatter!(t_list[3:3:end], linexp[3:3:end], label="LinExp", marker=:d)
plot!(t_list, exprb32, label="Exprb", linestyle=:dash)
xlabel!("t (ns)")
ylabel!("<X>")
title!("Free Evolution")

annealing = Annealing(H, u0)
sol_tsit = solve_unitary(annealing, tf, alg=Tsit5(),abstol=1e-6, reltol=1e-6)
sol_tsit(0.5 * tf)

sol_tsit = solve_unitary(annealing, tf, alg=Tsit5(), reltol=1e-6);
sol_trbdf = solve_unitary(annealing, tf, alg=TRBDF2(), reltol=1e-6, vectorize=true);
# LinearExponential is a fixed step size method, users need to specify the time steps using the keyword argument `tstops`.
sol_linexp = solve_unitary(annealing, tf, alg=LinearExponential(), tstops=range(0,tf,length=100), vectorize=true);
# Even though Exprb method is an adaptive method, it tends to skip a lot of middle points. So if you want an accurate evolution in the middle,
# it is better to manually add more points for the algorithm.
sol_exprb32 = solve_unitary(annealing, tf, alg=Exprb32(), tstops=range(0,tf,length=100), vectorize=true);

t_list = range(0,tf,length=100)
tsit = []
trbdf = []
linexp = []
exprb32 = []
for s in t_list
    state_tsit = sol_tsit(s) * u0
    state_trbdf = reshape(sol_trbdf(s), 2, 2) * u0
    state_linexp = reshape(sol_linexp(s), 2, 2) * u0
    state_exprb32 = reshape(sol_exprb32(s), 2, 2) * u0
    push!(tsit, real(state_tsit' * σx * state_tsit))
    push!(trbdf, real(state_trbdf' * σx * state_trbdf))
    push!(linexp, real(state_linexp' * σx * state_linexp))
    push!(exprb32, real(state_exprb32' * σx * state_exprb32))
end
scatter(t_list[1:3:end], tsit[1:3:end], label="Tsit", marker=:+, markersize=8)
scatter!(t_list[2:3:end], trbdf[2:3:end], label="TRBDF")
scatter!(t_list[3:3:end], linexp[3:3:end], label="LinExp", marker=:d)
plot!(t_list, exprb32, label="Exprb", linestyle=:dash)
xlabel!("t (ns)")
ylabel!("<X>")
title!("Free Evolution")

using HOQSTTutorials
HOQSTTutorials.tutorial_footer(WEAVE_ARGS[:folder],WEAVE_ARGS[:file])


## auc.jl (c) Area Under the ROC
## (c) 2016 David A. van Leeuwen
##
## Licensed under the MIT software license, see LICENSE.md

"""`AUC(::roc; pfa=1.0, pmiss=1.0, normalize=true)` computes the Area Under the Curve
with a sense `auc → 0` indicating better performance.

Optional parameters `pfa` or `pmiss` limit integration over only part of the ROC curve.
`normalize` indicates comparing the partial ROC to the trivial ROC.
"""
function auc(r::Roc; pfa=1.0, pmiss=1.0, normalize=true)
    if pfa != 1.0 || pmiss != 1.0
        if pfa != 1.0 && pmiss != 1.0 || ! (0.0 < pfa ≤ 1.0) || !(0.0 < pmiss ≤ 1.0)
            throw(DomainError)
        end
        if pfa < 1.0
            i = binsearch(-pfa, -r.pfa) ## find threshold where FAR >= pfa
            i < length(r) || error("Unexpected ROC data")
            norm = normalize ? 1/pfa/(2-pfa) : 1
            a = -dot(r.pmiss[i:end-1] + r.pmiss[i+1:end], diff(r.pfa[i:end])) / 2
            ## compensate for the the fact that pfa > r.pfa[i]
            a -= (r.pfa[i] - pfa) * (r.pmiss[i] - (r.pfa[i] - pfa) * (r.pmiss[i+1] - r.pmiss[i]) / 2(r.pfa[i+1] - r.pfa[i]))
            return norm * a
        else
            i = binsearch(pmiss, r.pmiss)
            i < length(r) || error("Unexpected ROC data")
            norm = normalize ? 1/pmiss/(2-pmiss) : 1
            a = dot(r.pfa[1:i] + r.pfa[2:i+1], diff(r.pmiss[1:i+1])) / 2
            a -= (r.pmiss[i+1] - pmiss) * (r.pfa[i+1] - (r.pmiss[i+1] - pmiss) * (r.pfa[i+1] - r.pfa[i]) / 2(r.pmiss[i+1] - r.pmiss[i]))
            return norm * a
        end
    else
        ## The probability that a random non-target score is higher than a target score
        return -dot(r.pmiss[1:end-1] + r.pmiss[2:end], diff(r.pfa)) / 2
    end
end

auc(tar::Vector, non::Vector; kwargs...) = auc(roc(tar, non); kwargs...)

"""`AUC(::roc; pfa=1.0, pmiss=1.0, normalize=true)` computes the traditional Area Under the Curve
with a sense `AUC → 1` indicating better performance.

Optional parameters `pfa` or `pmiss` limit integration over only part of the ROC curve.
`normalize` indicates comparing the partial ROC to the trivial ROC.

You can also call this function as `AUC(targets::Vector, nontargets::Vector; kwargs...)`
"""
function AUC(args...; pfa=1.0, pmiss=1.0, normalize=true)
    if normalize
        return 1 - auc(args...; normalize=normalize, pfa=pfa, pmiss=pmiss)
    else
        return min(pfa, pmiss) - auc(args...; normalize=normalize, pfa=pfa, pmiss=pmiss)
    end
end


# This file is part of the Julia package ModularForms.jl
#
# Copyright (c) 2018-2019: Lynn Engelberts and Alexandru Ghitza.

"""
    eta_quotient(g, prec=10)

Return the eta-quotient relative to the given collection of integers
`g` as a power series up to precision `prec`.

The input g is an array of pairs [[t_1,r_1], [t_2,r_2], ..., [t_s,r_s]]
where each t_j is a positive integer and each r_j a nonnegative integer.
An error is thrown if the sum of t_j*r_j for j in {1, ..., s} does not
equal 24, as the function only applies to eta-quotients that are cusp
forms. The output is a cusp form of integral weight.

# Arguments
- `g::Array{Array}`: an array of pairs, collection of integers
- `prec::Integer=10`: precision of the output

# Examples still missing
"""
function eta_quotient(g::Array{Array{Int,1},1}, prec::Int=10)

   R, q = PolynomialRing(ZZ, "q")
   poly = R(1)		#product
   sum = 0 		#error handling
   s = length(g)
   for i in 1:s
      t = g[i][1]
      r = g[i][2]
      poly = poly * eta_qexp(r, prec, q^t)	#check prec
      sum += t*r
   end

   if sum != 24
      error("sum of t_j*r_j for all j must equal 24 if it is a cusp form")
   end

   poly = q * poly
   power_series = poly_to_power_series(poly, prec)

   return power_series

end


#=
acceleration.jl

    Acceleration routines for proximal optimization algorithms, such as simple
    extrapolation steps, Nesterov momentum steps, etc.

@author: Quint Wiersma <q.wiersma@vu.nl>

@date: 2022/01/18
=#

"""
    update_acc!(acc, state)

Update the extrapolation parameter `ω` using a specific acceleration scheme,
such as simple extrapolation and Nesterov momentum.

#### Arguments
  - `acc::AbstractAccelScheme`	: acceleration scheme variables
  - `state::ProxGradState`		: state variables
"""
function update_acc! end

function update_acc!(acc::NoAccel, state::ProxGradState)
	# Do nothing...

	return nothing
end

function update_acc!(acc::Simple, state::ProxGradState)
	# Gradient based adaptive restart
    acc.k= dot(state.∇f, state.Δ) > zero(eltype(state.Δ)) ? 1 : acc.k

    # Simple extrapolation
    acc.ω= (acc.k - 1)*inv(acc.k + 2)

	return nothing
end

function update_acc!(acc::Nesterov, state::ProxGradState)
	# Gradient based adaptive restart
    acc.θ= dot(state.∇f, state.Δ) > zero(eltype(state.Δ)) ? one(acc.θ) : acc.θ

	# Clean up notation
	λ_prev= acc.ls.λ_prev
	λ= acc.ls.λ
	θ_prev= acc.θ

    # Quadratic formula
    b = θ_prev^2 * inv(λ_prev) - acc.m
    D= b^2 + 4 * inv(λ_prev * λ) * θ_prev^2

    # Update Nesterov momentum
    acc.θ= .5 * λ * (-b + sqrt(D))

    # Nesterov extrapolation
    acc.ω= λ * θ_prev * (one(θ_prev) - θ_prev) * inv(λ_prev * acc.θ + λ * θ_prev^2)

	return nothing
end


"""
ДАНО: Робот - в произвольной клетке ограниченного прямоугольного поля, на котором могут находиться также внутренние прямоугольные перегородки (все перегородки изолированы друг от друга, прямоугольники могут вырождаться в отрезки)

РЕЗУЛЬТАТ: Робот - в исходном положении, и в 4-х приграничных клетках, две из которых имеют ту же широту, а две - ту же долготу, что и Робот, стоят маркеры.
"""
function coordinat_marker(r::Robot)
    steps=0
    arr_of_steps=[]
    while ((isborder(r,Sud)) && (isborder(r,West)))==false
        push!(arr_of_steps,moves!(r,West))
        push!(arr_of_steps,moves!(r,Sud))
        steps+=2
    end
    vertical=0 #Счётчик - Движение по вертикали
    horizontal=0 #Счётчик - Движение по горизонтали
    for a in 1:steps
        if isodd(a)==true #Проверка на нечётность
            horizontal+=arr_of_steps[a] #Суммирование, значения берутся из массива
        else
            vertical+=arr_of_steps[a]
        end
    end
    side=Nord
    for b in 1:2 #Движение по вертикали и горизонтали
        vertical=coordinater(r,side,vertical)
        side=next(side)
        horizontal=coordinater(r,side,horizontal)
        side=next(side)
    end
    while (steps>0)==true #Аналогичная функция описана в Задаче 5
        side=isodd(steps) ? Ost : Nord
        move_by_points(r,side,arr_of_steps[steps])
        steps-=1
    end
end

function coordinater(r::Robot, side::HorizonSide, num_steps::Int)#Движение и маркировка
    move_by_points(r,side,num_steps)
    putmarker!(r)
    num_steps=moves!(r,side)
    return num_steps
end

function moves!(r::Robot,side::HorizonSide) #Возвращает количество шагов
    num_steps=0
    while isborder(r,side)==false
        move!(r,side)
        num_steps+=1
    end
    return num_steps
end

function move_by_points(r::Robot, side::HorizonSide, num_steps::Int) #Движение в заданную сторону
    for b in 1:num_steps #Количество шагов определено
        move!(r,side)
    end
end

function next(side::HorizonSide) #Возможно использование библиотечного файла
    HorizonSide(mod(Int(side)+3,4))
end
#=Возможна реализация с помощью других функций, в частности, <sum> - суммирование элементов массива=#


include("./Types.jl")
import Polyhedra, CDDLib
export primal
"""
    primal(Γ, P1, P2, D2, Cone, Dualcone ; ϵ=0.05)

implements Algorithm1 in
Löhne, A., Rudloff, B. & Ulus, F. Primal and dual approximation algorithms for convex vector optimization problems. J Glob Optim 60, 713–736 (2014). [Download](https://doi.org/10.1007/s10898-013-0136-0)
"""
function primal(Γ, P1, P2, D2, Cone, Dualcone; ϵ = 0.05)
    # any linearly independent q vectors in cone C in R^q
    # c1, c2, ..., cq-1, c where c in intC
    C = Cone
    # q geberators of vectors of the dual C+
    (_, q) = size(Cone)
    T = C
    Z = Dualcone
    # lines 1 and 3
    Xbar = [first(P1(Z[:, i])) for i = 1:q]
    # line 2
    Hs = [Polyhedra.HalfSpace(-Z[:, i], Z[:, i]' * Xbar[i]) for i = 1:q]
    P_hr = Polyhedra.hrep(Hs)
    P_poly = Polyhedra.polyhedron(P_hr)
    # line 3
    k = 1
    Tinv = inv(T)
    Tbar = [Float64.(T' * Z[:, i]) for i = 1:q]
    # line 4
    # D = D*(Tbar) = [Tbar₁, Tbar₂, ⋯, Tbar_q,   ]
    D = hcat([Tbar[i][1:q-1][1] for i in 1:q], [Z[:, i]' * Γ(Xbar[i]) for i = 1:q])

    # bags to collect the vertices
    Vknown = []
    # flag to break the loop
    go = true
    while go
        # line 7
        P_vr = Polyhedra.vrep(P_poly)
        # extract number of vecrtices from P_vr
        # that is |Pⱽ|
        num_v = Polyhedra.npoints(P_vr)
        # extract vecrtices from P_vr as row
        v = ((Polyhedra.MixedMatVRep(P_vr)).V)'
        vv = [v[:, i] for i = 1:num_v]
        ## by default stop the loop
        go = false

        # for each vertex found
        for v0 in vv
            # check if this vertex has been visited before?
            if ~(Vertex2D(v0...) ∈ Vertex2D.(Vknown))
                # if the vertex has not been visited
                # v0_as_vertic = Vertex2D(v0...)
                ## store this vertex as visited
                # Vknown[k] = v0_as_vertic
                ## collect it in the bag
                push!(Vknown, v0)
                # increase the index k
                k = k + 1
                # line 10
                (xv0, zv0) = P2(v0)
                (wv0, uv0) = D2(v0)
                # update Xbar and Tbar line 11
                push!(Xbar, xv0)
                tv0 = T'wv0
                push!(Tbar, tv0)
                Dstar_q = (wv0'*v0)[1] + zv0
                D = hcat(D, [tv0[1]; Dstar_q])
                # since zv0>ϵ we compute the huperplane in line 13
                if (zv0 > ϵ)
                    # H0Mat = [tv0[1:end-1][1] 1] * Tinv
                    H0Mat = (vcat(tv0[1:end-1], 1))' * Tinv
                    # if H = {y∈ Rq | aᵀ*y≤ b}, we construct it as
                    # Polyhedra.HalfSpace(a, b)
                    # if Hⱼ = {y∈ Rq | (zʲ)ᵗy≥ (zʲ)ᵗΓ(xⱼ)}
                    ## This halfspace is constructed as follows
                    # The HalfSpace of Polyhedra package demands that
                    #    Hⱼ = {y∈ Rq | ϕ(y,D*(Tᵀzʲ))≥0} = {y∈ Rq | ϕ(y,D*(Tᵀzʲ))≥0}
                    #    Hⱼ = {y∈ Rq | [(Tᵀzʲ)₁ (Tᵀzʲ)₂ ⋯ (Tᵀzʲ)q 1]*Tinv*y -D*(Tᵀzʲ)≥0}
                    #    Hⱼ = {y∈ Rq | [(Tᵀzʲ)₁ (Tᵀzʲ)₂ ⋯ (Tᵀzʲ)q 1]*Tinv*y≥D*(Tᵀzʲ)}
                    #    Hⱼ = {y∈ Rq | H0Mat*y≥Dstar_q}
                    # we write as
                    #    Hⱼ = {y∈ Rⁿ | - (zⱼ)ᵗy≤ - (zⱼ)ᵗΓ(xⱼ)}
                    # ∴  Hⱼ = {y∈ Rq | -H0Mat*y≤ -Dstar_q}
                    # i.e aᵀ = -H0Mat ⇒ a = -H0Mat'
                    #     b = -Dstar_q
                    H0 = Polyhedra.HalfSpace(-H0Mat', -Dstar_q)
                    P_hr = P_poly.hrep ∩ H0
                    ## we compute P_{k+1} in line 18 and break the loop
                    P_poly = Polyhedra.polyhedron(P_hr)
                    go = true
                end
            end
        end

    end
    return Xbar, Tbar, Vknown, P_poly
end


#=
timeseries decompositions, i.e. separating some data into components
=#
export seasonal_decomposition
"""
    seasonal_decomposition(A::ClimArray, fs = [1, 2]) → seasonal, residual
Decompose `A` into a seasonal and residual components, where the seasonal contains the
periodic parts of `A`, with frequencies given in `fs`, and residual contains what's left.

`fs` is measured in 1/year. This function works even for non-equispaced time axis (e.g.
monthly averages) and uses LPVSpectral.jl and SignalDecomposition.jl.
"""
function seasonal_decomposition(A::AbDimArray, fs::Vector)
    @assert hasdim(A, Time)
    E = _numbertype(eltype(A))
    method = Sinusoidal(E.(fs ./ DAYS_IN_ORBIT))
    seasonal = DimensionalData.basetypeof(A)(copy(Array(A)), dims(A))
    residual = DimensionalData.basetypeof(A)(copy(Array(A)), dims(A))

    t = dims(A, Time).val
    truetime = realtime_days(t, E)
    for i in otheridxs(A, Time)
        y = Array(A[i...])
        sea, res = SignalDecomposition.decompose(truetime, y, method)
        seasonal[i...] .= sea
        residual[i...] .= res
    end
    return seasonal, residual
end

function seasonal_decomposition(A::AbDimArray{T, 1}, fs::Vector) where {T}
    @assert hasdim(A, Time)
    E = _numbertype(eltype(A))
    method = Sinusoidal(E.(fs ./ DAYS_IN_ORBIT))
    seasonal = DimensionalData.basetypeof(A)(copy(Array(A)), dims(A))
    residual = DimensionalData.basetypeof(A)(copy(Array(A)), dims(A))

    t = dims(A, Time).val
    truetime = realtime_days(t, E)
    y = Array(A)
    sea, res = SignalDecomposition.decompose(truetime, y, method)
    seasonal .= sea
    residual .= res
    return seasonal, residual
end


using SafeTestsets

@safetestset "Kalman Filter tests" begin include("kalman_filter_tests.jl") end
@safetestset "Likelihood tests" begin include("likelihood_tests.jl") end
@safetestset "ODE tests" begin include("ode_tests.jl") end


using ITensors
using ITensorMPOCompression
using Test
using Printf

#
# Single index tests
#
function test_qx(::Type{ElT},qx::Function,M::Int64,N::Int64,pos::Bool) where {ElT<:Number}
    ir=Index(M,"row")
    ic=Index(N,"col")
    A=randomITensor(ElT,ir,ic)
    Q,LR=qx(A,ir;positive=pos)
    norm(A-Q*LR)
end

function test_xq(::Type{ElT},xq::Function,M::Int64,N::Int64,pos::Bool) where {ElT<:Number}
    ir=Index(M,"row")
    ic=Index(N,"col")
    A=randomITensor(ElT,ir,ic)
    LR,Q=xq(A,ir;positive=pos)
    norm(A-LR*Q)
end

@testset "QR decomposition, single index" begin
    eps=1e-14
    M,N=5,5
    M1,N1=10,10
    @test test_qx(Float64   ,qr,M ,N,false)<eps
    @test test_qx(Float64   ,qr,M ,N1,false)<eps
    @test test_qx(Float64   ,qr,M1,N1,false)<eps
    @test test_qx(Float64   ,qr,M ,N ,true )<eps
    @test test_qx(Float64   ,qr,M ,N1,true )<eps
    @test test_qx(Float64   ,qr,M1,N1,true )<eps
    @test test_qx(ComplexF64,qr,M ,N ,false)<eps
    @test test_qx(ComplexF64,qr,M ,N1,false)<eps
    @test test_qx(ComplexF64,qr,M1,N1,false)<eps
    @test test_qx(ComplexF64,qr,M ,N ,true )<eps
    @test test_qx(ComplexF64,qr,M ,N1,true )<eps
    @test test_qx(ComplexF64,qr,M1,N1,true )<eps
end

@testset "QL decomposition, single index" begin
    eps=1e-14
    M,N=5,5
    M1,N1=10,10
    @test test_qx(Float64   ,ql,M ,N,false)<eps
    @test test_qx(Float64   ,ql,M ,N1,false)<eps
    @test test_qx(Float64   ,ql,M1,N1,false)<eps
    @test test_qx(Float64   ,ql,M ,N ,true )<eps
    @test test_qx(Float64   ,ql,M ,N1,true )<eps
    @test test_qx(Float64   ,ql,M1,N1,true )<eps
    @test test_qx(ComplexF64,ql,M ,N ,false)<eps
    @test test_qx(ComplexF64,ql,M ,N1,false)<eps
    @test test_qx(ComplexF64,ql,M1,N1,false)<eps
    @test test_qx(ComplexF64,ql,M ,N ,true )<eps
    @test test_qx(ComplexF64,ql,M ,N1,true )<eps
    @test test_qx(ComplexF64,ql,M1,N1,true )<eps
end

@testset "LQ decomposition, single index" begin
    eps=1e-14
    M,N=5,5
    M1,N1=10,10
    @test test_xq(Float64   ,lq,M ,N,false)<eps
    @test test_xq(Float64   ,lq,M ,N1,false)<eps
    @test test_xq(Float64   ,lq,M1,N1,false)<eps
    @test test_xq(Float64   ,lq,M ,N ,true )<eps
    @test test_xq(Float64   ,lq,M ,N1,true )<eps
    @test test_xq(Float64   ,lq,M1,N1,true )<eps
    @test test_xq(ComplexF64,lq,M ,N ,false)<eps
    @test test_xq(ComplexF64,lq,M ,N1,false)<eps
    @test test_xq(ComplexF64,lq,M1,N1,false)<eps
    @test test_xq(ComplexF64,lq,M ,N ,true )<eps
    @test test_xq(ComplexF64,lq,M ,N1,true )<eps
    @test test_xq(ComplexF64,lq,M1,N1,true )<eps
end

@testset "RQ decomposition, single index" begin
    eps=1e-14
    M,N=5,5
    M1,N1=10,10
    @test test_xq(Float64   ,rq,M ,N,false)<eps
    @test test_xq(Float64   ,rq,M ,N1,false)<eps
    @test test_xq(Float64   ,rq,M1,N1,false)<eps
    @test test_xq(Float64   ,rq,M ,N ,true )<eps
    @test test_xq(Float64   ,rq,M ,N1,true )<eps
    @test test_xq(Float64   ,rq,M1,N1,true )<eps
    @test test_xq(ComplexF64,rq,M ,N ,false)<eps
    @test test_xq(ComplexF64,rq,M ,N1,false)<eps
    @test test_xq(ComplexF64,rq,M1,N1,false)<eps
    @test test_xq(ComplexF64,rq,M ,N ,true )<eps
    @test test_xq(ComplexF64,rq,M ,N1,true )<eps
    @test test_xq(ComplexF64,rq,M1,N1,true )<eps
end

#
#  Test multiple indicies using MPO matrices
#
@testset "QR,QL,LQ,RQ decomposition fo MPO matrices" begin
    N=5
    NNN=4
    hx=0.5
    eps=2e-15
    sites = siteinds("SpinHalf", N)
    H=make_transIsing_MPO(sites,NNN,hx)
    W=H[2]
    d,n,r,c=parse_links(W)

    Lind=noncommoninds(inds(W),c)
    Rind=noncommoninds(inds(W),r)

    Q,R=qr(W,Lind;positive=true)
    iqr=commonindex(Q,R)
    Id=Q*prime(Q,iqr)
    Idq=delta(iqr,iqr')
    @test norm(Id-Idq)<eps
    @test norm(W-Q*R)<eps

    Q,L=ql(W,Lind;positive=true)
    iql=commonindex(Q,L)
    Id=Q*prime(Q,iql)
    Idq=delta(iql,iql')
    @test norm(Id-Idq)<eps
    @test norm(W-Q*L)<eps

    L,Q=lq(W,Rind;positive=true)
    iql=commonindex(Q,L)
    Id=Q*prime(Q,iql)
    Idq=delta(iql,iql')
    @test norm(Id-Idq)<eps
    @test norm(W-L*Q)<eps

    R,Q=rq(W,Rind;positive=true)
    iql=commonindex(Q,R)
    Id=Q*prime(Q,iql)
    Idq=delta(iql,iql')
    @test norm(Id-Idq)<eps
    @test norm(W-R*Q)<eps
end

#using Printf
#Base.show(io::IO, f::Float64) = @printf(io, "%1.3f", f)
#println("-----------Start--------------")

@testset "QR,QL,LQ,RQ decomposition with rank revealing" begin
    N=10
    NNN=6
    hx=0.5
    eps=2e-15
    sites = siteinds("SpinHalf", N)
    #
    #  use lower tri MPO to get some zero pivots for QL and RQ.
    #
    H=make_transIsing_MPO(sites,NNN,hx,lower)
    W=H[2]
    d,n,r,c=parse_links(W)

    Lind=noncommoninds(inds(W),c)
    Rind=noncommoninds(inds(W),r)
    @assert dim(c)==dim(r)
    #
    #  RQ decomp
    #
    R,Q=rq(W,Rind;positive=true,rank=true)
    iq=commonindex(Q,R)
    @printf "RQ decomposition %4i rows were removed from R\n" dim(c)-dim(iq)
    Id=Q*prime(Q,iq)
    Idq=delta(iq,iq')
    @test norm(Id-Idq)<eps
    @test norm(W-R*Q)<eps
    #
    #  QL decomp
    #
    Q,L=ql(W,Rind;positive=true,rank=true)
    iq=commonindex(Q,L)
    @printf "QL decomposition %4i rows were removed from L\n" dim(c)-dim(iq)
    Id=Q*prime(Q,iq)
    Idq=delta(iq,iq')
    @test norm(Id-Idq)<eps
    @test norm(W-L*Q)<eps

    #
    #  use upper tri MPO to get some zero pivots for LQ and QR.
    #
    H=make_transIsing_MPO(sites,NNN,hx,upper)
    W=H[2]
    d,n,r,c=parse_links(W)

    Lind=noncommoninds(inds(W),c)
    Rind=noncommoninds(inds(W),r)
    @assert dim(c)==dim(r)

    #
    #  QR decomp
    #
    Q,R=ITensorMPOCompression.qr(W,Rind;positive=true,rank=true)
    iq=commonindex(Q,R)
    @printf "QR decomposition %4i rows were removed from R\n" dim(c)-dim(iq)
    Id=Q*prime(Q,iq)
    Idq=delta(iq,iq')
    @test norm(Id-Idq)<eps
    @test norm(W-R*Q)<eps
    #
    #  LQ decomp
    #
    L,Q=lq(W,Rind;positive=true,rank=true)
    iq=commonindex(Q,L)
    @printf "LQ decomposition %4i rows were removed from L\n" dim(c)-dim(iq)
    Id=Q*prime(Q,iq)
    Idq=delta(iq,iq')
    @test norm(Id-Idq)<eps
    @test norm(W-L*Q)<eps

end
 #=
@testset "QR,QL,LQ,RQ decomposition for block sparse matrices" begin
    N=5
    NNN=2
    Dw=3
    hx=0.0 #can't make and sx op with QNs in play
    ul=lower
    lr=left
    ms=matrix_state(ul,lr)

    rowq=Index(QN("Sz",0)=>Dw;dir=ITensors.In,tags="Link,l=1") #link index with qn's
    colq=Index(QN("Sz",0)=>Dw;dir=ITensors.Out,tags="Link,l=2")
    sites = siteinds("S=1/2",N;conserve_qns=true) #sites with qn's
    H=make_transIsing_MPO(sites,NNN,hx,ul)
    W=H[2]
    pprint(W,1e-14)
    # Q,R,lq=block_qx(W,ul;orth=lr)

    off=V_offsets(ms)
    Vblock=getV(W,off)
    Rind=filterinds(Vblock,tags="l=2")
    Linds=noncommoninds(Vblock,Rind)
    Vd=dense(Vblock)
    Qd,Ld,iq=ql(Vd,Linds;positive=true,epsrr=0.0)
    set_scale!(Ld,Qd,off) #rescale so the L(n,n)==1.0
    pprint(Qd,1e-14)
    # @show Qd Ld

    # Vt=NDTensors.tensor(Vblock)
    # for b in eachnzblock(Vt)
    #     bl=NDTensors.blockview(Vt, b)
    #     @show Vt
    # end
    Q,L,iq=ql(Vblock,Linds;positive=true,epsrr=0.0)
    # U,s,V,iu,iv=svd(Vblock,Linds)
    #@show U V
end =#


count=0;

function c(x)
    return sin(5000*x)*x;
end

function AdaptiveSimpson(f,a,b; abstol=1.0e-6)
    nf=3;
    ff=map(f,[a,(a+b)/2,b]);
    nf=3;
    I1=(b-a)*dot([1,4,1],ff)/6;
    function adaptrec(f,a,b,ff,I1,tol,nf)
        h=(b-a)/2;
        fm=map(f,[a+h/2,b-h/2]);
        nf=nf+2;

        fR=[ff[2],fm[2],ff[3]];
        fL=[ff[1],fm[1],ff[2]];
        IL=h*dot([1,4,1],fL)/6;
        IR=h*dot([1,4,1],fR)/6;
        I2=IL+IR;
        I=I2+(I2-I1)/15;

        if(abs(I-I2)>tol)
            IL,nf = adaptrec(f,a,a+h,fL,IL,tol/2,nf);
            IR,nf = adaptrec(f,b-h,b,fR,IR,tol/2,nf);
            I=IL+IR;
        end
        return I,nf;
    end
    return adaptrec(f,a,b,ff,I1,abstol,nf);
end

println(AdaptiveSimpson(c,-1,1));
println(count);


function calc_napier()
    n_trial =convert(Int,1e8)
    total_count=0
    for i in 1:n_trial
        counter = 0
        accumulated = 0.0
        while true
            accumulated += rand()
            counter += 1
            if accumulated >= 1.0
                break
            end
        end
        total_count+=counter
    end
    total_count/n_trial
end


# Section 6.1 of
# [RMT13] Rungger, Matthias and Mazo Jr, Manuel and Tabuada, Paulo
# Specification-guided controller synthesis for linear systems and safe linear-time temporal logic
# Proceedings of the 16th international conference on Hybrid systems: computation and control, 2013

using Compat, Compat.LinearAlgebra
using FillArrays
import LightGraphs
using MathematicalSystems
using HybridSystems
using Polyhedra

"""
    cruise_control_example(N, M; vmin = 5., vmax = 35., v = (15.6, 24.5), U = 4, D = 0.5, ks = 4500, kd = 4600, m = 1000, m0 = 100, H = 0.8, T = 2, h = H / T, sym = false)

Hybrid System representing the speed of a truck and its `M` trailers.
The mass of the truck is `m0` and the mass of each trailer is `m`.
The dynamic is discretized over step sizes of length `h`.
"""
function cruise_control_example(N, M; vmin = 5., vmax = 35., v = (15.6, 24.5), U = 4, D = 0.5, ks = 4500, kd = 4600, m = 1000, m0 = 100, H = 0.8, T = 2, h = H / T, sym = false, lib::Polyhedra.Library = Polyhedra.default_library(2M+2, Float64))
    function Pv(v, maxspeed)
        s = maxspeed ? 1. : -1.
        Pvi = intersect(HalfSpace([0, s], s*v))
        Pv0 = intersect(HalfSpace([s, 0], s*v))
        if M >= 1
            Base.power_by_squaring(Pvi, M) * Pv0
        else
            Pv0
        end
    end
    PD = HalfSpace([-1., 0], D) ∩ HalfSpace([1., 0], D)
    PU = HalfSpace([0., -1], U) ∩ HalfSpace([0, 1.], U)
    if M >= 1
        P0 = Base.power_by_squaring(PD, M) * PU
    else
        P0 = PU
    end
    if sym
        Pv0max = Pv(vmax, true) ∩ Pv(-vmax, false)
        Pvimax = Pv.(v, true) .∩ Pv.(.-(v), false)
        P1 = P0
    else
        Pv0max = Pv(vmax, true)
        Pvimax = Pv.(v, true)
        Pvmin = Pv(vmin, false)
        P1 = P0 ∩ Pvmin
    end

    G = LightAutomaton(N)
    if N == 1
        add_transition!(G, 1, 1, 1)
        safe_sets = [P1 ∩ Pvimax[1]]
    elseif N == 2
        add_transition!(G, 1, 2, 1)
        add_transition!(G, 2, 2, 1)
        safe_sets = [P1 ∩ Pv0max, P1 ∩ Pvimax[1]]
    else
        @assert N == 1 + (T+1) * length(v)
        safe_sets = Vector{typeof(P1)}(undef, N)
        function new_speed_signal!(u)
            for (j, vj) in enumerate(v)
                w = (T+1)*(j-1) + T+1
                add_transition!(G, u, w, 2)
            end
        end
        for (i, vi) in enumerate(v)
            for t in 1:(T+1)
                u = (T+1)*(i-1) + t
                if t == 1
                    add_transition!(G, u, u, 1)
                    safe_sets[u] = P1 ∩ Pvimax[i]
                else
                    add_transition!(G, u, u-1, 1)
                    safe_sets[u] = P1 ∩ Pv0max
                end
                new_speed_signal!(u)
            end
        end
        add_transition!(G, N, N, 1)
        safe_sets[N] = P1 ∩ Pv0max
        new_speed_signal!(N)
    end

    d = 2M + 2
    #s = DiscreteLinearControlSystem([1. 0; 0 1], reshape([1.; 0], 2, 1), U)
    A = zeros(d-1, d-1)
    for i in 1:M
        di = 2i-1
        vi = 2i
        vj = i == 1 ? d-1 : 2(i-1)
        A[di, vi] = -1
        A[di, vj] =  1
        A[vi, vj] =  kd/m
        A[vi, di] =  ks/m
        A[vi, vi] = -kd/m
        if i < M
            dj = 2i+1
            vk = 2(i+1)
            A[vi, vi] += -kd/m
            A[vi, dj]  = -ks/m
            A[vi, vk]  =  kd/m
        end
    end
    A[d-1, 1]   =  ks/m0
    A[d-1, d-1] = -kd/m0
    A[d-1, 2]   =  kd/m0
    A1 = [I+h*A [zeros(d-2); h]
          zeros(d-1)'  0]
    B = reshape([zeros(d-1); 1.], d, 1)
    s1 = LinearControlDiscreteSystem(A1, B)
    A2 = [Matrix(1.0I, d-1, d-1) zeros(d-1)
          zeros(d-1)'            0]
    s2 = LinearControlDiscreteSystem(A2, B)

    sw = AutonomousSwitching()

    M = LightGraphs.ne(G.G)

    S = ConstrainedDiscreteIdentitySystem.(d, polyhedron.(safe_sets, lib))
    Re = [s1, s2]
    Sw = Fill(sw, N)

    HybridSystem(G, S, Re, Sw)
end


function try_symbolic(T, eq, x, basis, Δbasis; kwargs...)
    n = length(basis)
    @syms θ[1:n]

    Δeq = sum(θ[j]*Δbasis[j] for j=1:n) - eq
    Δeq = expand(Δeq)

    terms = collect_terms(Δeq, x)
    eqs = collect(values(terms))
    # eqs = filter(p->length(get_variables(p)) > 0, eqs)

    sol = solve_symbolic(eqs)

    for i = 1:n
        if !haskey(sol, θ[i])
            sol[θ[i]] = 0
        end
    end

    p = substitute(expand(sum(θ[j]*basis[j] for j=1:n)), sol)
    p, 0
end

mutable struct Fragment
    eq
    lhs
end

function solve_symbolic(eqs)
    n = length(eqs)
    solved = Set()
    unfinished = true
    frags = Fragment[]
    k = 1

    while unfinished
        unfinished = false
        for eq in eqs
            δf = [v for v in get_variables(eq) if v ∉ solved]
            if length(δf) == 1
                push!(solved, δf[1])
                push!(frags, Fragment(eq, δf[1]))
                unfinished = true
            end
        end
    end

    sol = Dict()
    for f in frags
        eq = substitute(f.eq, sol)
        u = Symbolics.solve_for(eq ~ 0, f.lhs)
        sol[f.lhs] = nice_parameter(u)
    end

    sys = []
    vars = Set()

    for eq in eqs
        δf = [v for v in get_variables(eq) if v ∉ solved]
        if length(δf) > 1
            for v in δf
                push!(vars, v)
            end
            q = substitute(eq, sol)
            push!(sys, q)
        end
    end

    sys = unique(sys)
    vars = [v for v in vars]

    if !isempty(vars) && length(vars) == length(sys)
        try
            vals = nice_parameters.(Symbolics.solve_for(sys .~ 0, vars))
            # vals = Symbolics.solve_for(sys .~ 0, vars)
            for (v,u) in zip(vars, vals)
                sol[v] = u
            end
        catch e
            # println("from symbolic: ", e)
        end
    end

    sol
end

function collect_terms(eq::SymbolicUtils.Add, x)
    d = Dict{Any,Any}(1 => 0)
    for t in arguments(eq)
        if isdependent(t, x)
            for s in collect_terms(t, x)
                v, c = first(s), last(s)
                if haskey(d, v)
                    d[v] += c
                else
                    d[v] = c
                end
            end
        else
            d[1] += t
        end
    end
    d
end

function collect_terms(eq, x)
    c = coef(eq, x)
    return Dict(eq/c => c)
end


#some useful functions
glorot_uniform(dims...) = (rand(Float64, dims...) .- 0.5) .* sqrt(24.0/sum(dims))

relu(x::Real) = max(zero(x), x)

sigmoid(x::Real) = one(x) / (one(x) + exp(-x))

softplus(x::Real) = ifelse(x > 0, x + log1p(exp(-x)), log1p(exp(x)))


# This file is part of RLinearAlgebra.jl

module ProceduralTestLSVRProjFO

using Test, RLinearAlgebra, LinearAlgebra, Random

@testset "LSVR Projection Full Orthogonal -- Procedural" begin

    # Supertype
    @test supertype(LinSysVecRowProjFO) == LinSysVecRowProjection

    # Testing context
    Random.seed!(1010)

    A = rand(10,5)
    x = rand(5)
    b = A * x

    z = rand(5)

    rsub = LinSysVecRowProjFO()

    # Verify that S (storage) is nothing
    @test isnothing(rsub.S)

    # Verify that S (storage) is appropriately initialized
    @test let
        RLinearAlgebra.rsubsolve!(rsub, z, (A[1,:], b[1]), 1)

        typeof(rsub.S) <: Matrix
    end

    # Verify that solution is orthogonalized against all previous hyperplanes
    @test let
        for i = 1:10
            RLinearAlgebra.rsubsolve!(rsub, z, (A[i,:], b[i]), i)
        end

        norm(A * z - b) < 1e-15
    end
end

end # End Module


using FinancialToolbox, Test, FinancialFFT, FinancialMonteCarlo, ReverseDiff

A = 600.0;
N = 18;

S0 = 100.0;
K = 100.0;
r = 0.02;
T = 1.0;
d = 0.01
sigma = 0.2;
lam = 15.0;
mu1 = 0.03;
sigma1 = 0.02;
zero_rate = ZeroRate(r);

Model = BlackScholesProcess(sigma, Underlying(S0, d));

EUData = EuropeanOption(T, K)
Nsim = 100000;
Nstep = 30;
mc = MonteCarloConfiguration(Nsim, Nstep, FinancialMonteCarlo.AntitheticMC());
analytic_result = blsprice(S0, K, r, T, sigma, d);
function test_reverse_diff(x, y, toll)
    @test abs(sum(x .- y)) < toll
    nothing
end
x = Float64[sigma, S0, r]
method_cm = CarrMadanMethod(A, N);
method_lewis = LewisMethod(700.0, 200000);
method_cm_lewis = CarrMadanLewisMethod(A, N);
f(x) = blsprice(x[2], K, x[3], T, x[1], d);
f_lewis(x) = pricer(BlackScholesProcess(x[1], Underlying(x[2], d)), ZeroRate(x[3]), method_lewis, EUData);
analytic_result = ReverseDiff.gradient(f, x);
@show result_lewis = ReverseDiff.gradient(f_lewis, x);
toll = 1e-2;
@testset "ReverseDiff test" begin
    test_reverse_diff(analytic_result, result_lewis, toll)
end


function SeisSemblance(in,param=Dict())

	dt = get(param,"dt",0.002)
	dx = get(param,"dx",10)
	nt,nx = size(in)
	offset = get(param,"offset",[0:1:nx-1]*dx)
	vmin = get(param,"vmin",1500)
	vmax = get(param,"vmax",3500)
	nv = get(param,"nv",500)
	v = linspace(vmin,vmax,nv)
	tau = [0:4:nt-1]*dt
	ntau = length(tau)
	L = get(param,"L",5)

	S = zeros(ntau,nv)
	for it = 1:ntau
		for iv = 1:nv
			t = sqrt( tau[it]^2 + (offset/v[iv]).^2 )
			s = zeros(2*L+1,nx)
			for ig = -L:L
				ts = t + (ig-1)*dt
				for ix = 1:nx
					is = ts[ix]/dt+1
					i1 = floor(is)
					i2 = i1 + 1
					if (i1>=1 && i2<=nt)
						a = is-i1
						s[ig+L+1,ix] = (1.-a)*in[i1,ix] + a*in[i2,ix]   #Grab sample with Linear interpolation
					end
				end
			end
			#s = s.*H
			s1  = sum( (sum(s,2)).^2)
			s2  = sum( sum(s.^2))
			S[it,iv] = abs(s1-s2)
		end
	end
	S = S/maximum(S[:])

	return S,tau,v
end


@doc raw"""
    postproc(model::AshkinTeller, param::Parameter, obs::MCObservableSet)

# Observables to be calculated
In the following, $m$ is total magnetization per site and $\epsilon$ is total energy per site.

- `"Binder Ratio"`
    - $R := \frac{\left \langle m^4 \right \rangle}{\left \langle m^2 \right\rangle^2}$
- `"Susceptibility"`
    - $\chi := \frac{N}{T}\left(\left\langle m^2\right\rangle\right)$
- `"Connected Susceptibility"`
    - $\frac{N}{T}\left(\left\langle m^2\right\rangle - \left\langle |m| \right\rangle^2\right)$
- `"Binder Ratio sigma"`
    - Binder ratio with respct to ``\sigma``
- `"Susceptibility sigma"`
    - Susceptibility with respct to ``\sigma``
- `"Connected Susceptibility sigma"`
    - Connected susceptibility with respct to ``\sigma``
- `"Binder Ratio tau"`
    - Binder ratio with respct to ``\tau``
- `"Susceptibility tau"`
    - Susceptibility with respct to ``\tau``
- `"Connected Susceptibility tau"`
    - Connected susceptibility with respct to ``\tau``
- `"Specific Heat"`
    - $\frac{N}{T^2}\left(\left\langle \epsilon^2\right\rangle - \left\langle \epsilon \right\rangle^2\right)$
"""
function postproc(model::AshkinTeller, param::Parameter, obs::MCObservableSet)
    nsites = numsites(model)
    T = convert(Float64, param["T"])
    beta = 1.0/T

    jk = jackknife(obs)

    jk["Binder Ratio"] = jk["|Magnetization|^4"] / (jk["|Magnetization|^2"]^2)
    jk["Susceptibility"] = (nsites*beta)*jk["|Magnetization|^2"]
    jk["Connected Susceptibility"] = (nsites*beta)*(jk["|Magnetization|^2"] - jk["|Magnetization|"]^2)

    jk["Binder Ratio sigma"] = jk["Magnetization sigma^4"] / (jk["Magnetization sigma^2"]^2)
    jk["Susceptibility sigma"] = (nsites*beta)*jk["Magnetization sigma^2"]
    jk["Connected Susceptibility sigma"] = (nsites*beta)*(jk["Magnetization sigma^2"] - jk["|Magnetization sigma|"]^2)

    jk["Binder Ratio tau"] = jk["Magnetization tau^4"] / (jk["Magnetization tau^2"]^2)
    jk["Susceptibility tau"] = (nsites*beta)*jk["Magnetization tau^2"]
    jk["Connected Susceptibility tau"] = (nsites*beta)*(jk["Magnetization tau^2"] - jk["|Magnetization tau|"]^2)

    jk["Specific Heat"] = (nsites*beta*beta)*(jk["Energy^2"] - jk["Energy"]^2)
    return jk
end


import DifferentiableBackwardEuler
import NLsolve
import NonlinearEquations
using OrdinaryDiffEq
import Random
import Zygote

include("utilities.jl")
include("inputdeck.jl")
include("ex.jl")

@NonlinearEquations.equations exclude=(dirichletnodes, neighbors, areasoverlengths) function groundwater(h, Ks, neighbors, areasoverlengths, dirichletnodes, dirichleths, Qs, specificstorage, volumes)
	NonlinearEquations.setnumequations(length(h))
	for i = 1:length(h)
		if i in dirichletnodes
			NonlinearEquations.addterm(i, h[i] - dirichleths[i])
		else
			NonlinearEquations.addterm(i, Qs[i] / (specificstorage[i] * volumes[i]))
		end
	end
	for (i, (node_a, node_b)) in enumerate(neighbors)
		for (node1, node2) in [(node_a, node_b), (node_b, node_a)]
			if !(node1 in dirichletnodes) && !(node2 in dirichletnodes)
				NonlinearEquations.addterm(node1, Ks[i] * (h[node2] - h[node1]) * areasoverlengths[i] / (specificstorage[node1] * volumes[node1]))
			elseif !(node1 in dirichletnodes) && node2 in dirichletnodes
				NonlinearEquations.addterm(node1, Ks[i] * (dirichleths[node2] - h[node1]) * areasoverlengths[i] / (specificstorage[node1] * volumes[node1]))
			end
		end
	end
end

groundwater_residuals(us, Ks) = groundwater_residuals(us, Ks, neighbors, areasoverlengths, dirichletnodes, dirichleths, Qs, specificstorage, volumes)
groundwater_h(us, Ks) = groundwater_h(us, Ks, neighbors, areasoverlengths, dirichletnodes, dirichleths, Qs, specificstorage, volumes)
groundwater_Ks(us, Ks) = groundwater_Ks(us, Ks, neighbors, areasoverlengths, dirichletnodes, dirichleths, Qs, specificstorage, volumes)

function unpack(p)
	@assert length(p) == length(neighbors)
	Ks = p[1:length(neighbors)]
	return Ks
end

function f(u, p, t)
	Ks = unpack(p)
	return groundwater_residuals(u, Ks)
end

function f_u(u, p, t)
	Ks = unpack(p)
	return groundwater_h(u, Ks, neighbors, areasoverlengths, dirichletnodes, dirichleths, Qs, specificstorage, volumes)
end

function f_p(u, p, t)
	Ks = unpack(p)
	return groundwater_Ks(u, Ks, neighbors, areasoverlengths, dirichletnodes, dirichleths, Qs, specificstorage, volumes)
end

f_t(u, p, t) = zeros(length(u))

h0 = zeros(size(coords, 2))
h0[dirichletnodes] = dirichleths[dirichletnodes]
p = Ks
tspan = [0, 1e4]
odef = ODEFunction(f; jac=f_u, jac_prototype=f_u(h0, p, 0.0))
prob = ODEProblem(odef, h0, tspan, p)
@time soln_diffeq = solve(prob, ImplicitEuler())
@time soln_dbe = DifferentiableBackwardEuler.steps(h0, f, f_u, f_p, f_t, p, soln_diffeq.t; ftol=1e-8)
sum((soln_diffeq[:, :] .- soln_dbe) .^ 2)
@test isapprox(soln_diffeq[:, :], soln_dbe)
g(p) = DifferentiableBackwardEuler.steps(h0, f, f_u, f_p, f_t, p, soln_diffeq.t; ftol=1e-8)[obsnode, end]
@time dgdp_zygote = Zygote.gradient(g, p)[1]
@test isapprox(dgdp_zygote, grad)#this checks the zygote gradient against the steady state gradient that doesn't use zygote


using QMTK
using Compat.Test

@testset "Utils" begin

using QMTK.Consts.Pauli

@test kronprod(sigmax, sigmax, sigmai) == kron(kron(sigmax, sigmax), sigmai)
@test sigmax ⊗ sigmay ⊗ sigmaz == kron(kron(sigmax, sigmay), sigmaz)

h = @kron sigmax[1] ⊗ sigmaz[3] + sigmax[2] ⊗ sigmay[4]
ans = kronprod(σ₁, σ₀, σ₃, σ₀) + kronprod(σ₀, σ₁, σ₀, σ₂)
@test h == ans

h = @kron sigmax[1] * (sigmaz[2] + sigmay[4])
ans = kronprod(σ₁, σ₃, σ₀, σ₀) + kronprod(σ₁, σ₀, σ₀, σ₂)
@test h == ans

h = @kron sigmax[5] * (sigmax[1] * (sigmaz[2] + sigmay[4]))
ans = kronprod(σ₁, σ₃, σ₀, σ₀, σ₁) + kronprod(σ₁, σ₀, σ₀, σ₂, σ₁)
@test h == ans

end