\subsection{Cloud detection over landmarks with Seviri/MSG}

We focus on the problem of cloud identification over landmarks using Seviri MSG data. This satellite mission constitutes a fundamental tool for weather forecasting, providing images of the full Earth disc every 15 minutes. Matching the landmarks accurately is of paramount importance in image navigation and registration models and geometric quality assessment in the Level 1 instrument processing chain. Detection of clouds over landmarks is an essential step in the MSG processing chain, as undetected clouds are one of the most significant sources of error in landmark matching (see Fig.~\ref{fig:motivation}).

\begin{figure}[t!]
\centerline{\includegraphics[width=8.4cm]{./images/Landmarks_intro.pdf}}
\vspace{-0.25cm}
\caption{Landmarks are essential in image registration and geometric quality assessment. Misclassification of cloud contamination in landmarks degrades the correlation matching, which is a cornerstone for the image navigation and registration algorithms.}
\label{fig:motivation}
\end{figure}

\iffalse
Experimental setup:
\begin{itemize}
    \item Say something about the four different datasets: high, mid, low, night.
    \item Number of training instances used in training is the following: $n\in\{200,300,500,1000,2000\}$ for GPC and $n\in\{500,1000,2000,4000\}$ for GPC-RFF and GPC-VFF.
    \item Datasets are balanced.
    \item Number of original features. Number of Fourier features
    \item Number of test instances
    \item achieve the problem pixel-wise classification
\end{itemize}
\fi
\iffalse

\begin{table}[h!]
\begin{center}
\caption{Selected features for the pixel-base classification approach. Different features are used for the training and test sets for day ($0 \leq \text{SZA} \le 90$\textdegree) and night ($\text{SZA} \geq 90$\textdegree).}\label{tab:feats}
\begin{tabular}{llcc}
\hline
Number & Features & Day & Night \\
\hline
1 & B1 VIS0.6 $\mu$m & \checkmark & $\times$ \\
2 & B2 VIS0.8 $\mu$m & \checkmark & $\times$ \\
3 & B3 NIR1.6 $\mu$m & \checkmark & $\times$ \\
4 & B4 IR3.9 $\mu$m & \checkmark & \checkmark \\
5 & B7 IR8.7 $\mu$m & \checkmark &  $\times$ \\
6 & B9 IR10.8 $\mu$m & \checkmark &  $\times$ \\
7 & B10 IR12.0 $\mu$m & \checkmark & \checkmark \\
8 & Ratio~~$\frac{\text{R}_{0.8\mu m}}{\text{R}_{0.6\mu m}}$ & \checkmark & $\times$ \\
9 & Ratio~~$\frac{\text{R}_{0.6\mu m}-\text{R}_{1.6\mu m}}{\text{R}_{0.6\mu m}+\text{R}_{1.6\mu m}}$ & \checkmark & \checkmark  \\
10& $mean_{3x3}(B1)$  & \checkmark & \checkmark \\
11& $std_{3x3}(B1)$  & \checkmark & $\times$ \\
12& $mean_{5x5}(B1)$  & \checkmark & $\times$ \\
13& $std_{5x5}(B1)$  & \checkmark & $\times$ \\
14& $mean_{3x3}(B9)$  & \checkmark & \checkmark \\
15& $std_{3x3}(B9)$  & \checkmark & \checkmark \\
16& NDVI  & \checkmark & $\times$ \\
\hline
\end{tabular}
\end{center}
\end{table}

\fi

The dataset used in the experiments was provided by EUMETSAT, and contains Seviri/MSG Level 1.5 acquisitions for 200 landmarks of variable size for a whole year (2010). Landmarks mainly cover coastlines, islands, or inland waters. We selected all multispectral images from a particular landmark location, Dakhla (Western Sahara), which involves 35,040 MSG acquisitions with a fixed resolution of $20\times 26$ pixels. In addition, Level 2 cloud products were provided for each landmark observation, so the Level 2 cloud mask~\cite{Derrien05} is used as the best available `ground truth' to validate the results. We framed the problem for this particular landmark as a pixel-wise classification one.

A total amount of $16$ features were extracted from the images, involving band ratios, spatial, contextual and morphological features, and discriminative cloud detection scores. In particular, we considered: 7 channels converted to top of atmosphere (ToA) reflectance (R1, R2, R3, R4) and brightness temperature (BT7, BT9, BT10), 3 band ratios, and 6 spatial features.
On the one hand, the three informative band ratios were: (i) a cloud detection ratio, ${\text{R}_{0.8\mu m}}/{\text{R}_{0.6\mu m}}$; (ii) a snow index, $({\text{R}_{0.6\mu m}-\text{R}_{1.7\mu m}})/({\text{R}_{0.6\mu m}+\text{R}_{1.7\mu m}})$; and (iii) the NDVI, $({\text{R}_{0.8\mu m}-\text{R}_{0.6\mu m}})/({\text{R}_{0.8\mu m}+\text{R}_{0.6\mu m}})$. On the other hand, the six spatial features were obtained by applying average filters of sizes $3\times 3$ and $5\times 5$, as well as a standard deviation filter of size $3\times 3$, on both bands R1 and BT9.

Based on previous studies~\cite{Derrien05,Hocking10}, and in order to simplify the classification task, the different illumination conditions (and hence difficulty) over the landmarks are studied by splitting the day into four ranges (sub-problems) according to the solar zenith angle (SZA) values: \emph{high} (SZA$<$SZA$_{\rm median}$), \emph{mid} (SZA$_{\rm median}<$SZA$<80$\textdegree), \emph{low} (80\textdegree$<$SZA$<$90\textdegree), and \emph{night} (SZA$>$90\textdegree). Therefore, different classifiers are developed for each SZA range.

The final amount of pixels available for each illumination condition is $n=1500000$ for \emph{high}, \emph{mid}, and \emph{night}, and $n=1365083$ for \emph{low}. Moreover, each problem has different dimensionality: all the $d=16$ features were used for the three daylight problems, and $d=6$ was used for the night one (some bands and ratios are meaningless at night).

\iffalse
Each landmark chip is stored in the database with the variables showed on Table \ref{tab:DBchip}, which includes the optical and thermal MSG channels, the MSG level 2 mask and some additional information about the landmark location and acquisition time.

\begin{table*}[t!]
\begin{center}
\caption{Variables stored in the database for each landmark chip (e.g. landmark 0, Ad Dakhla, Morocco).} \label{tab:DBchip}
\begin{tabular}{l l l l l}
\hline
Name & Size & Bytes & Class & Description\\
\hline
H & e.g. $78\times60$ & 37440 & double & High resolution visible SEVIRI channel \\
X & e.g. $26\times20\times11$ & 45760 & double & Hypercube with optical and thermal SEVIRI channels \\
M & e.g. $26\times20$ & 4160 & double & MSG level 2 cloud mask \\
channels & $1\times13$ & 104 & double & SEVIRI channel numbers  \\
id & $1\times1$ & 8 & double    & EUMETSAT landmark identifier \\
name & $1\times9$ & 18 & char   & Name assigned to the landmark geographic location \\
num & $1\times1$ & 8 & double & Sequential landmark number (0-199) \\
centre & $1\times2$ & 16 & double & Pixel coordinates of the landmark center \\
latlon & $1\times2$ & 16 & double & Geographic coordinates of the landmark center \\
time & $1\times14$ & 28 & char & UTC acquisition time (format: YYYYMMDDhhmmss)\\
\hline
\end{tabular}
\end{center}
\end{table*}
Furthermore, in order to take into account the daily and seasonal variability and the different day/night conditions, the solar elevation must be known for each acquisition. The sun position (zenith and azimuth angles at the landmark location) was computed for the center of the landmark as a function of the landmark local time and position~\cite{Reda03}. The sun zenith angle (SZA) is also stored for each landmark chip and allows us to easily select the landmark chips to be analyzed attending to their solar illumination or their real local time acquisition.

The values of the L2 cloud mask provided by EUMETSAT for each chip are: `0': space/no data, `50': water, `100': land, and `200': cloud. We computed basic statistics and found that some masks present space/no-data values, which does these pixels and cloud masks useless for the training and test of the classifiers. The percentage of no-data values and cloudy pixels are computed from the L2 cloud mask and are only used to clean and improve the selection of the training chips.
From the MSG L2 cloud mask we also generated a land-cover (land-water) classification mask {\em per} landmark by computing the max-vote of the mask of all cloud-free landmarks.
Next, we assign the class `land' or `water' of the most voted in each pixel to get a general land-cover mask by landmark.

\fi

\subsection{Cloud detection with the IASI/AVHRR data}\label{sec:data_IAVISA}

The IAVISA dataset is part of the study ``IASI/AVHRR Visual Scenes Analysis and Cloud Detection'' (\url{http://www.brockmann-consult.de/iavisa-info-web/}), whose aim is to improve the IASI cloud detection by optimizing the coefficients used for a predefined set of cloud tests performed in the IASI Level 2 processing. The dataset was derived by visual analysis of globally distributed data, and served as input for the optimization and validation of the IASI cloud detection. Each collected IASI sample was classified concerning its cloudiness based on the visual inspection of the AVHRR Level 1B inside the IASI footprint.
Each sample classifies a single IASI instantaneous field of view (IFOV) as being cloud-free (clear sky, 28\%), partly cloudy low (26\%), partly cloudy high (26\%), or cloudy (20\%). For the sake of simplicity, here we focus on discriminating between cloudy and cloud-free pixels.

In order to ensure the representativeness of the dataset for the natural variability of clouds, labeling further considered additional conditions depending on: 1) the surface type (see Table~\ref{tab:IAVISA}), 2) the climate zone (K\"oppen classification over land, geographical bands over sea), 3) the season and 4) day/night discrimination.
First, the surface type database used as ancillary information was the IGBP (International Geosphere-Biosphere Programme) scene types in the CERES/SARB (Clouds and the Earth's Radiant Energy System, Surface and Atmospheric Radiation Budget) surface map. The 18-class map was used to identify surface properties of a given region. The distribution of the surface types in the map is given in Table~\ref{tab:IAVISA}, showing the even distribution of clouds across land cover types which ensures representativeness (natural variability) of the database. Second, the different climate zones were sampled as follows: tropical ($n=7499$), dry ($n=3237$), temperate ($n=8150$), cold ($n=2205$), and polar zones ($n=3832$). Third, seasonality was also taken into account, and yielded the following distribution: Spring ($n=5862$), Summer ($n=6930$), Autumn ($n=6662$), and Winter ($n=5469$).
The global sampling and some cloudy and cloud-free chips are shown in Fig.~\ref{fig:sampling}. The final database consists of $n=24923$ instances and $d=8461$ original features, which have been summarized to $d=100$ more informative directions through a standard Principal Component Analysis.

\begin{table}[h!]
\small
\begin{center}
\caption{The samples distribution per surface types.}
\label{tab:IAVISA}
\begin{tabular}{|c|c|c|}
\hline
ID & Surface Type & Samples\\ \hline \hline
1 & Evergreen Needle Forrest & 603 \\ \hline
2 & Evergreen Broad Forrest & 876 \\ \hline
3 & Deciduous Needle Forrest & 89 \\ \hline
4 & Deciduous Broad Forrest & 324 \\ \hline
5 & Mixed Forest & 602 \\ \hline
6 & Closed Shrubs & 329 \\ \hline
7 & Open Shrubs & 1484 \\ \hline
8 & Woody Savannas & 768 \\ \hline
9 & Savannas & 656 \\ \hline
10 & Grassland & 886 \\ \hline
11 & Wetlands & 147 \\ \hline
12 & Crops &1294 \\ \hline
13 & Urban & 44 \\ \hline
14 & Crop/Mosaic & 1436 \\ \hline
15 & Snow/Ice & 1443 \\ \hline
16 & Barren/Desert & 1379 \\ \hline
17 & Water & 12150 \\ \hline
18 & Tundra & 413 \\ \hline
\end{tabular}
\end{center}
\end{table}

\iffalse
\begin{table}[h!]
\small
\begin{center}
\caption{The distribution of the surface types.\added[id=pma]{\emph{(what is the `pixels sel.' column?)}}} \label{tab:IAVISA}
\begin{tabular}{|c|c|c|c|}
\hline
ID & Surface Type & Pixels Sel. & Samples\\ \hline \hline
1 & Evergreen Needle Forrest & 33170 & 603 \\ \hline
2 & Evergreen Broad Forrest & 35114 & 876 \\ \hline
3 & Deciduous Needle Forrest & 10500 & 89 \\ \hline
4 & Deciduous Broad Forrest & 11366 & 324 \\ \hline
5 & Mixed Forest & 35405 & 602 \\ \hline
6 & Closed Shrubs & 10817 & 329 \\ \hline
7 & Open Shrubs & 78789 & 1484 \\ \hline
8 & Woody Savannas & 46660 & 768 \\ \hline
9 & Savannas & 28638 & 656 \\ \hline
10 & Grassland & 42696 & 886 \\ \hline
11 & Wetlands & 5789 & 147 \\ \hline
12 & Crops & 58910 & 1294 \\ \hline
13 & Urban & 275 & 44 \\ \hline
14 & Crop/Mosaic & 60314 & 1436 \\ \hline
15 & Snow/Ice & 272144 & 1443 \\ \hline
16 & Barren/Desert & 57968 & 1379 \\ \hline
17 & Water & 1517554 & 12150 \\ \hline
18 & Tundra & 26691 & 413 \\ \hline
\end{tabular}
\end{center}
\end{table}
\fi

\begin{figure}[t!]
\centerline{
\includegraphics[height=2.6cm]{./images/IAVISA/final-global-sampling-map.png}~
\includegraphics[height=2.6cm]{./images/IAVISA/cloudy.png}}
\vspace{-0.25cm}
\caption{Global sample coverage for all seasons, times of day, and cloud cases (left), and examples of cloud-free and cloudy samples (right).}
\label{fig:sampling}
\end{figure}

\subsection{Cloud detection in the landmarks dataset}

In this large scale problem, the number of training examples is selected as $n\in\{10000,50000,100000,300000\}$ for RFF-GPC and VFF-GPC, and $n\in\{5000,10000,15000\}$ for GPC.
Notice that the improvement at training computational cost ($\mathcal{O}(nD^2)$ for the proposed methods against $\mathcal{O}(n^3)$ for GPC) enables us to consider much greater training datasets for our methods.
In fact, as we will see in Figure \ref{fig:LANDMARKS_main}, even with $n=300000$ RFF-GPC and VFF-GPC are computationally cheaper than GPC with $n=15000$.
Indeed, higher values of $n$ are not considered for GPC to avoid exceeding the (already expensive) $10^6$ seconds of training CPU time needed with just $n=15000$. Regarding the number $D$ of Fourier frequencies, we use $D\in\{10,25,50,100,150,200\}$.

The experimental results, which include predictive performance (test overall accuracy), training CPU time, and test CPU time, are shown in Figure \ref{fig:LANDMARKS_main}.
Every single value is the average of five independent runs under the same setting.
Namely, for each illumination condition, a test dataset is fixed and five different balanced training datasets are defined with the remaining data.
Notice that a general first observation across Figure \ref{fig:LANDMARKS_main}
suggests that higher accuracies are obtained for higher illumination conditions (SZA).

\begin{figure*}[t!]
\begin{center}
\small
\setlength{\tabcolsep}{0.5pt}
\begin{tabular}{cccc}
\includegraphics[width=.25\textwidth]{images/HIGH/testOA.pdf}  &
\includegraphics[width=.25\textwidth]{images/HIGH/trainTime.pdf}  &
\includegraphics[width=.25\textwidth]{images/HIGH/OAvsTIME.pdf} &
\includegraphics[width=.25\textwidth]{images/HIGH/testTime.pdf}
\\
& & & \\
\includegraphics[width=.25\textwidth]{images/MID/testOA.pdf}  &
\includegraphics[width=.25\textwidth]{images/MID/trainTime.pdf}  &
\includegraphics[width=.25\textwidth]{images/MID/OAvsTIME.pdf} &
\includegraphics[width=.25\textwidth]{images/MID/testTime.pdf}
\\
& & & \\
\includegraphics[width=.25\textwidth]{images/LOW/testOA.pdf}  &
\includegraphics[width=.25\textwidth]{images/LOW/trainTime.pdf}  &
\includegraphics[width=.25\textwidth]{images/LOW/OAvsTIME.pdf} &
\includegraphics[width=.25\textwidth]{images/LOW/testTime.pdf}
\\
& & & \\
\includegraphics[width=.25\textwidth]{images/NIGHT/testOA.pdf}  &
\includegraphics[width=.25\textwidth]{images/NIGHT/trainTime.pdf}  &
\includegraphics[width=.25\textwidth]{images/NIGHT/OAvsTIME.pdf}  &
\includegraphics[width=.25\textwidth]{images/NIGHT/testTime.pdf}
\end{tabular}
\vspace{-0.25cm}
\caption{Experimental results for LANDMARKS dataset. From top to bottom, the rows correspond with the described \emph{high}, \emph{mid}, \emph{low}, and \emph{night} illumination conditions. For each row, the first column shows the test overall accuracy (OA) of RFF-GPC, VFF-GPC, and GPC for the different values of $n$ (number of training examples) and $D$ (number of Fourier frequencies) considered.
The second column is analogous, but displays the CPU time (in seconds) needed to train each method (instead of the test OA).
The third column summarizes the two previous ones, providing a trade-off between test OA and training CPU time.
The last column is analogous to the first and second ones, but showing the CPU time used at the test step (production time).
The legend for the second and fourth columns is the same as in the first one. However, notice that in the third column plots the GPC lines degenerate into single points (since GPC does not depend on $D$). In both legends, the numbers indicate the amount $n$ of training examples used, which determines the width/size of the lines/points too.
As further explained in the main text, shown results are the mean over five independent runs.}
\label{fig:LANDMARKS_main}
\end{center}
\end{figure*}

\iffalse
\begin{figure*}[t!]
\begin{center}
\small
\setlength{\tabcolsep}{7pt}
\begin{tabular}{cc}
 \includegraphics[width=.4\textwidth]{images/HIGH/testOA.eps}  &
\includegraphics[width=.4\textwidth]{images/HIGH/trainTime.eps}  \\
\includegraphics[width=.4\textwidth]{images/HIGH/OAvsTIME.eps} &
\includegraphics[width=.4\textwidth]{images/HIGH/testTime.eps}
\end{tabular}
\caption{
\textcolor{blue}{
Experimental results for LANDMARKS dataset, \emph{high} illumination condition. From left to right and top to bottom, the first plot shows the test overall accuracy (OA) of RFF-GPC, VFF-GPC, and GPC for the different values of $n$ (number of training examples) and $D$ (number of Fourier frequencies) considered.
The second one is analogous, but displays the CPU time needed to train each method (instead of the test OA).
The third plot summarizes the two previous ones, providing a trade-off between test OA and training CPU time.
The last plot is analogous to the first and second ones, but showing the CPU time used at the test step (production time).
The legend for second and fourth plots is the same as the one in the first plot. However, notice that in the third plot the GPC lines degenerate into single points (since GPC does not depend on $D$). In both legends, the numbers indicate the amount $n$ of training examples used, which determines the width/size of the lines/points too. As further explained in the main text, shown results are the mean over five independent runs.
}
}
\label{fig:LANDMARKS_main}
\end{center}
\end{figure*}
\fi

Figure \ref{fig:LANDMARKS_main} reveals an overwhelming superiority of RFF-GPC and VFF-GPC over standard GPC: our proposed methods achieve a higher predictive performance while investing substantially lower training CPU time. This is very clear from the third column plots, where for any blue point we can find orange and yellow points which are placed more north-west (i.e. higher test OA and lower CPU training time). Furthermore, the fourth column shows an equally extraordinary reduction in test CPU time (production time), where the proposed methods are more than $100$ times faster than GPC. In particular, this makes RFF-GPC and VFF-GPC better suited than standard GPC for real-time EO applications.

Regarding the practical differences between RFF-GPC and VFF-GPC, we observe that RFF-GPC is faster (at training) whereas VFF-GPC is more accurate. This is a natural consequence of their theoretical formulations: the estimation of the Fourier frequencies $\bW$ in VFF-GPC makes it more flexible and expressive, but involves a heavier training.
Therefore, in this particular problem, the final practical choice between the two proposed methods would depend on the relative importance that the user assigns to test accuracy (where VFF-GPC stands out) and training cost (where RFF-GPC does so).
In terms of test cost, both methods are very similar, as expected from the identical $\mathcal{O}(D^2)$ theoretical test complexity. The independence of this quantity on $n$ is also intuitively reflected in the experiments, with all the RFF-GPC and VFF-GPC lines collapsing onto a single one in the fourth column plots of Figure \ref{fig:LANDMARKS_main}.

At this point, it is worth to analyze a bit further the role of $D$ in the performance of our methods. Recall (Section \ref{sec:theory_RFF}) that RFF-GPC is an approximation to GPC, with an error that exponentially decreases with the ratio $D/d$ between the dimensions of the projected Fourier features space and the original one. Therefore, it is theoretically expected that the performance of RFF-GPC increases with $D$, becoming equivalent to GPC when $D\rightarrow\infty$. Actually, this is supported by the first column of Figure \ref{fig:LANDMARKS_main}. Moreover, since our problem here presents a low $d$ ($16$ for \emph{high}, \emph{mid}, and \emph{low}, and $6$ for \emph{night}), it is natural that RFF-GPC with just $D=200$ already gets very similar (even better in some cases) results to standard GPC with the same $n$ (yet far much faster, compare GPC and RFF/VFF for $n=10000$).
In the case of VFF-GPC, where the Fourier frequencies are model parameters to be estimated, the number $D$ is directly related to the complexity of the model. Therefore, its increase should not always mean a higher performance in test OA, since large values may provoke over-fitting to the training dataset (this will be clear in the next dataset, whereas it does not occur in LANDMARKS). Furthermore, unlike RFF-GPC, the performance of VFF-GPC is not directly affected by $d$.

It is also reasonable to expect that both test OA and training CPU time increase with the training dataset size $n$. More specifically, and from a practical perspective in which the computational resources are finite, the first column in Figure \ref{fig:LANDMARKS_main} shows that test OA becomes stalled when only one of $n$ or $D$ increases. However, greater improvements in test OA are achieved when $n$ and $D$ are jointly increased. Notice that this is also justifiable from a theoretical viewpoint: the higher the dimensionality of the projected Fourier features space (which is $2D$), the larger number $n$ of examples are required to identify the separation between classes.

\if false
\subsection{Computational cost}
\begin{figure}[h!]
\centerline{\includegraphics[width=8.4cm]{./images/CPU_time.pdf}}
\vspace{-0.25cm}
\caption{Landmarks are essential in image registration and geometric quality assessment. Any misclassification of a landmark due to cloud contamination degrades the correlation matching which is a cornerstone for the image navigation and registration (INR) algorithms.}
\label{fig:motivation}
\end{figure}
\fi

\subsection{Cloud detection with the IAVISA dataset}

As explained in Section \ref{sec:data_IAVISA}, this problem involves a total amount of $n=24923$ instances.
We performed five-fold cross-validation, which produces five pairs of training/test datasets with (approximately) $20000/5000$ instances each. Results are then averaged.
Since RFF-GPC and VFF-GPC are conceived for large-scale applications (they scale linearly with $n$, recall their $\mathcal{O}(nD^2)$ training cost), they will not be utilized with values of $n$ lower than this training dataset size of $\approx 20000$ (even GPC is able to cope with this size).
Indeed, in the case of GPC we use the values $n\in\{1000,5000,10000,15000, \mathrm{ALL}\approx20000\}$.
Regarding the number of Fourier frequencies $D$, we consider the grid $D\in\{1:10, 15:5:25, 50:25:150\}$.
The experimental results, which include the same metrics as those used for the previous problem on landmarks, are shown in Figure \ref{fig:IAVISA}.

\begin{figure*}[t!]
\begin{center}
\small
\setlength{\tabcolsep}{7pt}
\begin{tabular}{cc}
\includegraphics[width=.35\textwidth]{images/IAVISA/testOA.pdf}
&
\includegraphics[width=.35\textwidth]{images/IAVISA/trainTime.pdf}
\\
\includegraphics[width=.35\textwidth]{images/IAVISA/OAvsTIME.pdf}
&
\includegraphics[width=.35\textwidth]{images/IAVISA/testTime.pdf}
\end{tabular}
\caption{
Experimental results for the IAVISA dataset. From left to right and top to bottom, the first plot shows the test overall accuracy (OA) of RFF-GPC, VFF-GPC, and GPC for the different values of $n$ (number of training examples) and $D$ (number of Fourier frequencies) considered.
The second column is analogous, but displays the CPU time needed to train each method (instead of the test OA).
The third column summarizes the two previous ones, providing a trade-off between test OA and training CPU time.
The last column is analogous to the first and second ones, but showing the CPU time used at the test step.
The legend for second and fourth plots is the same as the one in the first plot. However, in the third plot the GPC lines degenerate into single points (since GPC does not depend on $D$). In both legends, the numbers indicate the amount $n$ of training examples used, which determines the width/size of the lines/points too (\textrm{ALL} means the whole training dataset, i.e. $n\approx 20000$). As explained in the main text, the results are the mean over five independent runs.
}
\label{fig:IAVISA}
\end{center}
\end{figure*}

In this case, we again observe a clear outperformance of VFF-GPC against GPC: it achieves higher test OA while requiring less training and test CPU times. Moreover, the improvement in test OA is greater than $3\%$, and train/test CPU times are around $100$ and $1000$ times lower respectively.
However, unlike in the previous problem of cloud detection over landmarks, RFF-GPC does not exhibit such a clear superiority over GPC in this application. Whereas it does drastically decrease the train/test CPU times, it is not able to reach the test OA of GPC with $n=10000$.
Therefore, in practice, the optimal choice for this application is VFF-GPC. RFF-GPC would only be recommended if the training CPU time is a very strong limitation.

The main reason why RFF-GPC is not completely competitive in this problem is its theoretical scope: as an efficient approximation to GPC, it is conceived for large scale applications which are out of the reach of standard GPC. If the size of the problem allows for using GPC (as in this case), then RFF-GPC will only provide a more efficient alternative (less training and test CPU times), but its predictive performance will be always below that of GPC. Moreover, the difference in this performance is directly influenced by the original dimension $d$ of data (recall that the kernel approximation behind RFF-GPC exponentially degrades with $d$, Section \ref{sec:theory_RFF}). This is precisely a second hurdle that RFF-GPC finds in IAVISA: the high $d=100$ makes RFF-GPC with the full dataset be quite far from the corresponding GPC at predictive performance (test OA). In conclusion, the ideal setting for RFF-GPC is a large scale problem (high $n$) with few features (low $d$), precisely the opposite to the IAVISA dataset.

Interestingly, VFF-GPC bypasses these limitations of RFF-GPC by {\em learning a new kernel and not just approximating} the SE one. First, VFF-GPC is not just a GP adaptation well-suited for large scale applications, but a general-purpose, expressive, and very competitive kernel-based classifier that scales well with the number of training instances. Second, as it does not rely on the kernel approximation, VFF-GPC is not affected by the original dimension $d$ of data. Both ideas are empirically supported by the results obtained in IAVISA.

The first plot of Figure \ref{fig:IAVISA} shows that the predictive performance of VFF-GPC does not necessarily improves by increasing $D$.
This is the expected behavior from the theoretical formulation of VFF-GPC, where the Fourier frequencies are $D$ parameters to be estimated.
Thus, a higher amount of them confers VFF-GPC a greater flexibility to learn hidden patterns in the training dataset, but also the possibility to over-fit very particular structures of it which do not generalize to the test set.
This is the classical problem of the \emph{model complexity} in machine learning, and it is further illustrated in Figure \ref{fig:IAVISA2}. Together with the first plot in Figure \ref{fig:IAVISA}, it shows the paradigmatic behaviour of train and test performance in presence of over-fitting: train OA grows with the model complexity (great flexibility allows for learning very particular structures of the training set, even reaching a $100\%$ of train OA), whereas test OA initially grows (the first patterns are part of the ground truth and thus general to the test set) but then goes down (when the learned information is too specific to the training set).
Notice that this over-fitting phenomena did not occur at LANDMARKS, where test OA monotonically increased with $D$. In addition to the different nature of the problems, the training dataset size $n$ plays a crucial role at this: smaller datasets (like IAVISA) are more prone to over-fitting than larger ones (LANDMARKS) under the same model complexity.

Finally, it is worth noting that VFF-GPC achieves its maximum test OA when using just $D=5$ Fourier frequencies. This reflects (i) a not very sophisticated internal structure of the IAVISA dataset (since just $5$ directions are enough to correctly classify $85\%$ of the data), and (ii) the VFF-GPC capability to learn those discriminative directions from data.
In particular, this shows that VFF-GPC can be used not only as a classifier, but also as a method that learns the most relevant discriminative directions in a dataset.
Unfortunately, RFF-GPC is not able to benefit from these privileged directions that may exist in some datasets, since it randomly samples and fix the Fourier frequencies from the beginning.

\begin{figure}[htp!]
\begin{center}
\small
\includegraphics[width=.35\textwidth]{images/IAVISA/trainOA.pdf}
\vspace{-0.25cm}
\caption{Train OA in the IAVISA dataset for RFF-GPC, VFF-GPC, and GPC with different values of $n$ (number of training examples) and $D$ (number of Fourier frequencies). These results complement the first plot in Figure \ref{fig:IAVISA}, showing that high values of $D$ make VFF-GPC over-fit to the training dataset. The legend and its interpretation are the same as there.}
\label{fig:IAVISA2}
\end{center}
\end{figure}

\subsection{Explicit classification maps for cloud detection}

The last two sections were dedicated to thoroughly analyze the performance of the proposed methods, empirically understand their behavior, weaknesses, and strengths, and compare them against GPC. In order to illustrate the explicit cloud detection behind the experiments, here we provide several explanatory classification maps obtained by the best model (in terms of predictive performance) for the LANDMARKS dataset: VFF-GPC with $n=300000$ and $D=200$.

The classification maps are obtained for the whole year 2010 at the Dakhla landmark,
with a total of 34940 satellite acquisitions. The acquired window size is $26\times20$ pixels. Relying on the proposed feature extraction procedure, we trained the four necessary models (high, mid, low, night), and then proceed to predict over the whole available amount of chips acquired in the 2010 year.\footnote{A full video with all the classification maps is available at \url{http://decsai.ugr.es/vip/software.html} and \url{http://isp.uv.es/code/vff.html}. RFF-GPC and VFF-GPC codes are also provided.}

In Figure~\ref{fig:classification-maps}, several chips are provided with the aim of illustrating different behaviors.
In the first situation (first row), we can see a characteristic error of the L2 cloud mask, which sometimes tends to wrongly label the coastline pixels as cloudy. However, VFF-GPC leads to a better classification, identifying just one cloudy pixel and thus avoiding this negative coastline effect\footnote{As a clarification note, the coastline pixels were removed from the training dataset by applying a carefully designed morphological filter around coastlines.}.
In the second row, the visual channels show large clouds crossing the landmark. In the bottom-right of the image, a long cloud is unlabeled in the L2 cloud mask but correctly detected by VFF-GPC. While being formally accounted as an error, such discrepancy is actually positive for our method. Moreover, VFF-GPC shows an interesting cloud-sensitive behaviour at the top-left cloudy mass, identifying a larger cloudy area than that provided by EUMETSAT. This is a desirable propensity in cloud detection applications, where we prefer to identify larger clouds (and then thoroughly analyze them) rather than missing some of them.
In the third row, the RGB channel allows for visually identifying three main cloudy masses at the landmark.
The L2 mask poorly labels the central cloudy band, and does not detect the lower cloud. Both deficiencies are overcome by VFF-GPC.
Finally, the fourth chip shows a huge cloudy mass that is undetected by the L2 mask but is correctly identified by VFF-GPC.

Therefore, although VFF-GPC is trained with an imperfect ground truth, we observe that it is able to bypass some of these deficiencies, and exhibits a desirable cloud-sensitive behavior. This improvement can be also related to the particular design of the training datasets, splitting the problem into four different cases depending on the illumination conditions.

\iffalse
\begin{figure}[ht!]
\begin{center}
\includegraphics[height=3cm]{images/MAPS/f2_115_NIGHT.pdf}
\vspace{-0.25cm}
\caption{Ad Dakhla landmark, situated at the western Sahara. On the left, the land cover binary map, where water appears in blue and land in red. On the right, a more detailed image of the location.}
\label{fig:AdDakhla}
\end{center}
\end{figure}
\fi

\begin{figure}[ht!]
\begin{center}
\includegraphics[width=.24\columnwidth]{images/MAPS/f1_115_NIGHT.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS/f3_115_NIGHT.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS/f4_115_NIGHT.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS/f5_115_NIGHT.pdf}
\\
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/322_01_LOW.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/322_03_LOW.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/322_04_LOW.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/322_05_LOW.pdf}
\\
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/324_01_MID.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/324_03_MID.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/324_04_MID.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/324_05_MID.pdf}
\\
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/994_01_LOW.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/994_03_LOW.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/994_04_LOW.pdf}
\includegraphics[width=.24\columnwidth]{images/MAPS-BIG/994_05_LOW.pdf}

\vspace{-0.25cm}
\caption{Explicit classification maps for the Dakhla landmark. The rows correspond with four different acquisitions.
The first column shows the visible RGB channels (which are not informative for \emph{night} acquisitions such us the first one), the second column is the infrared 10.8$\mu m$ spectral band (very illustrative in night scenarios), the third column represents the ground truth obtained by EUMETSAT (the L2 cloud mask), and the last one is the VFF-GPC classification map. In the last two columns, the red color is used for cloudy pixels and blue for cloud-free ones.}
\label{fig:classification-maps}
\end{center}
\end{figure}

\section{Introduction}\label{sec:intro}
\input{intro.tex}

\section{Large Scale Gaussian Process Classification}\label{sec:theory}
\input{theory.tex}

\section{Data Collection and Preprocessing}\label{sec:data}
\input{data.tex}

\section{Experiments}\label{sec:experiments}
\input{experiments.tex}

\section{Conclusions and Future Work}\label{sec:conclusions}
\input{conclusions.tex}

\iffalse
\begin{figure*}[t!]
\begin{center}
\small
\setlength{\tabcolsep}{0.5pt}
\begin{tabular}{cccc}
 \includegraphics[width=.25\textwidth]{images/MID/testOA.eps}  &
\includegraphics[width=.25\textwidth]{images/MID/trainTime.eps}  &
\includegraphics[width=.25\textwidth]{images/MID/OAvsTIME.eps} &
\includegraphics[width=.25\textwidth]{images/MID/testTime.eps}
\\
& & & \\
 \includegraphics[width=.25\textwidth]{images/LOW/testOA.eps}  &
\includegraphics[width=.25\textwidth]{images/LOW/trainTime.eps}  &
\includegraphics[width=.25\textwidth]{images/LOW/OAvsTIME.eps} &
\includegraphics[width=.25\textwidth]{images/LOW/testTime.eps}
\\
& & & \\
 \includegraphics[width=.25\textwidth]{images/NIGHT/testOA.eps}  &
\includegraphics[width=.25\textwidth]{images/NIGHT/trainTime.eps}  &
\includegraphics[width=.25\textwidth]{images/NIGHT/OAvsTIME.eps}  &
\includegraphics[width=.25\textwidth]{images/NIGHT/testTime.eps}
\end{tabular}
\vspace{-0.25cm}
\caption{
\textcolor{blue}{
Additional experimental results for LANDMARKS dataset: \emph{mid} (first row), \emph{low} (second row), and \emph{night} (third row) illumination conditions.
Each row contains the same four plots as in the main Figure \ref{fig:LANDMARKS_main}. Namely, the first column shows the test overall accuracy (OA), the second one the CPU time needed to train each method, the third column shows the trade-off between test OA and training CPU time, and the last one provides the CPU time needed at the test step (production time).
The legend, which is common to all plots and is included only in the top left one, is also the same as in Figure \ref{fig:LANDMARKS_main}.
}
}
\label{fig:LANDMARKS_suppl}
\end{center}
\end{figure*}
\fi

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

\subsection{Random Fourier Features}\label{sec:theory_RFF}

The main issue with large scale applications of GP is its $\mathcal{O}(n^3)$ cost at the training phase, which comes from the $n\times n$ kernel matrix inversion.
The work \cite{rahimiRFF} presents a general methodology (based on Bochner's theorem \cite{rudin2011}) to approximate any positive-definite shift-invariant kernel $k$ by a linear one. This is achieved by explicitly projecting the original $d$-dimensional data $\bx$ onto $\mathcal{O}(D)$ random Fourier features $\bz(\bx)$, whose linear kernel $k_L$ approximates $k$.
This linearity will enable us to work in the primal space of features and substitute $n\times n$ matrix inversions by $\mathcal{O}(D)\times\mathcal{O}(D)$ ones, resulting in a total $\mathcal{O}(nD^2+D^3)$ computational cost.
In large-scale applications, one can set a $D\ll n$, and thus the obtained $\mathcal{O}(nD^2)$ complexity represents an important benefit over the original $\mathcal{O}(n^3)$. Moreover, the complexity at test is also reduced from $\mathcal{O}(n^2)$ to $\mathcal{O}(D^2)$, even becoming independent on $n$.

In this work we use the well-known SE (or Gaussian) kernel $k(\bx,\bx') = \gamma\cdot\exp(-||\bx-\bx'||^2/(2\sigma^2))$. Following the methodology in \cite{rahimiRFF}, this kernel can be linearly approximated as
\begin{align}
k(\bx,\bx')\approx k_L(\bx,\bx') = \gamma\cdot\bz(\bx)^\intercal\bz(\bx'),\label{eq:kerapprox}
\end{align}
where
\begin{multline}\label{feats_general}
\bz(\bx)^\intercal = D^{-1/2} \cdot\left(\cos\left(\bw_1^\intercal \bx\right),\sin\left(\bw_1^\intercal \bx\right),\dots \right.\\
\left. \dots,\cos\left(\bw_D^\intercal \bx\right),\sin\left(\bw_D^\intercal \bx\right) \right)\in\mathbb{R}^{2D},
\end{multline}
and the \emph{Fourier frequencies} $\bw_i$ must be sampled from a normal distribution $\mathcal{N}(\mathbf{0},\sigma^{-2}\bI)$. As explained in \cite[Claim 1]{rahimiRFF}, this approximation exponentially improves with the number $D$ of Fourier frequencies used (and also exponentially worsens with $d$, the original dimension of $\bx$). However, obviously, increasing $D$ in our methods will go at the cost of increasing the $\mathcal{O}(nD^2)$ and $\mathcal{O}(D^2)$ complexities.
Other kernels different from the SE one could be used, but that would imply sampling from a different distribution.

Our novel RFF-GPC method considers a standard Bayesian linear model over these new features $\bz$. Such a linear model corresponds to GP classification with the linear kernel $k_L$ \cite[Chapter 6]{BishopBook}. Since $k_L$ approximates the SE kernel $k$, our RFF-GPC constitutes an approximation to GP classification with SE kernel. However, RFF-GPC is well suited for large scale applications, as it works in the primal space of $\mathcal{O}(D)$ features and thus presents a $\mathcal{O}(nD^2)$ (resp. $\mathcal{O}(D^2)$) train (resp. test) complexity.

Notice that RFF-GPC needs to sample the Fourier frequencies $\bw_i$ from $\mathcal{N}(\mathbf{0},\sigma^{-2}\bI)$ from the beginning, whereas hyperparameters $\sigma$ and $\gamma$ must be estimated during the learning process (just as in standard GP classification). In order to uncouple $\bw_i$ and $\sigma$, in the sequel we consider the equivalent features
\begin{multline}\label{feats_RFF}
\bz(\bx|\sigma,\bW)^\intercal = D^{-1/2} \cdot\left(\cos\left(\sigma^{-1}\bw_1^\intercal \bx\right),\sin\left(\sigma^{-1}\bw_1^\intercal \bx\right),\dots \right.\\
\left. \dots,\cos\left(\sigma^{-1}\bw_D^\intercal \bx\right),\sin\left(\sigma^{-1}\bw_D^\intercal \bx\right) \right),
\end{multline}
with $\bw_i$ now sampled from $\mathcal{N}(\mathbf{0},\bI)$ and fixed. Notice that we have collectively denoted $\bW=[\bw_1,\dots,\bw_D]^\intercal\in\mathbb{R}^{D\times d}$.

At this point, it is natural to consider other possibilities for the Fourier frequencies $\bW$, rather than just randomly sample and fix them from the beginning. The proposed VFF-GPC model treats them as hyperparameters to be estimated (so as to maximize the likelihood of the observed data), just like $\sigma$ and $\gamma$ in the case of RFF-GPC. This makes VFF-GPC more expressive, flexible, and tailored to the data, although it may no longer constitute an approximation to the SE kernel (for which the $\bw_i$ must be normally distributed). More specifically, VFF-GPC does start with an approximated SE kernel (since the $\bw_i$ are initialized with a normal distribution), but the maximum a posteriori (MAP) optimization on $\bw_i$ makes it \emph{learn} a new kernel which may no longer approximate a SE one.
Therefore, for VFF-GPC we also use $\bz(\bx|\sigma,\bW)$ as in eq.~\eqref{feats_RFF}, with now both $\sigma$ and $\bW$ to be estimated.
Interestingly, VFF-GPC extends the Sparse Spectrum Gaussian Process model originally introduced for regression in \cite{SSGPR}, to GP classification.

More specifically, the authors there also sparsify the SE kernel by working on the primal space of $\cos$ and $\sin$ Fourier features, see \cite[Equation 5]{SSGPR}. However, our classification setting involves a sigmoid-based (logistic) observation model for the output given the latent variable (see the next eq.~\eqref{obsModel}), whereas in regression this is just given by a normal distribution. Therefore, VFF-GPC needs to additionally deal with the non-conjugacy of the sigmoid, which motivates the variational bound of eq.~\eqref{varbound} and the consequent variational inference procedure described in Section \ref{sec:varInf}. It is interesting to realize that VFF-GPC is introduced here as a natural extension of RFF-GPC, whereas there is not a regression analogous for RFF-GPC in \cite{SSGPR}.

Another possibility for the Fourier frequencies would be to estimate them (just as in VFF-GPC) but considering alternative prior distributions $\p(\bW)$ (which means utilizing alternative kernels).
Moreover, instead of maximum a posteriori inference, we could address the marginalization of the Fourier frequencies $\bW$.
Alternatively, to promote sparsity, the use of Gaussian Scale Models (GSM) \cite{babacan:2012} could also be investigated.
These possibilities will be explored in future work, and here we will concentrate on RFF-GPC and VFF-GPC.

\subsection{Models formulation}

As anticipated in previous section, RFF-GPC and VFF-GPC are standard Bayesian linear models working on the explicitly mapped features $\bz(\bx|\sigma,\bW)$ of eq.~\eqref{feats_RFF}. In the case of RFF-GPC, $\bW$ is sampled from $\mathcal{N}(\mathbf{0},\bI)$ at the beginning and fixed, with $\sigma$ to be estimated. In the case of VFF-GPC, both $\bW$ and $\sigma$ are estimated, with a $\mathcal{N}(\mathbf{0},\bI)$ prior over $\bW$. In order to derive both methods in a unified way, $\Phi$ will denote $\sigma$ for RFF-GPC and both $(\bW,\sigma)$ for VFF-GPC.

Since we are dealing with binary classification, we consider the standard logistic observation model
\begin{equation}\label{obsModel}
    \p(y = 1 | \bbeta,\Phi,\bx) = \psi(\bbeta^\intercal\bz) =(1+\exp(-\bbeta^\intercal \bz))^{-1},
\end{equation}
where $\bz=\bz(\bx|\Phi)$. For the weights $\bbeta\in\mathbb{R}^{2D}$ we utilize the prior normal distribution $\p(\bbeta|\gamma) = \mathcal{N}(\bbeta|\mathbf{0},\gamma\bI)$, with $\gamma$ to be estimated, see eq.~(\ref{eq:kerapprox}).

For an observed dataset $\mathcal{D}=\{(\bx_i,y_i)\}_{i=1}^n\subset\mathbb{R}^d\times\{0,1\}$, the joint p.d.f. reads
\begin{align}
\p(\by,\bbeta|\Phi,\gamma,\bX)&=\p(\by|\bbeta,\Phi,\bX)\p(\bbeta|\gamma) \nonumber \\
&= \left(\prod_{i=1}^n \p(y_i| \bbeta,\Phi,\bx_i)\right) \p(\bbeta|\gamma),\label{eq:jpdf}
\end{align}
where we collectively denote $\by = (y_1,\dots,y_n)^\intercal$ and $\bX = [\bx_1,\dots,\bx_n]^\intercal$. For the sake of brevity, from now on we will systematically omit the conditioning on $\bX$.

\subsection{Variational inference}\label{sec:varInf}

Given the observed dataset $\mathcal{D}$, in this section we seek point estimates of $\gamma$ and $\Phi$ by maximizing the marginal likelihood $\p(\by|\Phi,\gamma)$ (in VFF-GPC, the additional prior $\p(\bW)=\mathcal{N}(\bW|\mathbf{0},\bI)$ yields maximum a posteriori inference, instead of maximum likelihood one, for $\bW$).
After that, we obtain (an approximation to) the posterior distribution $\p(\bbeta|\by,\Phi,\gamma)$. Due to the non-conjugate observation model, the required integrals will be mathematically intractable, and we will resort to the variational inference approximation~\cite[Section 10.6]{BishopBook}.

First, notice that integrating out $\bbeta$ in eq.~\eqref{eq:jpdf} is not analytically possible due to the sigmoid functions in the observation model $\p(\by|\bbeta,\Phi)$. To overcome this problem, we use the variational bound
\begin{equation}\label{varbound}
\log\left(1+e^{x}\right) \leq \lambda(\xi)(x^2-\xi^2)+\frac{x-\xi}{2}+\log\left(1+e^{\xi}\right),
\end{equation}
which is true for any real numbers $x,\xi$ and where $\lambda(\xi) = (1/2\xi)\left( \psi(\xi)-1/2 \right)$ \cite[Section 10.6]{BishopBook}. Applying it to every factor of $\p(\by|\bbeta,\Phi)$, we have the following lower bound
\begin{equation}\label{varboundHere}
\p(\by|\bbeta,\Phi) \geq \exp\left(-\bbeta^\intercal \bZ^\intercal \bLambda\bZ \bbeta + \bv^\intercal\bZ\bbeta
\right)\cdot C(\bxi).
\end{equation}
Here we write $\bZ=[\bz_1,\dots,\bz_n]^\intercal\in\mathbb{R}^{n\times 2D}$ for the projected-data matrix (which depends on $\Phi$), $\bLambda$ is the diagonal matrix $\diag(\lambda(\xi),\dots,\lambda(\xi_n))$, $\bv=\by-(1/2)\cdot\mathbf{1}_{n\times 1}$, and the term $C(\bxi) = \prod_i\exp\left(\lambda(\xi_i)\xi_i^2+(1/2)\xi_i-\log(1+\exp(\xi_i))\right)$ only depends on $\bxi$. The key is that this lower bound for $\p(\by|\bbeta,\Phi)$ is conjugate with the normal prior $\p(\bbeta|\gamma)$ (since it is the exponential of a quadratic function on $\bbeta$), and thus it allows us integrating out $\bbeta$ in eq~\eqref{eq:jpdf}. In exchange, we have introduced $n$ additional hyperparameters $\bxi=(\xi_1,\dots,\xi_n)^\intercal$ that will need to be estimated along with $\Phi$ and $\gamma$.

Therefore, substituting $\p(\by|\bbeta,\Phi)$ for its bound, eq.~\eqref{eq:jpdf} can be lower bounded as
\begin{gather}\label{varGlobal}
\p(\by,\bbeta|\Phi,\gamma) \geq F(\by,\Phi,\gamma,\bxi)  \cdot\mathcal{N}(\bbeta|\bmu,\bSigma),
\end{gather}
where we have denoted
\begin{gather}
\nonumber F(\by,\Phi,\gamma,\bxi) = C(\bxi)\left\vert \gamma^{-1}\bSigma \right\vert^{1/2}\exp\left(\frac{1}{2}\bmu^\intercal\bSigma^{-1}\bmu\right), \\
\bSigma = \left(\bZ^\intercal(2\bLambda)\bZ+\gamma^{-1}\bI \right)^{-1}, \quad \bmu = \bSigma\bZ^\intercal\bv.\label{posterior}
\end{gather}

Now it is clear that we can marginalize out $\bbeta$, and iteratively estimate the optimal values of $\Phi$, $\gamma$ and $\bxi$ as those that maximize $F(\by,\Phi,\gamma,\bxi)$ (or equivalently $\log F(\by,\Phi,\gamma,\bxi)$) for the observed $\by$.
Starting at $\bxi^{(1)}$, $\Phi^{(1)}$, and $\gamma^{(1)}$, we can calculate  $\bxi^{(k)}$, $\Phi^{(k)}$, and $\gamma^{(k)}$ for $k \ge 1$.
In the case of $\bxi$, we make use of the local maximum condition $\partial F/\partial\bxi = 0$.
From there, it is not difficult to prove that the optimal value satisfies \cite{BishopBook}
\begin{equation}\label{updateXi}
    \bxi^{(k+1)} = \sqrt{\diag\left(\bZ^{(k)}\bSigma^{(k)}\left(\bZ^{(k)}\right)^\intercal\right)+\left(\bZ^{(k)}\bmu^{(k)}\right)^2},
\end{equation}
where the square and square root of a vector are understood as element-wise.
In the case of $\Phi$ and $\gamma$, we use nonlinear conjugate gradient methods \cite{minimize} and obtain (notice that for VFF-GPC we can collapse $\bW$ and $\sigma$, removing the prior on $\bW$)
\begin{align}\label{eq:Phi_gamma}
&\left(\Phi^{(k+1)},\gamma^{(k+1)}\right) = \arg\max_{\Phi,\gamma}\left\{ -\log\left\vert 2\gamma\bZ^\intercal\bLambda^{(k+1)}\bZ+\bI \right\vert  \right. \nonumber \\ &\hspace*{1cm}+ \left. \bv^\intercal \bZ \left( 2\bZ^\intercal\bLambda^{(k+1)}\bZ+\gamma^{-1}\bI\right)^{-1} \bZ^\intercal \bv \right\}.
\end{align}

Once the hyperparameters $\Phi$, $\gamma$, and $\bxi$ have been estimated by $\hat\Phi$, $\hat\gamma$, and $\hat\bxi$ respectively, we need to compute the posterior $\p(\bbeta|\by,\hat\Phi,\hat\gamma)$. As before, this is mathematically intractable due to the sigmoids in the observation model. Therefore, we again resort to the variational bound in eq.~\eqref{varGlobal} to get an optimal approximation $\hat\p(\bbeta)$ to the posterior $\p(\bbeta|\by,\hat\Phi,\hat\gamma)$. Namely, we do it by minimizing (an upper bound of) the KL divergence between both distributions:
\begin{align*}
& \mathrm{KL}\left(\hat\p(\bbeta)||\p(\bbeta|\by,\hat\Phi,\hat\gamma)\right) = \int\hat\p(\bbeta)\log\frac{\hat\p(\bbeta)}{\p(\bbeta|\by,\hat\Phi,\hat\gamma)}\md \bbeta   \\
&= \log\p(\by|\hat\Phi,\hat\gamma)+ \int\hat\p(\bbeta)\log\frac{\hat\p(\bbeta)}{\p(\by,\bbeta|\hat\Phi,\hat\gamma)}\md \bbeta  \\
&\leq \log\p(\by|\hat\Phi,\hat\gamma) - \log F(\by,\hat\Phi,\hat\gamma,\hat\bxi) + \mathrm{KL}\left(\hat\p(\bbeta)||\mathcal{N}(\bbeta|\hat\bmu,\hat\bSigma)\right).
\end{align*}
Thus, the minimum is reached for $\hat\p(\bbeta) = \mathcal{N}(\bbeta|\hat\bmu,\hat\bSigma)$, with $\hat\bmu$ and $\hat\bSigma$ calculated in  eq.~\eqref{posterior} using $\hat\Phi$, $\hat\gamma$, and $\hat\bxi$.

In summary, at training time, our methods RFF-GPC and VFF-GPC run iteratively until convergence of the hyperparameters $\Phi$, $\gamma$, and $\bxi$ to their optimal values $\hat \Phi$, $\hat\gamma$, and $\hat\bxi$ (see Algorithm \ref{algorithm}). The computations involved there suppose a computational complexity of $\mathcal{O}(nD^2+D^3)$ (which equals $\mathcal{O}(nD^2)$ when $n\gg D$), whereas standard GPC scales as $\mathcal{O}(n^3)$.
At test time, the probability of class $1$ for a previously unseen instance $\bx_*\in\mathbb{R}^d$ is:
\begin{align}\label{eq:test}
\p(y_*=1) &\approx
\int \p(y = 1|\bbeta,\hat \Phi,\bx_*)\hat\p(\bbeta){\rm d}\bbeta \approx \nonumber \\
& \approx \psi\left(\hat\bz_*^\intercal \hat\bmu\cdot\left(1+(\pi/8)\hat\bz_*^\intercal\hat\bSigma \hat\bz_*\right)^{-1/2}
\right),
\end{align}
with $\psi$ being the sigmoid function. Whereas GPC presents a computational cost of $\mathcal{O}(n^2)$ for each test instance, eq.~\eqref{eq:test} implies a complexity of $\mathcal{O}(D^2)$ in the case of our methods. In particular, notice that this is independent on the number of training instances. These significant reductions in computational cost (both at training and test) make our proposal suitable for large scale and real-time applications in general, and in EO applications in particular.

\begin{algorithm}
\caption{Training RFF-GPC and VFF-GPC.}
\label{algorithm}
\begin{algorithmic}
\REQUIRE Data set $\mathcal{D}=\{(\bx_i,y_i)\}_{i=1}^n \subset \mathbb{R}^d\times\{0,1\}$ and the number $D$ of Fourier frequencies.
\STATE Only for RFF-GPC, sample the Fourier frequencies $\bw_i$ from $\mathcal{N}(\mathbf{0},\bI)$ and fix them.

\STATE Initialize $\bxi^{(1)} = \mathbf{1}_{n\times 1}$, $\gamma^{(1)}=1$, and $\Phi^{(1)}$. For RFF-GPC (where $\Phi=\sigma$), $\Phi^{(1)}$ is initialized as the mean distance between (a subset of) the training data points $\bx_i$. For VFF-GPC (where $\Phi=(\sigma,\bW)$), $\sigma$ is initialized as described for RFF-GPC and $\bW$ with a random sample from $\mathcal{N}(\mathbf{0},\mathbf{I})$.
\REPEAT \STATE{Update $\bxi^{(k+1)}$ with eq.~\eqref{updateXi}.}
\STATE{Update $\Phi^{(k+1)}$ and $\gamma^{(k+1)}$ with eq.~\eqref{eq:Phi_gamma}, using $\Phi^{(k)}$ and $\gamma^{(k)}$ as initial values for the conjugate gradient method.}
\UNTIL{convergence}
\RETURN Optimal hyperparameter $\hat \Phi$ and the posterior distribution $\hat\p(\bbeta)=\mathcal{N}(\bbeta|\hat{\bmu},\hat{\bSigma})$.
\end{algorithmic}
\end{algorithm}

Finally, regarding the convergence of the proposed methods, we cannot theoretically guarantee the convergence to a global optimum (only a local one), since we are using conjugate gradient methods to solve the non-convex optimization problem in eq. \eqref{eq:Phi_gamma}.
However, from a practical viewpoint, we have experimentally checked that both methods have a satisfactory similar convergence pattern. Namely, in the first iterations, the hyperparameters experiment more pronounced changes, widely exploring the hyperparameters space. Then, once they reach a local optimum vicinity, these variations become smaller. Eventually, the hyperparameters values hardly change and the stop criterion is satisfied.

\iffalse

{
\color{blue}
\subsection{Geometrical interpretation of RFF and VFF}

Our proposal not only as an approximation of standard GP for classification (specially relevant for VFF, where we do not necessarily take very large $D$).

Theoretical partitions of the space through $\sin$ and $\cos$. Role of the length-scale parameter. Can be supported with some experiment of low $D$.

Re-interpretation of the prior distribution over $W$ (which exists for VFF): not only as fulfilling Rahimi's results, but equally weigthing all projecting directions.

For the toy example, use dataset \emph{high} with features (1,3) or (2,9).

}

\fi


\section{Introduction}
Because of the existence of anomalous systems which do not seem to obey the rules of common statistics
generally associated to non equilibrium processes, a more general statistics has been proposed
based on superstatistics \cite{Obregon,Tsallis2,Beck} which considers large fluctuations of intensive quantities \cite{Beck}.
We review here this formulation in which the intensive fluctuating quantity is the temperature.
This fluctuation gives rise to a certain probability distribution characterized by a generalized Boltzmann factor.
We can, in principle, associate an entropy for
every probability distribution and in this context the Boltzmann-Gibbs entropy corresponds to the usual Boltzmann factor. It is, however,
possible to obtain other generalized expressions for entropies associated to different probability distributions \cite{Tsallis1} depending on one or
several parameters \cite{Beck,Tsallis2}. In \cite{Obregon,Obregon1}, it was shown how to generate an entropy depending only on the probability.
The particular entropy considered here arises from a generalized gamma distribution depending only on the probability $p_l$.
This entropy has several interesting features and it seems to be relevant for particular thermodynamic systems
like confined systems \cite{Obregon1} and in this context we find an interesting and necessary application of such generalized entropies.
The quantum version of this entropy which is a generalization of the Von Neumann entropy arises by means of a natural generalization
of the replica trick \cite{Nana,Calabrese}.

In the formalism of the geometric approach to thermodynamics, a geometric structure is given for usual thermodynamic systems by means of
Riemannian geometry \cite{Weinhold,Rup,Rup1,Quevedo2,Quevedo1,Quevedo3}. Particularly in the context of the so called geometrothermodynamics \cite{Quevedo2,Quevedo3},
the geometrical relevant quantities, like the thermodynamic metric, are invariant with respect to Legendre transformations resembling the fact that
the thermodynamic information does not depend on what fundamental relation (thermodynamic potential) is used.
In this formalism, the representation invariance of the metric and its corresponding curvature scalar has been proved for simple thermodynamic systems \cite{Quevedo3}.
On the other hand, inspired in fluctuation theory, a distance between points in a thermodynamic space can also be defined \cite{Rup,Rup1,Rup2}, and we can
associate to this space a thermodynamic metric, a corresponding Riemann tensor and consequently a curvature scalar $R$.
Both approaches coincide in the physical interpretation of the curvature scalar as a
manifestation of the existence of intermolecular interactions. When the curvature
associated to the corespondent thermodynamic metric is non-zero, an interaction of some nature is present \cite{Rup1,Quevedo1}, this is known
as the curvature/interaction hypothesis. Other physical aspects that the scalar reveals, which has been proven for several thermodynamic systems,
is the existence of first order phase transitions. The curvature scalar diverges at some point if a phase transition exists. For some systems, the
point where the scalar diverges happens to be the critical point where the phase transition occurs \cite{Rup2,Quevedo3}.
In the thermodynamic geometry of fluctuation theory, the sign of the curvature scalar also provides additional information. For some systems it is clear that the sign of $R$
represents the kind of interaction, being attractive for a negative scalar and repulsive for a positive scalar \cite{Rup3}. The sign of $R$ can
also be associated to the bosonic or fermionic nature of the thermodynamic system, the Bose and Fermi ideal gases are a clear example of this,
we have $R < 0$ in the first case and $R > 0$ in the second case \cite{Janyszek}. For some other systems, it appears a change of sign in the curvature scalar, from negative
to positive or the other way around. These cases generally appear where a different statistics, other than that of Boltzmann's, is considered \cite{Rup3}.
Following the vast literature of the subject related to the thermodynamic geometry of the two approaches considered here, we find that the sign
interpretation is more clear in the formalism of G. Ruppeiner \cite{Rup3} but even so, it is not at all clear that this interpretation could be valid for
all thermodynamic systems.

In this work we particularly consider the curvature of a non-extensive ideal gas characterized by a generalized non-extensive entropy.
The particular entropy we use depends only on the probability distribution and arises in the realm of
superstatistics \cite{Obregon,Obregon1}. In order to get a better insight of the physics behind the curvature scalar of our thermodynamic system,
we calculate the curvature scalar using the two formalisms mentioned earlier, we will call these two scalars,
the geometrothermodynamic scalar for the scalar calculated following the formalism followed in \cite{Quevedo1} and the fluctuation theory scalar,
to the scalar calculated following \cite{Rup3}. We will find that the particular entropy (statistics) we propose \cite{Obregon,Obregon1} modifies the
geometric structure of the generalized thermodynamic space considered, namely a generalized ideal gas, giving rise to the appearance of
an effective interaction.

The paper is organized as follows: First in section II we explain how our modified entropy, and its associated Boltzmann factor, arises by assuming
a particular probability distribution. In section III we briefly introduce first the formalism of geometrothermodynamics developed by H. Quevedo \cite{Quevedo1}
and describe how the thermodynamic metric is calculated. In this same section we also introduce the thermodynamic metric
in the formalism of G. Ruppeiner \cite{Rup1}. We calculate the curvature scalar in both formalisms to further analyze and compare
the thermodynamic information contained in the scalars using the interpretation of both formalisms.
In section IV we make a discussion about the interpretation of both scalars and in section V we conclude and present the main results of our work.

\section{Generalized entropies depending only on the probability distribution}
The Boltzmann factor depending on the energy $E$ of a microstate associated with a local cell of average temperature $1/\beta$ is given by

\begin{equation}
B(E) = \int f(\beta)e^{-\beta E} d \beta
\end{equation}

\noindent and different distributions $f(\beta)$ lead to different Boltzmann factors. Following the procedure stated in \cite{Beck,Tsallis1} it is possible, in principle, to
associate a modified entropy to every Boltzmann factor. As an example we have that for the distribution $f(\beta) = \delta(\beta-\beta_0)$ the usual
Boltzmann factor is recovered and from this, the Boltzmann-Gibbs entropy follows directly \cite{Tsallis1}.
In \cite{Obregon,Obregon1} a Gamma distribution of the form

\begin{equation}
f_{p_l}(\beta) = \frac{1}{\beta_0 p_l \Gamma \left( \frac{1}{p_l} \right)} \left(  \frac{\beta}{\beta_0}\frac{1}{p_l}  \right)^{\frac{1-p_l}{p_l}}e^{-\beta/\beta_0 p_l}
\end{equation}

\noindent was proposed where, by maximizing the appropriate information measure, the parameter $p_l$ can be identified with the probability and $\beta_0$ is the average inverse temperature.
This distribution yields to the Boltzmann factor

\begin{equation}
B_{p_l} (E) = (1+p_l\beta_0 E)^{-\frac{1}{p_l}}
\end{equation}

\noindent that leads to the following generalized entropy \cite{Obregon1}

\begin{equation}
S = k\sum_{l = 1}^{\Omega} (1-p_{l}^{p_{l}}).
\end{equation}

Associated to this generalized entropy there is a generalized $H$ function,
\begin{eqnarray}
H = \int d^3p \ \left(  e^{f \ln f} - 1 \right)
\end{eqnarray}

which it can be shown that it satisfies a generalized H-theorem \cite{Torres}.
Using a Maxwell distribution to calculate this new $H$ function, keeping only the first order correction and the relation $H = - S/kV$, it follows

\begin{align}
S_{eff} =  & -kN\left[  \ln{ (n \lambda^3) } -\frac{3}{2} \right] \label{Sgen} \\ \nonumber
& - \frac{kVn^2\lambda^3}{2^{5/2}} \left[ \ln^2{(n \lambda^3)}  -\frac{3}{2}\ln{(n \lambda^3)} +\frac{15}{16}  \right],
\end{align}

\noindent where $\lambda = \frac{h}{\sqrt{2\pi mkT}}$ can be identified with the mean thermal wavelenght, $k$ is the Boltzmann constant,
$V$ is the volume and $T$ is the absolute temperature. The authors in \cite{Torres} study the thermodynamic properties of the corresponding generalized ideal gas.
In this context, analysis of response functions shows a first correction having a universal form, that is, the same functional correction to
all thermodynamic quantities derived from the generalized equations of state.

In order to obtain a thermodynamic potential, we say that the conventional  linear relation between internal energy and temperature holds.
Within this approximation we obtain the following thermodynamic fundamental relation

\begin{align}
S = & kN\ln{v} + \frac{3kN}{2}\ln{\frac{u}{b}} + \frac{3kN}{2} \label{GenEntropy} \\ \nonumber
& -\frac{kN}{2^{5/2}}\frac{b^{3/2}}{u^{3/2}v} \left[  \ln^2{ \left( \frac{b^{3/2}}{vu^{3/2}} \right)}  - \frac{3}{2}\ln{\left( \frac{b^{3/2}}{vu^{3/2}} \right)} +  \frac{15}{16} \right]
\end{align}

\noindent where $u = \frac{U}{N}, v = \frac{V}{N}$ and $b = \frac{3h^2}{4\pi m}$. The first terms correspond to the usual entropy of the ideal gas,
 $S = kN\ln{v} + \frac{3kN}{2}\ln{\frac{u}{b}} + \frac{3kN}{2}$.
We notice that the Sackur-Tetrode expression for the entropy of the ideal gas $S = kN\ln{v} + \frac{3kN}{2}\ln{\frac{u}{b}} + \frac{5kN}{2}$
can be recovered by an ad-hoc fixing term as it was originally proposed by Gibbs. It is not possible to recover the $5kN/2$ term from the classical calculation we made
but this does not affect the further analysis which involves derivatives of the entropy, and this constant term does not affect the final result.
At this point we have to clarify that the linear relation between
internal energy and temperature that we have assumed makes our calculations to be more accurate for low densities or high temperatures and we will take this into account
to make the interpretation of the behavior of the curvature scalars.

A previous thermodynamic analysis was made corresponding to a system characterized by a particular interaction.
The authors in \cite{Torres} considered a system of gas particles exposed to square-well and Lennard-Jonnes potentials.
These potentials are well defined and a respective Boltzmann factor $B(E) = e^{-\beta U}$ can be associated to these systems.
A further analysis with Monte Carlo simulations showed that when considering the generalized statistics and generalized entropy (\ref{GenEntropy})
the resulting thermodynamics can be modeled with a classical Boltzmann factor but with an effective interaction

 \begin{equation}
 B(E) = e^{-\beta U_{eff}},
 \end{equation}

 in the sense that the same form of the potential is recovered but with the redefinition of its parameters.
 For the particular interactions considered and for particular thermodynamic states, the contribution generated by the new statistics
 in the generalized potential turned out to be repulsive \cite{Torres} but the nature of the interaction depends in general on the thermodynamic states considered.

 In the next section we move to the thermodynamic geometry formalism and calculate the scalar curvature $R$ associated to the fundamental relation (\ref{GenEntropy}).
 It is our purpose to investigate and interpret the physical information contained in $R$ in two different approaches. We will calculate the geometrothermodynamic scalar
 \cite{Quevedo1} and the fluctuation theory scalar \cite{Rup}.
 The main reason to make the calculation in these two different formalisms
 is because we are interested to see wether or not the two calculations show the same qualitative behavior in order to make an accurate physical interpretation
 according to these two, in principle, different theories.
 The motivation and the origin of these two approaches is different but they have some common points of view regarding the physical interpretation of $R$
 in terms of its behavior.

\section{Generalized thermodynamic metric and curvature}

As we have mentioned earlier, in this section we will calculate the thermodynamic metric and its corresponding curvature scalar associated to the entropy (\ref{GenEntropy})
in two formalisms; i) using geometrothermodynamics, namely using Quevedo's metric. In this formalisms we will refer to the curvature scalar as $R_G$. ii) using fluctuation theory,
namely using Ruppeiner's metric. In this formalisms we will refer to the curvature scalar as $R_F.$

\subsection{Geometrothermodynamic approach}

In this formalism, the way in which the metric of the thermodynamic space is calculated makes it an object invariant under Legendre transformations.
This fact allows to include in the geometrical description of thermodynamics the invariance under thermodynamic potential representation \cite{Quevedo1,Quevedo3}. The
thermodynamic potential is an essential part of the so called contact manifold which represents the thermodynamic space and its geometrical properties will be
encoded in its related metric and geometrical objects related to it \cite{Quevedo1}. A contact manifold is characterized by three specific components
$(\tau,\Theta,G)$, a $2n+1$-dimensional differential manifold $\tau$ with tangent space $T(\tau)$, a
differential 1-form $\Theta$ on the cotangent manifold $T^{*}(\tau)$ such that an existing field of hyperplanes $\nu$ satisfies $\nu = \ker \Theta$ and a non degenerate
metric $G$ on $\tau$ \cite{Quevedo1,Quevedo2}. We can choose a coordinate patch on $\tau$ $Z^A = \{\  \Phi,E^a,I^a \}\ $ with $a = 1,...,n$. In the particular thermodynamic
representations $\Phi$ is identified with the thermodynamic fundamental relation, namely any thermodynamic potential containing all the thermodynamic information of the system.
$E^a$ and $I^a$ correspond to the extensive and intensive variables, respectively. The general metric $G$ is given by

\begin{equation}
G = (d\Phi -I_adE^a)^2 + \Lambda(E_{a}I_a)^{2k+1}dE^{b}dI^{b}. \label{Metric1}
\end{equation}

\noindent where $k$ is an integer and $E_a = \delta_{ab}E^a,~I_a = \delta_{ab}I^a$. $\Lambda$ is an arbitrary Legendre invariant real function of the variables $E^a$ and $I^a$.
One special characteristic of thermodynamic systems is the fact that extensive and intensive variables are related among them through the equations of state.
The functional relation among these variables is given in a geometric way by means of a harmonic map
$\varphi : \{\  E^a \}\ \rightarrow \{\ Z^A(E^a)  \}\ = \{\ \Phi (E^a),E^a,I^a(E^a) \}\ $. The space of coordinates $\mathcal{E}$ is a sub manifold of $\tau$ which is equipped
with the non degenerate metric $G$, so the pullback of the harmonic map $\varphi^*$ induces a metric on $\mathcal{E}$, i. e.  $g = \varphi^*$.
Given the equilibrium conditions given by the equations of state the simplest thermodynamic metric for the entropy representation in terms of
internal energy $u$ and volume $v$ $s(u,v)$ is given by

\begin{align}
g = & \left[  \left( u\frac{\partial s}{\partial u}  \right)^{-1}\frac{\partial^2 s }{\partial u^2}du^2
+ \left( v\frac{\partial s}{\partial v}  \right)^{-1}\frac{\partial^2 s }{\partial v^2}dv^2 \right]  \label{Metric2}  \\ \nonumber
& + \left[ \left( u \frac{\partial s}{\partial u}  \right)^{-1} + \left( v\frac{\partial s}{\partial v}  \right)^{-1}  \right] \frac{\partial^2 s}{\partial u \partial v}dudv,
\end{align}

which can be computed straightforwardly if the fundamental relation $s(u,v)$ is known.  We should remark that particular values of $\Lambda$ and $k$ has been chosen
in order to get the metric (\ref{Metric2}). It is not clear that an arbitrary choice of $\Lambda$ and $k$ will display the same physical information in the curvature scalar and
It could also be possible that these kind of general metrics (\ref{Metric1}) will not be in general invariant under infinitesimal Legendre transformations \cite{Monsalvo}.
However, it has been shown that this metric is able to predict accurately the divergence of the curvature scalar at the critical point where
the phase transition occurs for some ordinary thermodynamic systems \cite{Quevedo3}. In that sense, we will consider this metric as a good one to
infer if our generalized system has some interaction or a phase transition, information
that can be deduced from its corresponding curvature scalar. Nevertheless, we will compare the results with the curvature scalar using the metric of the fluctuation theory
approach (\ref{RMetric1}) which was constructed with no reference to Legendre invariance at all \cite{Rup3}.

We can compute now the components of the metric associated to the entropy (\ref{GenEntropy}), these are given by

\begin{align}
& g_{uu} =- \frac{3}{2u^2} \left[ \frac{1-\frac{21\sqrt{2}}{256}x + \frac{17\sqrt{2}}{32}x\ln{ x } + \frac{5\sqrt{2}}{16}x \ln^2{ x }}{\frac{3}{2} - x \left(  \frac{27\sqrt{2}}{256}  -\frac{3\sqrt{2}}{32} \ln{x } -\frac{3\sqrt{2}}{16} \ln^2{x} \right) } \right], \label{QMetric} \\ \nonumber
& g_{uv} = g_{vu} = \frac{x}{uv}  \left[ \frac{\frac{3\sqrt{2}}{256} - \frac{15\sqrt{2}}{32}\ln{x } - \frac{3\sqrt{2}}{16} \ln^2{x}}{1 - x \left(  \frac{9\sqrt{2}}{128}  -\frac{\sqrt{2}}{16} \ln{x} -\frac{\sqrt{2}}{8} \ln^2{x} \right) } \right], \\ \nonumber
& g_{vv} =- \frac{1}{v^2} \left[ \frac{ 1-\frac{5\sqrt{2}}{64}x + \frac{3\sqrt{2}}{8}x\ln{x} + \frac{\sqrt{2}}{4}x \ln^2{x}}{ 1 - x \left(  \frac{9\sqrt{2}}{128}  -\frac{\sqrt{2}}{16} \ln{x} -\frac{\sqrt{2}}{8} \ln^2{x}      \right) } \right], \\ \nonumber
\end{align}

\noindent where we have introduced the dimensionless factor $x = \frac{b^{3/2}}{u^{3/2}v}$.  We notice that
in the limit of low densities and high temperatures, that is when $x \rightarrow 0$, the entropy (\ref{GenEntropy}) and metric (\ref{QMetric}) approach respectively
to those corresponding to the ideal gas \cite{Quevedo1,Quevedo3}. The curvature scalar for the metric (\ref{QMetric}) in terms of $x$ is then given by

\begin{align}
& R_G = -\frac{3 \cdot 2^{7}\left[  I_0 - 2^3 I_1 \ln{x}
- 2^5 I_2 \ln^2{x} + 2^7 I_3 \ln^3{x} \right] }{G_0D(x)^3 } \label{QScalar} \\ \nonumber
& -\frac{3 \cdot 2^{7}\left[ 2^{8} I_4 \ln^4{x} - 2^{12}I_5\ln^5{x} -2^{14}I_6 \ln^6{x} \right]}{G_0D(x)^3} \\ \nonumber
& +\frac{3 \cdot 2^{7}\left[ 2^{16} I_7 \ln^7{x} - 2^{16} I_8 \ln^8{x} - 2^{19} I_9\ln^9{x} \right]}{ G_0D(x)^3} \\ \nonumber
& - \frac{3 \cdot 2^{7} \left[ 2^{21}I_{10} \ln^{10}{x} + 11 \cdot 2^{23} I_{11}\ln^{11}{x} - I_{12}\ln^{12}{x}  \right] }{ G_0D(x)^3 }, \\ \nonumber
& D(x) = (G_1 + G_2)\ln{x} + G_3\ln^2{x} + G_4\ln^3{x} + G_5\ln^4{x}.
\end{align}

\noindent where the $I_i's$, and $G_i's$ are dimensionless functions depending on $x$. We will analyze the behavior of this scalar
 in the section of discussions where we will show the plot of $R_G$. In order to have a better insight with respect to the physical interpretation of the curvature scalar,
 in the next section we will also compute the curvature scalar in the fluctuation theory of thermodynamic geometry \cite{Rup,Rup1}.

\subsection{Fluctuation thermodynamic geometry approach}

Now we turn to the calculation in the formalism developed by G. Ruppeiner \cite{Rup,Rup1,Rup3}. The thermodynamic curvature in this formalism
comes from fluctuation theory. The distance in the thermodynamic space arises from the calculation of the probability of a fluctuation of some variable
$x^{\alpha}$ away from equilibrium of a system with fixed volume $V,$ using Einstein's formula \cite{Rup1}

\begin{equation}
\text{Probability} \propto \exp{\left( -\frac{V}{2}(\Delta l)^2 \right)},
\end{equation}

where the differential of distance squared $(\Delta l)^2$ defines a thermodynamic information metric $(\Delta l)^2 = g_{\alpha \beta} \Delta x^{\alpha} \Delta x^{\beta}$,
with $\Delta x^{\alpha} = (x^{\alpha} - x^{\alpha}_0)$, being $x^{\alpha}_0$ the value of the fluctuating variable $x^{\alpha}$ at equilibrium.
The thermodynamic space
could be in general considered as $n$-dimensional and the $x^{\alpha}$ represent the independent fluctuating variables in this thermodynamic space.
Considering the fluctuation of two variables, $x^1$ and $x^2$ are two independent extensive variables or mechanical parameters.
The entropy can be considered to be a function of internal energy and the number of particles. With fixed volume, the variables
of entropy $s(u,n) = S/V$ are the densities, $u = U/V$ $n=N/V$. The expansion to second order of the conditional probability allows to define the following metric

\begin{equation}
g_{\alpha \beta} = -\frac{1}{kV}\frac{\partial^2 s}{\partial x^{\alpha} \partial x^{\beta}}, \label{RMetric1}
\end{equation}

where $k$ is Boltzmann's constant and $V$ is the fixed volume. We will need to rewrite the entropy (\ref{GenEntropy}) in terms of the densities $u,n$ in order to
calculate the curvature scalar associated to (\ref{RMetric1}), the entropy is given by

\begin{align}
& S = kn \ln{\frac{1}{n}} + \frac{3kn}{2}\ln{\frac{u}{bn}} + \frac{3kn}{2} \label{REntropy} \\ \nonumber
& -\frac{kn}{2^{5/2}}\frac{b^{3/2}n^{5/2}}{u^{3/2}} \left[  \ln^2{ \left( \frac{b^{3/2}n^{5/2}}{u^{3/2}} \right)} - \frac{3}{2}\ln{\left( \frac{b^{3/2}n^{5/2}}{u^{3/2}} \right)} +  \frac{15}{16} \right],
\end{align}

and the components of the metric following this formalism are given by

\begin{align}
& g_{uu} = \frac{3n}{2u^2V} \left[ 1- \frac{21\sqrt{2}}{256} z + \frac{17\sqrt{2}}{32}z \ln{z} + \frac{5\sqrt{2}}{16}z \ln^2{z}  \right],  \label{RMetric3} \\ \nonumber
& g_{nn} = \frac{5}{2nV} \left[ 1- \frac{23\sqrt{2}}{256} z + \frac{27\sqrt{2}}{32}z \ln{z} + \frac{7\sqrt{2}}{16}z\ln^2{z}  \right],  \\ \nonumber
& g_{un} = -\frac{3}{2uV} \left[1- \frac{23\sqrt{2}}{256} z + \frac{27\sqrt{2}}{32}z \ln{z} + \frac{7\sqrt{2}}{16}z \ln^2{z}  \right].
\end{align}

where $g_{un} = g_{nu}$. In the last expression, we have used the dimensionless variable $z = \frac{b^{3/2}n^{5/2}}{u^{3/2}}$ which is convenient to measure how the system
gets away from ideality. It is clear that in the limit $z \rightarrow 0$, the entropy (\ref{REntropy}) and metric (\ref{RMetric3})
tend to the entropy and metric corresponding to those of the ideal gas \cite{Rup1}.  With the metric (\ref{RMetric3}) the curvature scalar is given in this case by

\begin{align}
& R_F = \frac{5 \cdot 2^{10}z\left[  \bar{I}_0 - 2^7 \bar{I}_1 \ln{z} - 2^8 \bar{I}_2 \ln^2{z} \right]}{n D(z)^3} \label{RScalar1} \\ \nonumber
& +\frac{5 \cdot 2^{10}z\left[ - 2^9 \bar{I}_3 \ln^3{z} + 2^{9} \bar{I}_4 \ln^4{z} + 2^{14} \bar{I}_5\ln^5{z} \right]}{n D(z)^3} \\ \nonumber
& + \frac{ 5 \cdot 2^{10}z \left[ 2^{15}\bar{I}_6 \ln^6{z} + 7 \cdot 2^{17} \bar{I}_7 \ln^7{z} + 7^22^{16} \bar{I}_8 \ln^8{z} \right] }{n D(z)^3 }. \\ \nonumber
& D(z) = \bar{G}_0 + 2^4\bar{G}_1\ln{z} + 2^5 \bar{G}_2\ln^2{z} + \bar{G}_3 \ln^3{z}+ \bar{G}_4\ln^4{z}
\end{align}

where $n = N/V$, so $R_F$ has dimensions of volume. Also here the $\bar{I}_i's$, and $\bar{G}_i's$ are dimensionless functions depending in this case on $z$.
In the next section we will show the plots of these scalars and analyze
them in terms of what their respective formalisms tell us about the interpretation of the
behavior of $R$ as function of its independent variables.

\section{Results and discussions}

Before we show the plots of the scalars (\ref{QScalar}) and (\ref{RScalar1}), we notice the following; $R_G$ is dimensionless and $R_F$ has dimensions of volume
and we can see from the generalized entropy Eq. (\ref{GenEntropy}) that we have the natural parameters $m$ and $h$ involved in our expressions, so we will introduce the Bohr radius $a_0$
as a parameter of length in order to construct dimensionless quantities and express the plots of the curvature scalars in definite units.
We can easily construct with $m,h,a_0$ the quantities $u_0 = \frac{h^2}{ma_0^2}$ of energy and $v_0 = a_0^3$ of volume. Using these quantities
we construct and plot the reduced curvature scalars $R^{*}_{G}(u^{*},v^{*})$ and $R_F^{*}(u^{*},n^{*})$ where $u^{*} = u/u_0$, $v^{*} = v/v_0$, $n^{*} = nv_0$ and $R_{F}^{*} = R_F/v_0$.

In order to define the entropy (\ref{GenEntropy}) we have assumed a linear relation between internal energy and temperature, approximation which is
valid in the region of low densities and high temperatures, so we are confident that our results are more accurate in these regions, that is when $x \rightarrow 0$ and $z \rightarrow 0$
which correspond to small deviations from the entropy of the ideal gas.
Following this argument, we have plotted the scalars as function of inverse of internal energy to show the regime of high temperatures
near the origin as it is the region of validity of the approximation made. Figure 1 shows the reduced curvature scalar corresponding to Quevedo's metric $R^{*}_G$
in terms of inverse reduced internal energy.
Figure 2 shows the reduced scalar $R^{*}_F$ corresponding to Ruppeiner's metric.
In both figures we fixed the reduced particle number density to $n^{*} = 10^{-4}$. A typical low density value was chosen as
$n = 10^{29}$, for which we have a reduced value of $n^{*} = 10^{-4}$.
From these figures we see that both scalars consistently tend to the ideal gas scalar (zero scalar)
in the limits $x \rightarrow 0$ and $z \rightarrow 0$. Figure 3 shows both scalars for a fixed lower density ($n^{*} = 10^{-5}$).

\begin{figure}[h]
\centering
\includegraphics[width=7cm]{QuevPlot1}
\label{Figure1}
\caption{Reduced curvature geometrothermodynamic (GT) scalar $R^{*}_G$  as function of inverse of internal energy for fixed particle density $n^{*}=10^{-4}$. For high temperatures, or low values of $1/u^{*}$, the scalar
consistently goes to zero. This limit corresponds to the limit $x \rightarrow 0$ which is the region where the approximations we made is valid.}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=7cm]{RupPlot1}
\label{Figure2}
\caption{Reduced fluctuation theory (FT) curvature scalar $R^{*}_F$ as function of inverse of internal energy for fixed particle density $n^{*}=10^{-4}$.}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=7cm]{QuevRupPlots1}
\label{Figure3}
\caption{Simultaneous plots of $R^{*}_F$ (small dashed line) and $R^{*}_G$ (large dashed line) as function of inverse internal energy for fixed lower particle number density $n^{*} = 10^{-5}$.}
\end{figure}

As we have mentioned earlier, the interpretation of $R$ has its little differences in both approaches but they both agree in at least two things; First, they both coincide
in the interpretation of the non-zero scalar as a manifestation of a non-vanishing intermolecular interaction; Second, the divergence of $R$ at some point
signs the existence of a phase transition. We also found in the literature that the sign of $R$ has a clear interpretation in the formalism
of fluctuation theory and is less clear in the geometrothermodynamic approach. The fact that the sign of $R$ in Quevedo's formalisms does not clarify
the kind of interaction is because the metric itself depends on an arbitrary global multiplicative function \cite{Quevedo1}.
According to the results followed by Ruppeiner, the sign of $R_F$ shows the nature
of the interaction, being attractive when $R < 0$ and repulsive when $R > 0$. In some cases, it reflects the fact that the system is composed of bosonic ($R< 0$)
or fermionic ($R>0$) particles, then some quantum aspects of the thermodynamic system are also contained in $R_F$ \cite{Janyszek}.
A generic discussion of the systems where the interpretation of the sign of $R$ is clear can be found in \cite{Rup3}.

The scalars $R_F$ and $R_G$ of our thermodynamic system show a non-zero effective interaction and in the right limit, both consistently approach to the
curvature scalar of the ideal gas, that is $R = 0.$ We have already clarified that a linear relation between internal energy and temperature has been assumed
in the fundamental relations (\ref{GenEntropy}), (\ref{REntropy}) and in this expressions for the entropy, it is clear that in the respective limits $x \rightarrow 0$
and $z \rightarrow 0$, the generalized entropy tends to that of the ideal gas.

\begin{figure}[h]
\centering
\includegraphics[width=7cm]{QuevPlot2}
\label{Figure3}
\caption{Dimensionless geometrothermodynamic curvature scalar $R^{*}_G$ as function of the parameter $x$.}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=7cm]{RupPlot2}
\label{Figure3}
\caption{Dimensionless Fluctuation theory curvature scalar $R^{*}_F$ as function of the parameter $z$.}
\end{figure}

We show also in Fig. 4 and Fig. 5, the dimensionless scalars $R^{*}_G$ and $R^{*}_F$ in terms of the parameters
$x$ and $z$ respectively. In this plots we can also see that the scalars have the right behavior in the ideal gas limits and here we notice that none of these scalars have
divergences in the valid range of the approximation, they indeed do not diverge at any point in the whole range. This imply according to the interpretation
of both formalisms that it does not exist a phase transition, so we have
that an effective interaction appears as a consequence of assuming a different statistics, statistics which on the other hand defines the generalized
non-extensive entropy of our non-extensive ideal gas.

\section{Conclusions}

We calculated the curvature scalar of a particular thermodynamic system,
one that corresponds to the generalization of an ideal gas generated by a modified entropy.
The modified entropy arises from a generalized Boltzmann factor (generalized statistics), namely a modified
probability distribution. We found that such modification in the probability introduces an effective interaction.
Using the curvature/interaction hypothesis of thermodynamic geometry we found that
such interaction shows up in a non-zero curvature scalar $R$.
Two different formalisms were used, the geometrothermodynamic
approach and the fluctuation theory one. Both formalisms recover the limit
of a conventional ideal gas characterized by a zero curvature scalar.
Despite the non-zero value for the curvature $R$ implying a non-zero interaction, no evidence
of a phase transition is obtained, namely the curvature $R$ does not diverge at any point.
It is remarkable the similarity in the results even when the metrics (\ref{Metric2}) and (\ref{RMetric1})
are in principle different.

In the fluctuation theory approach, a formal interpretation to the sign of $R$ can be given describing the effective interaction,
being attractive when $R < 0$ or repulsive when $R > 0$. We have therefore, according to Fig. 2, the appearance of an effective interaction
as a consequence of introducing a different statistics. Near the limit of the ideal gas, the effective interaction is attractive but it also shows regions where the interaction
can become repulsive.

A more rigorous statistical analysis is needed to better understand non-equilibrium inspired systems that obey a
generalized statistic however, the curvature scalar
is a useful tool revealing well defined characteristics of the thermodynamic system.

\acknowledgments{ \noindent O. Obreg\'on was supported by CONACyT Projects No. 257919 and 258982, Promep and
UG projects. J. Torres-Arenas acknowledge the University of Guanajuato for the Grant 740/2016 Convocatoria Institucional de Investigaci\'on
Cient\'ifica and J. L. L\'opez was supported by CONACyT Grant 329847 and a PRODEP
posdoctoral grant.}


\section*{Introduction}

\subsection*{The law of leaky abstractions}
Software platforms invariably expose an abstract API to hide certain complexity from their users. However, it can be argued that they can never completely succeed in concealing the intricacies of the underlying subsystems and that computer abstractions are therefore always imperfect. This phenomenon was first described by Kiczales \cite{Kiczales92}, and the terminology "law of leaky abstractions" was introduced by Spolsky \cite{Spolsky02}. As an example, virtually every function call in Python might raise a \texttt{MemoryError} exception, yet this is usually not part of their specification. The common abstraction is one of infinite memory. Another example is that different, yet logically equivalent, SQL queries may have very different performance characteristics \cite{Spolsky02}.

As a corollary, users of such computer abstractions can, and do, run into situations where understanding the abstraction does not suffice for attaining the desired results. A Python developer who is confronted with \texttt{MemoryError} exceptions needs to understand \emph{how} their function call is allocating memory, and possibly how to decrease these allocations. An SQL developer who is confronted with slow-running queries needs to understand the inner workings of their database's query optimizer to understand how to obtain the desired performance.

\subsection*{Abstractions in online experimentation platforms}
Abstraction in software systems is not limited to hiding complexities of underlying software or hardware systems. In many cases, it applies to theoretical concepts or even objects in physical space. Experimentation platforms in particular commonly abstract away the following aspects of experimental design:

\begin{description}[style=unboxed,leftmargin=0cm]

\item [Statistical unit and tracking.] The platform determines what is the statistical unit of observation (e.g. login account, cookie or uninterrupted sessions). Then for a given event (e.g. http request) it finds the corresponding unit and tracks accordingly.

\item [Sampling.] The platform imposes treatment to a (usually randomized) subset of traffic. It takes responsibility for ensuring that this selection of current traffic is representative of traffic as a whole. When reporting the resulting metrics, and showing them with confidence intervals and predictions of overall impact, it is implicitly taking responsibility for the assumption that both the treatment and control selections of current traffic are representative of future traffic. It also assumes that on the side of the IT infrastructure, showing treatment to a percentage of traffic has the same (e.g. performance) characteristics as showing treatment to full traffic.

\item [Definition and implementation of metrics.] The platform defines how metrics are gathered and thereby also their exact definition.

\item [Business meaning of metrics.] The platform will likely interpret an improvement in click-through rate, revenue, or conversion as being "good". It may highlight this result in a positive way, or even end the experiment and select the treatment as the new default.

\end{description}

We observe that all four of these abstractions can leak, in which case the experimenter needs to be well-informed of their intricacies in order to base a sensible decision on the collected data. This has been alluded to by Tang et al. \cite{Tang10}

\begin{displayquote}

There are times (\ldots) where something in the actual implementation goes awry, or something unexpected happens. In those cases, the discussion is as much a debugging session as anything else. Having experts familiar with the entire stack of binaries, logging, experiment infrastructure, metrics, and analytical tools is key.

\end{displayquote}

The conceptual framework put forward in this paper can help explain why the need for "experts familiar with the entire stack" can not be avoided, as well as enable us to categorize common pitfalls in terms of which specific abstraction is leaking.

\section*{Examples of leaky abstraction}

Let us consider a few known experimentation pitfalls through the lens of leaky abstraction. Kohavi et al. \cite{Kohavi12} describe a situation where a certain experiment showed an unexpected but statistically convincing uplift in click-through rate.

\begin{displayquote}

The "success" of getting users to click more was not real, but rather an instrumentation difference. Chrome, Firefox, and Safari are aggressive about terminating requests on navigation away from the current page and a non-negligible percentage of click-beacons never make it to the server. (\dots) Adding even a small delay [in the treatment] gives the beacon more time, and hence more click request beacons reach the server.

\end{displayquote}

We can regard this situation as the experiment framework implementing the metric of "click-through rate" as "successful request beacons reaching the server". This implementation is abstracted away, but in this particular instance, it leaks to the surface. Knowing that this is a situation where this abstraction is known to fail, we could mitigate the future impact of leaks by warning experiment implementers when using beacons close to a point where the user is likely to initiate navigation.

A key point in this previous example is that \emph{it is the treatment itself that interferes with the platform's workings}. This means that implementing the feature and measuring its impact in an experiment are not concerns that can be separated. The experimenter needs to understand the platform's implementation.

As another example, consider the situation where treatment involves caching certain data for each user. One might imagine a situation where 50\% of traffic will have caching needs that can be satisfied by the infrastructure with a very low eviction rate, whereas 100\% of traffic would lead to a much higher eviction rate and consequently worse performance. This would likely result in abstraction leakage, as the experimentation platform silently assumes that exposing 50\% of traffic to treatment will have the same effect as exposing 100\% of traffic to it.

In this case, there is \emph{nothing the experimentation platform could do} to flag this issue. The experimenter has to rely on their own experience and analysis to foresee this possible problem. A possible mitigation strategy would be to also test the experiment at different traffic splits (e.g. if the treatment is positive for 50\% of traffic, then run the experiment again for 90\% of traffic). In this particular example, one can also carefully monitor cache eviction rates.

\section*{Mitigating abstraction leakage}

The law of leaky abstractions states that leakage is unavoidable. Nonetheless, we propose three general methods for reducing the risks:

\begin{description}[style=unboxed,leftmargin=0cm]

\item [Increased user awareness.] Users of experimentation platforms must be made conscious of the abstractions the platform is hiding from them. Personal experience tells us that users are often not aware of the design decisions that were made by platform developers. By making these chosen defaults explicit, users are in a better position to design experiments that work with the platform rather than against it.

\item [Expert experiment review.] Expert review and assistance setting up experiments should be made available when needed. A formal process or some additional user education may be required to help users identify potentially problematic scenarios.

\item [Automated warnings for known pitfalls.] Automated warnings for known pitfalls should be implemented as part of the platform user interface. This can flag potential issues for more in-depth review; possibly aided by experts. Alerting users to potential issues and guiding them in their analysis of the underlying complexity has the added benefit of serving as additional “training on the job”, which might help experimenters better identify future issues which the system does not detect.

\end{description}

\section*{Conclusion}

In this paper we have identified four aspects of experimental design commonly abstracted away by experimentation platforms: statistical unit and tracking, sampling, definition and implementation of metrics and business meaning of metrics. Two examples of abstraction leakage are presented: click beacons and cache eviction. Finally, we suggest how to mitigate these through increased user awareness, expert experiment review and automated warnings for known pitfalls.

The conceptual framework provided in this paper aids implementers and users of online experimentation platforms to better understand and tackle the challenges they face. Leaky abstraction can function as a helpful lens through which to consider several previously known pitfalls.

\section*{Acknowledgements}

The ideas put forward in this paper were greatly influenced by our work on the in-house experimentation platform at Booking.com, as well as conversations with colleagues and other online experimentation practitioners.