Imbalanced classes (balance of train, validation, and test) 1) I am currently trying to set up a feedforward neural network with highly imbalanced classes (binary classification) in which the number of observations of class 1 is very low (and the class of interest to predict).

The dataset is very large, so in order to make the network better to predict class 1, I have downsampled the class 0 observations in the training set (ending up with a training set of approximately 600,000 observations).

Right now, I am estimating the neural network using 'accuracy' as the metric to optimize, while using early stopping to optimize the accuracy on an unbalanced validation set (monitor='val_acc').

My question is, therefore, whether I also should downsample class 1 observations in the validation set? The final test will naturally be unbalanced with a low number of class 1 observations.

Edit:
My initial thought was to do use a function that takes the unbalanced classes into account, but the problem was that the dataset is too large, so I run into memory errors before I am able to run the neural network.
After studying different approaches, I've read several papers suggesting to balance the classes (in which the problem of memory was also solved).
My thought was that by balancing the classes, the accuracy function would make sense to use, but since I optimize the network directly on the validation within the model, I'm confused whether the validation set should also be balanced



2) Additionally, is there any common order on how to optimize a neural network (etc. first determine the number of hidden layers, number of neurons, and so)? A paper or similar arguing for its order of optimization would be very interesting (haven't managed to find one myself).

Thanks in advance!
If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

&gt;&gt;&gt; from pycm import *

&gt;&gt;&gt; cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

&gt;&gt;&gt; print(cm.recommended_list)
["Kappa", "SOA1(Landis &amp; Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]


After that, each of these parameters you want to use as the loss function can be used as follows:

&gt;&gt;&gt; y_pred = model.predict      #the prediction of the implemented model

&gt;&gt;&gt; y_actu = data.target        #data labels

&gt;&gt;&gt; cm = ConfusionMatrix(y_actu, y_pred)

&gt;&gt;&gt; loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)