When should you balance a time series dataset? I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?
If you can change the Loss function of the algorithm, It will be very helpful and as a result you don't need to down sample your data. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

&gt;&gt;&gt; from pycm import *

&gt;&gt;&gt; cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

&gt;&gt;&gt; print(cm.recommended_list)
["Kappa", "SOA1(Landis &amp; Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]


After that, each of these parameters you want to use as the loss function can be used as follows:

&gt;&gt;&gt; y_pred = model.predict      #the prediction of the implemented model

&gt;&gt;&gt; y_actu = data.target        #data labels

&gt;&gt;&gt; cm = ConfusionMatrix(y_actu, y_pred)

&gt;&gt;&gt; loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)