Training with multi-series of different length with stateful LSTM
To begin with, you can think of the batch size as a way to control the smoothness of the learning curve. With a huge batch size, you are taking the average of many errors for each update, and this average loss (on average), doesn't have great variance.

Using a batch size of 1, your cost on each iteration is solely dependent on the single sample that you fed the network. Each sample is hopefully a little different from the others, which will lead to a very noisy loss curve.

Assuming model parameters are well-suited and the model converges irrespective of batch size, you should reach similar results. However, there are works that analyse batch size and show other trade-off, which are usually training-times, memory consumption and the like.

This recent paper (Masters, Luschi) analyses the trade-offs of batch size on a standard dataset (CIFAR10). Here is Figure 15 from the paper that shows this pretty succinctly:



Warning: all models and datasets could behave differently, however, so don't take any results like these as fact!



I know that it is possible to train a model that has varying input shapes by using a technique called Adaptive Pooling (or adaptive average pooling, in PyTorch, but you would likely have to come up with your own function that is able to do such a thing within the constraints of a stateful LSTM.

So as the shape of your dataset must be divisible by the batch size, there are a few ways to make that a reality:


using the highest common multiple of your datasets: so with your example datasets having 4000 days (set A) and 500 days (set B), you would compute the


This of course limits the possible choices of sequence length, but for the two examples of 4000 and 500, you could choose from these: 1, 2, 4, 5, 10, 20, 25, 50, 100


Trimming the different datasets so that you get a nice sequence length that works for both


This means possible leaving out some of your data, which is undesireable, but it might not be much.

Which of those two possiblities you might go for will depend on your specific dataset sizes. It might be optimal to use both - so trimming a few datasets to allow the computation of a nice Highest Common Factor.

You can compute the HCF in Python using something like this:

def hcf(a, b):
    while b:
        a, b = b, b % a
    return a




One final idea from my comment below:

... in order to change the batch size between datasets, you could copy them model weights from a trained model (from your first dataset), then compile a new model with the batch size required for the next dataset, but set the weights equal to those from the first model. This is almost like a manual way of implementing stateful behaviour. Have a look here for a simple example.