Transfer learning by concatenating the last classification layer
It turns out that it was an issue related with the order of classes being loaded in the model.

Let's assume I have the following structure

root
  -train1
    +a
    +r
  -train2
    +b
    +y
  -all
    +a
    +r
    +b
    +y


Where a, r, b and y are classes (folders with images). I first train with the ones at train1, so the network assigns each output to a class. Then I train a second network (everything but last prediction layer frozen) with the second training folder train2.

The order of the outputs of the prediction layer will change depending on the algorithm, but assuming it's loading the paths in a sorted way:

In the first network, the first output will be for a and the second for r.

In the second network, the first output will be for b and the second one for y, and when concatenated to the first network, it will be a, r, b, y

When I load the whole (merged) model, and I load a folder will all classes inside (all), the network assumes the outputs are for a, b, r, y in this order, whilst the concatenated network had a, r, b, y as the output order.

In this example, the outputs for a and y will be in the same position, while r and b will be reversed, which will lead to a bad accuracy.

TL;DR:

The order in which classes are loaded while you do the "incremental learning" needs to be the same always, so be careful in which order your classes are being loaded.