Transfer learning by concatenating the last classification layer Before going into an obvious XY problem, I will explain you what I'm trying to do.

I'm training a simple MobileNet pre-trained with Imagenet for multiclass classification. What I do is freeze all the convolutional part, and then create a new Prediction Layer (conv2D 1x1xN where N is the number of classes).

Let's say I trained with 5 classes, and then I got a new set of images of 5 new classes. I want to keep the knowledge from the last training and be able to train only with the new classes.

What I do is to train a new model with the same frozen pre-trained weights, but only with the new 5 classes, then concatenate the weights from the old and the new model into a last layer that outputs for 10 classes.

The concatenation works perfectly, but when I run the evaluation I got a horrible accuracy (like if it was predicted randomly). For example, for 30 clasess trained in "sets" of 5, I got an accuracy of 0.03.

Is what I'm doing viable? A problem that I think it can be causing this is that the "labels" for every prediction are not kept in the same order or something, so that even if I copy the weights, the predictions may be unordered.
It turns out that it was an issue related with the order of classes being loaded in the model.

Let's assume I have the following structure

root
  -train1
    +a
    +r
  -train2
    +b
    +y
  -all
    +a
    +r
    +b
    +y


Where a, r, b and y are classes (folders with images). I first train with the ones at train1, so the network assigns each output to a class. Then I train a second network (everything but last prediction layer frozen) with the second training folder train2.

The order of the outputs of the prediction layer will change depending on the algorithm, but assuming it's loading the paths in a sorted way:

In the first network, the first output will be for a and the second for r.

In the second network, the first output will be for b and the second one for y, and when concatenated to the first network, it will be a, r, b, y

When I load the whole (merged) model, and I load a folder will all classes inside (all), the network assumes the outputs are for a, b, r, y in this order, whilst the concatenated network had a, r, b, y as the output order.

In this example, the outputs for a and y will be in the same position, while r and b will be reversed, which will lead to a bad accuracy.

TL;DR:

The order in which classes are loaded while you do the "incremental learning" needs to be the same always, so be careful in which order your classes are being loaded.