Make predictions with an old model without losing the current model
Would it be possible to use the weights to do it?
Put the weigh of your targets to 1 and the weights of your not-targets to 0
At first, you'd have your weight tensor at [1]*n + [0]*m (+ as in concat).
Then you'd assign it to [1]*(n+m) when you want to add you m new targets
and so forth.
I am training a model that incrementally learns new classes, e.g. n target classes during the first 70 or so epochs, then the original n classes plus m new target classes, etc. When training the model on n+m target classes, the loss function requires predictions from the model trained on n target classes. How can I restore the old model efficiently? 
It seems I can do this by creating two separate sessions for each batch and loading the old model in one before training the new model in the other, but this is terribly inefficient and makes training go from taking hours to days.