Which model is better for incremental learning? I'm trying to implement face recognition. I'm planning to use some model (like DeepFace) to extract discriminative features and then use a classifier to recognize the faces. I'm confused as to which classifier to use.

My setup is as follows:
Initially, I'll have a few labelled samples and I train a classifier. Then I start getting more labelled samples and I want to update my classifier with the new data. Also, the newer samples may contain new classes (faces). So, I want my classifier to be able to learn to classify new classes as well. My question is which classifier is better suited for this task. I would also prefer to get multiple predictions with a confidence score.


K Nearest Neighbors algorithm: Here I can use distance as some form of confidence score.
As presented here, I can use SVM: Train on the initial data and retain support vectors alone. Repeat the process as and when new data comes up.


But, I have no idea if the above 2 classifiers are suited for the task at hand or how they would perform. Any inputs regarding this is greatly appreciated. I'm also open to any other classifiers as well. Thanks!
I will say, it's an either Or situation
You can pick one of &quot;Incremental/Online&quot; training Or &quot;addition of new class&quot;.
 You may do a fine-tuning approach with a Neural network by adjusting the o/p layer and training the last few layers. But this approach expects the new data to be quite similar to the training set.
KNN - Can do the online stuff but it doesn't do training. It simply calculates all the distances at the time of prediction. So, no reduction in computing. But you might have to compromise on accuracy if it is not the best
Scikit-Learn SGD Classifier can help with online training but can't support new classes

For classification, a somewhat important thing to note is that although a stateless feature extraction routine may be able to cope with new/unseen attributes, the incremental learner itself may be unable to cope with new/unseen targets classes.
SGDClassifier

Other models e.g. SVM/DT doesn't support incremental learning naturally. Though there are suggested approaches on the internet. But may not be simple. See these references.
journalofbigdata$\hspace{1cm}$
A Good SE read
I am not sure how will you figure it out if it is a new Class unless you have a separate arrangement because the model will predict it an existing Class anyway. Assuming you have a setup for that. We may try below approach -
Till the time, the prediction is within &quot;known class&quot; - do an Online training.
When the data is for a &quot;New Class&quot;, do a full-data training.
 With this approach, you can reduce the frequency of full-data training.
A simple Neural network can be a good candidate for both purposes. You can also get class probability with it.
Please see these useful links too
Creme
Keras + online learning