Can a novelty detection model overfit?
Answering your question: yes, depending on the hyperparameters you choose, you could overfit the considered normal data, if you fit your separating hyperplane between normal and novel points being too much &quot;shaped&quot; on your input data.
There are, for instance in case of one-class support vector machines, some important hyperparams like nu or gamma:

nu: with this one, you tell the oc-SVM the fraction of novel points (i.e. anomalies) you want to consider in your input data; this way, you don't overfit your model by not considering normal all the input data (it also depends on your use case, how sure you want to be about normality of the input data points...)
You can test it with the scikit-learn package this way:

with nu=0.01

VS with nu=0.1, where you tell the model to consider a higher fraction of points as being abnormal:

so, the lower nu value is, the more you are &quot;overfitting&quot; your novelties detector (which could be better or worse depending on how well you know your input data).

gamma: now, look at the effect of the value of gamma, which is crucial for overfitting your model:

with gamma=0.1 (and rbf kernel), you have the following decision surface:

VS with gamma = 10

where, with this last option, your are overfitting so much.