How to properly use oversampling without inflating results? I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i decided to apply oversampling (SMOTE). Here is what i did:

Using the entire dataset (original 192 samples):
Create synthetic samples for each class using SMOTE, so i get a total of 500 samples per class (2000 in total)

I have a big suspicion about this procedure because when i apply SMOTE i get very high accuracy rates (even 100% in some cases) with the simplest models such as a 15-neuron MLP. So, i have some questions to assert the correctness of my experiments:

Is it ok to oversample the entire dataset or should i apply the SMOTE only in train data (keep in mind that this will leave few samples for test)?

SMOTE is originally used to deal with imbalanced datasets, creating samples for the classes with few sample. Is it ok to use it for generating samples for ALL classes in order to enlarge the entire dataset?
First of all - have you tried to do modeling without oversampling? If you dont have imbalanced classes, why do you need oversampling? You dont always need tons of data to get a good model as a baseline.
Regarding should i apply the SMOTE only in train data - always leave some test data out, dont apply any transformation on the whole dataset (even normalization or missing values imputing). If you have some preprocessing - do it on train data set, train model. Then apply the preprocessing pipeline on test data and predict on it.
simplest models such as a 15-neuron MLP. - it doesnt sound as simplest solution to me. have you tried any other approaches? also, have you checked confusion matrix to see the classification error on test data for each of the class?