What if Training and testing dataset comes from the same source? I am working on a classification problem in which I have to distinguish between healthy and damaged plates. when I use the combination of k-means clustering and SVM algorithm together with 10-fold cross validation, I can achieve the accuracy up to 95%. All the training and validation datasets come from the experiment.
For the testing, can I get the datasets after repeating same experiments with same specimen or I have to use different sets of specimens?
You have to use a different set of specimens. Or you can keep one or two specimens from the original set aside and use them as test. Use data augmentation and transfer learning in that case.