Classifier not predicting real data I'm trying to train a classifier to recognize my own signature. This is how I built my classifier

How did I collect data?


Signed on a piece of paper for 50 times and created 50 images out of it.
for negative test cases, I downloaded IAM Handwriting database. which contains around 600MB of handwriting data. this is to negate other possible matches.


How did I do Feature Engineering?

Step 1 : Read and convert image in gray scale. Perform median blur.

img = cv2.imread(image,0)
img = cv2.medianBlur(img,5)


Step 2 : Perform adaptive threshold followed by morphological opening to reduce noise in the image.

edges = cv2.adaptiveThreshold(img,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY_INV,11,2)    
kernel = np.ones((2,2),np.uint8)
dilation = cv2.morphologyEx(edges, cv2.MORPH_OPEN, kernel)


How did I do model training?


Extracted ORB features from all the sample images for training (matrix size 
RX32) and used a RandomForestClassifier for training.


The problem

The accuracy of my classifier is a whopping 0.9874066374996978, but it fails to recognize almost all of my new signature samples, signed on same paper under same lighting conditions. I'm new to applied ML. What do you experts think I should check??
You are effectively trying to classify images. Convolutional neural networks are much better at this task. And Keras is an easy to use library for neural networks. 

Check tutorials such as: https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/ 

and  

https://www.pyimagesearch.com/2017/12/11/image-classification-with-keras-and-deep-learning/