python: why add new features of image to be worse result with xgboost
Guessing by your reported accuracies and the 93 sample points, it looks like you have just 19 samples in your validation set (the three accuracies you report then are 11/19, 13/19, and 15/19).  So the smallest change in accuracy you can get is more than 5 percentage points, and your reported swings don't seem so drastic.

However, continuous variables can lead to overfitting, if your model learns each particular value separately (which would also lead to your continuous variables getting high feature importance scores).  The default XGBClassifier parameter max_depth=3 makes that a little less likely, but the default n_estimators=100 for only 74 samples might well be at fault.  I'd agree with bradS that you should tune hyperparameters a bit.  Also, it's probably worth at least throwing something else like logistic regression at the data.  But again I'm not sure how much fine-tuning you can realistically do with just 19 validation points.