RandomForest surprisingly high accuracy I've been experimenting with Random Forests on Python after trying Naive Bayes which gave me lower accuracy than I expected, 62%. My csv file has around 14,000 records, I use 80% for the training set and 20% for testing set. I tried with different parameters like 100 trees, 500 and 1000, -1 for n_jobs and so on but during all those tests the accuracy never changed too much, it was always around 74% or 75%, almost 76% at times. I checked that the online tutorial which teaches how to implement got the accuracy of 73%, as my tests got higher than that, I wasn't worried, thought it's normal for this kind of algorithm.

However, my surprise came during my latest test, the only difference is I saved the model and vectorizer to file so I don't have to train the model each time. The new script loaded those files, applied the loaded (trained) model to the testing set and wow, I got the highest accuracy I've seen so far, 93.989%, almost 94%.

Anyone got a similar experience? Is this big increase related to having saved the model to a file and loaded it? The rest of the code is all the same. Or was I just too lucky and if I try more times I will go back to the usual around 75% accuracy?
You can be sure that saving the model to disk won't make the model have higher accuracy when loaded again. Saving the models to disk would only cause worse performance if parameters are truncated, for example. 

To establish a fair and rigorous comparison method, you have to keep track of your training/test splitting result. In that sense, you can:


Make your code reproducible, using random seeds conveniently 
Before saving the model, you can save your splitting results by either saving the data or returning the indices for the training set and the test set. For that purpose you have to store the data-set in a list (for example, a list with the file names…)