Significant difference in testing vs cross validation accuracy
Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted. 

A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation. 

Possible strategies to get a better estimation of the model performance would be:


useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)
considering simpler models (e.g., less variables, more general parameters)
considering other machine learning algorithms