Why does the same algorithm give very different metrics on similar datasets? I used Random Forest and hypertuned the parameters for a binary classification problem on a dataset (dataset A). I got a F1 score of 0.78. I then used a second dataset (dataset B). It was very similar to dataset A (same variables and the distribution of classes in the target variable). I again built and trained a different Random Forest algorithm for dataset B. I expected the f1 score to be around 0.78, but the f1 score for dataset B was 0.50.

Why could there be such a large difference between the f1 scores of the 2 datasets? 

Both datasets (A &amp; B) are very similar to each other and I trained separate models on both of them.
Simple answer: the datasets are different. Apparently, your approach worked well for A but not for B. From this you might infer, that B and A are not so similar after all. I suspect that B has a more complicated relation between the X and targets y. So maybe try boosting on B?!