Gap leaderboard score and model scoring on a Competition
As you have commented, you are concerning about over-fitting.

In fact, cross validation will help to weaken over-fitting, but it can't eliminate over-fitting. Model scoring on TRAIN dataset sometimes exceeds scoring on TEST dataset. Here are some examples I can find:


Deep Residual Learning for Image Recognition (the famous ResNet paper). Checkout figure.1
This kernel of Two Sigma Financial Modeling Challenge on Kaggle says:



  we are getting a public score of 0.0169 which is slightly better than the previous one.
  Submitting this model to the LB gave me a score of 0.006


In my practice, I am more willing to concern the scoring gap between TRAIN, VALIDATION(DEVELOP) and TEST dataset. Personally speaking, TRAIN>VALIDATION=TEST is better than TRAIN=VALIDATION>TEST.

Edit: For class imbalance problem, there are some resources:


8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset. This blog shows a common workflow dealing with imbalanced class issue.
Class Imbalance Problem in Data Mining: Review. This paper compares several algorithms created for solving the class imbalance problem.