Random Forest Modelling? I use random forest to train on my data (My data had imbalance in the target class, i.e. rare 1 and abundant 0). I face 3 issues about the stability of estimator and its prediction power. I think these problems could be common on many machine learning algorithms.


I found the ROC_AUC_Score highly changeable when I resample the training set (the rest was test set). It can vary from $0.85$ to $0.45$ while changing the training set. 
The parameter tuning can also cause the move of the estimator and ROC_AUC_Score but the effect was weaker than the first case above. 
When running some iteration of model fitting, the results were also different from each other but the effect was the weakest. 


For 2, 3, I think we can get the best parameter setting and fitting results by recording each iteration and parameter tuning. (Maybe can do it more efficiently, please advice)

Please also advice how to deal with the first problem to make the fitting convinced and reliable? Cross Validated?

Many thanks
You should definitely use nested cross-validation for model selection and performance estimation. I have also found that the AUC of the precision-recall (PR) curve (compared to that of the ROC curve) to be a better, that is, more stable, estimator of the performance of my Random Forest classifiers when I have a highly unbalanced dataset; there is research on this topic, showing that the AUC of the PR curve is more informative than that of the ROC curve. You can use average_precision_score() in scikit-learn to use the PR AUC score. Along the lines of resampling the data, you could try approaches like EasyEnsenble and BalanceCascade; search for the papers titled "Exploratory Undersampling for
Class-Imbalance Learning" and "Learning from Imbalanced Data" for more information.