Random Forest Stacking Experiment for Imbalanced Data-set Problem
The underlying idea is fine, but you've fallen into a common data leakage trap.  By recombining the data and then resplitting, your second model's test set includes some of the first model's training set.  The first model knows the labels on those datapoints and, especially if overfit, passes along that information in its predictions.  So the score you see for the ensemble is probably optimistically biased.

The most common approach to fixing this is to use k-fold cross-validation to produce out-of-fold predictions on the entire training dataset for the second model.  

Note that sklearn now has such stacked ensembles builtin:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html