Using a random forest, would a RandomForest performance be less if I drop the first or the last tree? Suppose I've trained a RandomForest model with 100 trees. I then have two cases:

I drop the first tree in the model.
I drop the last tree in the model.

Would the model performance be less in the first or the second case?
As the last tree should be the best trained one, I would say that the first scenario should be less performant than the last one.
And what if I was using another model like a Gradient Boosting Decision tree?  I guess it should be the same.
I am okay with some math to prove it, or any other way that might prove it.
Update
I tried with two different learning rates 0.1 and 8. With 0.1 I get:
# For convenience we will use sklearn's GBM, the situation will be similar with XGBoost and others
clf = GradientBoostingClassifier(n_estimators=5000, learning_rate=0.01, max_depth=3, random_state=0)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:, 1]
# &quot;Test logloss: {}&quot;.format(log_loss(y_test, y_pred)) returns  0.003545821535500366

def compute_loss(y_true, scores_pred):
    '''
        Since we use raw scores we will wrap log_loss 
        and apply sigmoid to our predictions before computing log_loss itself
    '''
    return log_loss(y_true, sigmoid(scores_pred))
    

'''
    Get cummulative sum of *decision function* for trees. i-th element is a sum of trees 0...i-1.
    We cannot use staged_predict_proba, since we want to manipulate raw scores
    (not probabilities). And only in the end convert the scores to probabilities using sigmoid
'''
cum_preds = np.array([x for x in clf.staged_decision_function(X_test)])[:, :, 0] 

print (&quot;Logloss using all trees:           {}&quot;.format(compute_loss(y_test, cum_preds[-1, :])))
print (&quot;Logloss using all trees but last:  {}&quot;.format(compute_loss(y_test, cum_preds[-2, :])))
print (&quot;Logloss using all trees but first: {}&quot;.format(compute_loss(y_test, cum_preds[-1, :] - cum_preds[0, :])))

which gives:
Logloss using all trees:           0.003545821535500366
Logloss using all trees but last:  0.003545821535500366
Logloss using all trees but first: 0.0035335315747614293

Whereas with 8 I obtain:
clf = GradientBoostingClassifier(n_estimators=5000, learning_rate=8, max_depth=3, random_state=0)
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:, 1]
# &quot;Test logloss: {}&quot;.format(log_loss(y_test, y_pred)) returns 3.03310165292726e-06

cum_preds = np.array([x for x in clf.staged_decision_function(X_test)])[:, :, 0] 

print (&quot;Logloss using all trees:           {}&quot;.format(compute_loss(y_test, cum_preds[-1, :])))
print (&quot;Logloss using all trees but last:  {}&quot;.format(compute_loss(y_test, cum_preds[-2, :])))
print (&quot;Logloss using all trees but first: {}&quot;.format(compute_loss(y_test, cum_preds[-1, :] - cum_preds[0, :])))

gives:
Logloss using all trees:           3.03310165292726e-06
Logloss using all trees but last:  2.846209929270204e-06
Logloss using all trees but first: 2.3463091271266125
The two slightly-smaller models will perform exactly the same, on average.  There is no difference baked in to the different trees: &quot;the last tree will be the best trained&quot; is not true.  The only difference among the trees is the random subsample they work with and random effects while building the tree (feature subsetting, e.g.).
Gradient boosted trees are a different story.  If you drop the first tree after you finish training, the resulting model will be mostly garbage.  Every subsequent tree was trained to improve upon the fit of the previous trees, and removing any single tree will put all future trees out of context.  (To give an extreme example, suppose the first tree actually captures &quot;the correct&quot; model.  All future trees will just fit on the remaining noise.)  On the other hand, removing the final tree is equivalent to having just trained one fewer tree, which may be good or bad depending on your bias-variance tradeoff at that point.