Effect of removing duplicates on Random Forest Regression
I agree with the previous answer about your specific question: your model is likely to be less accurate if you just remove training data.
Moreover:

With very large sample counts some models (like Random Forests) become very large (several GB when pickled).

It sounds like the size of your final model is the reason why you're considering pruning your dataset. Assuming you are using scikit-learn, I'd recommend looking at the documentation and the parameters that you can tweak that impact your final model size.
Specifically, parameters like n_estimators, max_depth, min_samples_split, and min_samples_leaf all have default values that tend to generate large models. For example, max_depth is set to None and therefore (as per the docs) &quot;nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.&quot; If you experiment with these you are likely to be able to generate much smaller models, in terms of size-on-disk, without needing to prune your dataset.