How to find optimal number of trees in random forest using Grid search in R?
mtry is the parameter indicating how many of the features are checked in each split decision.  http://topepo.github.io/caret/train-models-by-tag.html#random-forest , see method 'rf'

In Random Forest, usually more trees give more stable results, and overfitting due to number of trees is rare.  Moreover, since the trees are built independently, you could just fit many trees then take subsets to get smaller models.  But, see e.g. hyperparameter tuning in mlr for how to perform grid, random, Bayesian, and other hyperparameter searches, or grid in caret or random in caret.