Forward or Backward Stepwise Regression
Imagine you have 20 coefficients to test for and also have target accuracy (or whatever metric you're interested in) that you aim to beat. It acts as a threshold. 

One tradeoff could be that performing "backwards regression" means you would in theory start with you maximum accuracy and be decreasing each time you remove a parameter. You might cut your threshold after removing only 3 parameters! So don't necessarily need to continue. 

Conversely, starting with a single parameter and working your way up, you would have to try 17 models 5o reach your threshold.

This scenario could of course be reversed, so it is important to understand your data and make an informed decision about what suits your situation best.