Parameter tuning for machine learning algorithms
In general, your approach will get stuck in local minima. This is why it is not scientifically accepted. 
(Notice that this may be different in very special cases, in particular if the performance of the algorithm is a strictly convex function of all input parameters).

To see how the approach fails, suppose your machine learning algorithm has two parameters, $x$ and $y$, which can be either $0$ or $1$. The default values are $x=1$ and $y=1.$

The performance of your machine learning algorithm is $f$ and should be as high as possible.
Assume the following performance levels $f(x,y)$:

    | x=0 | x=1 
----|-----|-----
y=0 | 0.9 | 0.2
y=1 | 0.1 | 0.3


Your approach would do the following:


First, choose the default value $x=1$ and compute $f(x=1, y=0) = 0.2$ and $f(x=1, y=1) = 0.3$. Second, choose the default value $y=1$ and compute $f(x=0,y=1)=0.1$ and $f(x=1, y=1) = 0.3$.
Change the default values to the best value. In this case, this requires no change since $x=1$ and $y=1$ are the best values, respectively.
The result did not change. Report $(x=1, y=1)$ as the best parameter combination.


But the global performance maximum occurs at $(x=0, y=0).$