What is a logical approach to developing an algorithm which can find the optimal parameters for a function which make it best fit a given data set?
The answer depends on what is your notion of "optimal parameters", and on the specific model you have in mind.

Suppose your model is $f(x,\overline{p})$, where $p$ is the parameters vector, and you wish to find the vector $\overline{p}$ which minimizes $
\sum\limits_{i=1}^n\left( y_i-f(x_i,\overline{p})\right)^2$. This problem is known as least squares fitting, and if your model $f(x,\overline{p})$ is linear, then the optimal value of $\overline{p}$ is given by ${(x^Tx)^{-1}x^Ty}$. For more complicated models, the solution might not be so easy to find analytically, and one might need to settle for approximations. One option is to use iterative methods, such as the Gauss-Newton algorithm.

There is no general way to deal with any type of error function. If for example your cost function does not allow any mistake, and your model $f$ is polynomial in the parameters, then the task reduces to solving a system of polynomial equations. Different types of error functions can take you to different areas in optimization, or even in mathematics in general.