Why use parametric based estimation methods - confusion regarding terms Using the probability density function (pdf) we can estimate an unknown parameter using methods such as Maximum Likelihood estimation. If the pdf is not available, then Least Squares can  be used. Other methods for pdf unavailable situations are Least Mean Squares.

Then there are other methods which are called as heuristic based approaches such as particle swarm optimization, genetic algorithm, ant colony optimization. These are also applied to find an unknown parameter without using the pdf. This is quite a long post, so  a big thank you and I really appreciate the patience in taking the time out to read and understand my concerns. My questions are:

1) What is the difference between parameter estimation and optimization?

2) Is there a rule of thumb when to use estimation theory and optimization?

Many research articles and text books use these two terms interchangeably based on my understanding. For example, the abstract in the paper titled, "Parameter estimation with bio-inspired meta-heuristic optimization: modeling the dynamics of endocytosis" which is given below:

We address the task of parameter estimation in models of the dynamics of biological systems based on ordinary differential equations (ODEs) from measured data, where the models are typically non-linear and have many parameters, the measurements are imperfect due to noise, and the studied system can often be only partially observed. A representative task is to estimate the parameters in a model of the dynamics of endocytosis, i.e., endosome maturation, reflected in a cut-out switch transition between the Rab5 and Rab7 domain protein concentrations, from experimental measurements of these concentrations. The general parameter estimation task and the specific instance considered here are challenging optimization problems, calling for the use of advanced meta-heuristic optimization methods, such as evolutionary or swarm-based methods.

However, in signal processing, optimization is rarely combined or taught in conjunction with parameter estimation. For example, in the book by Steven Kay titled, "Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory" never mentions optimization as another method to do estimation. However, Gradient Descent is an optimization algorithm which uses the derivative of the function and equates to zero. If this is the usual way of doing optimization, then Maximum Likelihood estimation also find the derivatives with respect to the unknown parameter using the log-likelihood function, which are equated to zero to obtain the estimates. This brings to my last 2 questions which are:

3) Is MLE an optimization or estimation technique?

4) Heuristic approaches such as particle swarm etc do not find derivatives to obtain the minima or maxima of the objective function. Then why are they called optimization methods if they are not optimizing the cost function in the usual way which is taking the derivatives and equating them to zero?
Hi: I'll try to answer as briefly as possible and only with respect to statistics. not dsp. 

In statistics, if you have a nice pdf such as the normal distribution, then maximizing the likelihood is equivalent to minimizing the sum of squares of the residuals ( often called errors ).

In other cases, where you either have a complicated distribution ( maybe the model is non-linear and results in a more complex likelihood), then maximizing the likelihood is not equivalent to minimizing the residual sums of squares. So, you may still choose to maximize the likelihood but because the likelihood is complex, one often needs to turn to a numerical optimization algorithms such as gradient descent, BFGS.

So, I think your confusion is due to the fact that it ( maybe unfortunately ) happens to be the case that maximizing the likelihood of the normal is equivalent to minimizing the residual sums of squares. 

Then there are other cases maximizing the likelihood is too difficult or the likelihood is not even derivable in closed form.  In this case, people define other objective functions ( heuristics ) that still measure how "well" the model fits the data. Doing that, allows one to use the genetic, particle swarm type optimization schemes that you refer to. These schemes are often called non-parameteric because these objective functions are not tied to a parametric likelihood ( such as the normal,t, gamma etc ). In fact, the likelihood is often not even known.

Basically, what type of optimization one uses depends on whether you have a pdf and how complicated it is. 

A reallly rough categorization is as follows:

1) normal case: : straightforward: maximize it or minimize residuals squared.

2) non-normal case: likelihood is complex, maybe non-linear but known  : bfgs or gradient descent or some other numerical method that can handle the 
closed form version of the likelihood.

3) likelihood too complex or unknown or doesn't exist: define one's own objective function that measures the quality of the model and use algorithms such as particle swarm etc.

Note that in 2) above, you have to worry about whether the optimum you hit is local, global etc and this can be extremely tricky.
This may be the case in 3) also. I'm not as familar with genetic algs as the numerical methods.

I hope this helps some. I'm not sure of a good reference for what you asked about. Kenneth Lange has a book "numerical methods in statistics' that may be relevant but I don't have it. Good luck.