Estimate the Filter Coefficients of 1D Filtration (Convolution)
This is a nice question.
I will try solving it using 2 approaches (Which are basically the same).

The solution is the Least Squares Solution:

$$ \hat{h} = \arg \min_{h} \frac{1}{2} \left\| h \ast x - y \right\|_{2}^{2} $$

We assume data is given in finite discrete form (As it is in practice).
The convolution is done in valid mode (Like in MATLAB valid property).

Direct Solution

One could write the above as:

$$ \hat{h} = \arg \min_{h} \frac{1}{2} \left\| X h - y \right\|_{2}^{2}  $$

Basically using the commutative property of the convolution one could the above as $ x $ was the filter and hence build a Convolution Matrix from $ x $.

Now, the solution is the usual Least Squares:

$$ \hat{h} = \arg \min_{h} \frac{1}{2} \left\| X h - y \right\|_{2}^{2} = \left( {X}^{T} X \right)^{-1} {X}^{T} y $$

Pay attention that for long signals the matrix is huge.
Hence this method is practical only for small signals.

Iterative Solution

Well, one could use the Gradient Descent Method to minimize the cost function:

$$ f \left( h \right) = \frac{1}{2} \left\| h \ast x - y \right\|_{2}^{2} $$

The Gradient is given by (Correlation is the Adjoint Operator of Convolution):

$$ \frac{d}{d h} f \left( h \right) = \left( h \ast x - y \right) \star h $$

Where $ \star $ stands for Cross Correlation.

In MATLAB Code it is given by:

hObjFun = @(vH) 0.5 * mean( (conv2(vX, vH, 'valid') - vY) .^ 2 );
vG      = conv2(flip(vX, 1), (conv2(vX, vHEst, 'valid') - vY), 'valid');


Where all data is in Column Vector (The conv2() is used as it is faster since conv() is just a wrapper around it).

This method is fast and not limited by the size (Or less, to be accurate).



The full MATLAB Code is in my StackExchange Signal Processing Q18993 GitHub Repository.