Transfer function and deconvolution
How can I justify the expression of G′(ν)?


That's easy enough. Denominator is the sum of the signal energy $|X(\omega)|^2$ and the noise energy $\lambda ^2$. If the signal energy is significantly larger, then the whole expression simplifies to $G(\omega)$. If the noise is larger, we can't do a anything useful with the information and the $1/ \lambda ^2$ term will drive the whole expression towards zero and some continuous way. 


  Is there a good reference to apprehend this class of problem?


Deconvolution does work fine here. Convolution is commutative, so it doesn't really matter if you are trying to calculate the impulse response or the input signal (at least in your case). 

There are a few other methods that can also work well which are typically based on a least square error optimization in either the time or the frequency domain. 

Any of these methods can be greatly improved by using any a-priori knowledge of the system. For example, if you that it's a relatively well behaved physical system with a some known degrees of freedom, you can simply do a least square error fit trying to determine a small number of poles and zeros. 

Each application has it's own specific requirement and the best method depends a lot on what you have and what exactly you need.


  How is called the oscillating effect at the ends of the signal?


Not sure. I think your problem here may be that your noise estimate is not a function of frequency and that somehow a single or a few discrete frequencies get overal amplified in the error signal.