How to find the perfect regularization number? I have an issue!

Assume that we are going to solve $Ax=b$ but $x$ contains noise.To minimize that noise we can use regularization:

$$x = (A^TA + \lambda I)^{-1}A^Tb$$

Where $\lambda$ is a small number $&gt; 0$, or sometimes $\lambda$ can be huge!

Here are three pictures of one situation where I solve $Ax=b$ with noise free and one $Ax=b$ with noise and without regularization and one with noise + regularization.

Noise free case: Here we can see that it works smooth. In this case, $x$ is a matrix because it contains two signals.



Here we are using 5% noise, that will cause much trouble. No regularization.


Here we are using 5% noise and a large number for regularization.



Question:

Is there a way to determine an optimized regularization parameter $\lambda$?

Source:
I made a library named Mataveid and it's made for system identification.
I have no problem with the noise actually, but I want to find a way to automize the choice of regularization parameter $\lambda$.
Approaches can be any of the following:


Model the noise that you expect, for ex: gaussian (least squares in Minimum variance unbiased estimator for a linear signal model in presence of gaussian noise). Based on this model try and estimate the noise variance, the regularization term should be close to noise varainace.
Deploy machine learning techniques based on signal parameters such as dynamic range, frqeuency, FFT noise floor etc. to estimate noise
Follow a recursive approach with adjusting the regularization to minimize a suitable objective, ex: gradient descent etc.


In general your regularization is as good as how close it is to the actual noise or estimation of noise. This goes back to the idea of whether a frequentist approach is better or Bayesian approach. The answers is if the modelling of the parameter is accurate to it's actual behaviour then the Bayesian approach is better but if you model a parameter with a distribution but it is actually not following that closely then Bayesian would lead to suboptimal result and better go for a frequentist approach.