What optimization algorithms are used in scikit-learn?
There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:


LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients
Lasso and ElasticNet use coordinate descent
OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients
ARDRegression and BayesianRidge use something like EM algorithm
SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.
HuberRegressor uses BFGS (a second-order optimization method)