Gradient descent parameter estimation Package for R
Ok, after a lot of looking I found the "optim" routine which is in "stats", one of the packages that is always loaded. It has quite a few methods including conjugate gradients and BGGS and a few others and worked well on the first few examples I tried. It doesn't seem to get a lot of attention strangely. I guess optimization people tend to use Matlab.

I knew there had to be something.