title: Implementing automatic differentiation for 2nd derivative: algorithm for traversing the computational graph?
body: <p>I am attempting to implement <a href="http://en.wikipedia.org/wiki/Automatic_differentiation" rel="nofollow noreferrer">automatic differentiation</a> for a Python statistics package (the problem formulation is similar to optimization problem formulations).  </p>
  
  <p>The computational graph is generated using operator overloading and factory functions for operations like sum(), exp(), etc. I have implemented automatic differentiation for the gradient using reverse accumulation. However, I have found implementing automatic differentiation for the second derivative (the Hessian) much more difficult. I know how to do the individual 2nd partial gradient calculations, but I have had trouble coming up with an intelligent way to traverse the graph and do the accumulations. Does anyone know of good articles that give algorithms for automatic differentiation for the second derivative or open source libraries that implement the same that I may try to learn from? </p>