automatic differentiation in deep learning is technique that helps in achieving for estimating very high order integration using only few series of values the approximation errors for forward and backward differences are in the order of hthat iso( )--whereas those for central difference and richardson extrapolation are ( and ( )respectively the key problems with numerical differentiation are the computational costswhich grow with the number of parameters in the loss functionthe truncation errorsand the round off errors the truncation error is the inaccuracy we have in the computation of '(xdue to not being zero the round off error is inherent to using floatingpoint numbers and floating-point arithmetic (as opposed to using infinite precision numberswhich would be prohibitively expensivenumerical differentiation is thus not feasible approach for computing gradients while building deep learning models the only place where numerical differentiation comes in handy is quickly checking whether gradients are being computed correctly this is highly recommended when you have computed gradients manually or with new/unknown automatic differentiation library ideallythis check should be put in as an automated check/assertion before starting sgd note numerical differentiation is implemented in python package called scipy we do not cover it hereas it is not directly relevant to deep learning