automatic differentiation in deep learning an example of this is when your loss function involves an if-else clause or for/while loop in sensesymbolic differentiation is differentiating (closed formmathematical expressionit is not differentiating given computational procedure another problem with symbolic differentiation is that naive application of symbol rewriting rulesin some casescan lead to an explosion of symbolic terms (expression swelland make the process computationally unfeasible typicallya fair amount of compute effort is required to simplify such expressions and produce closed form expression of the derivative note symbolic differentiation is implemented in python package called sympy we do not cover it hereas it is not directly relevant to deep learning automatic differentiation fundamentals the first key intuition behind automatic differentiation is that all functions of interest (which we intend to differentiatecan be expressed as compositions of elementary functions for which corresponding derivative functions are known composite functions thus can be differentiated by applying the chain rule for derivatives this intuition is also at the basis of symbolic differentiation the second key intuition behind automatic differentiation is that rather than storing and manipulating intermediate symbolic forms of derivatives of primitive functionswe can simply evaluate them (for specific set of input valuesand thus address the issue of expression swell because intermediate symbolic forms are being evaluatedwe do not have the burden of simplifying the expression note that this prevents us from