title: How many FLOPs does tanh need?
body: <p>I would like to compute how many flops each layer of LeNet-5 (<a href="http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf" rel="noreferrer">paper</a>) needs. Some papers give FLOPs for other architectures in total (<a href="https://arxiv.org/pdf/1512.06473.pdf" rel="noreferrer">1</a>, <a href="https://groups.google.com/forum/#!topic/caffe-users/cUD3IF5NMOk" rel="noreferrer">2</a>, <a href="https://arxiv.org/pdf/1511.06530.pdf" rel="noreferrer">3</a>) However, those papers don't give details on how to compute the number of FLOPs and I have no idea how many FLOPs are necessary for the non-linear activation functions. For example, how many FLOPs are necessary to calculate <a href="https://www.tensorflow.org/api_docs/python/nn/activation_functions_#tanh" rel="noreferrer"><code>tanh(x)</code></a>?</p>
  
  <p>I guess this will be implementation and probably also hardware-specific. However, I am mainly interested in getting an order of magnitude. Are we talking about 10 FLOPs? 100 FLOPs? 1000 FLOPs? So chose any architecture / implementation you want for your answer. (Although I'd appreciate answers which are close to "common" setups, like an Intel i5 / nvidia GPU / Tensorflow)</p>