Why does the activation function degrade the time performance of the CNN model?
I'm testing the CNN model time and accuracy performance with some different small changes. In this model some activation layers between convolutions are skipped. So I added them. I noticed that model with added ELU activations has

performance 3.07 ms per image and 92.26% accuracy on test set

And model without them has

performance 3.52 ms per image and 92.34% accuracy on test set

Although the number of model parameters is the same. So my question is why does activations decrease model's time performance so much?
Here for instance part of code:
...
nn.Conv2d(28 * filters_multiplier, 28 * filters_multiplier, kernel_size=(3, 1), padding=(1, 0)),
nn.Conv2d(28 * filters_multiplier, 28 * filters_multiplier, kernel_size=(1, 3), padding=(0, 1)),

# nn.ELU(), 

nn.Conv2d(28 * filters_multiplier, 28 * filters_multiplier, kernel_size=(3, 1), padding=(1, 0)),
nn.Conv2d(28 * filters_multiplier, 40 * filters_multiplier, kernel_size=(1, 3), padding=(0, 1)),
...

So code with commented block works faster than not commented.
UPD I just tested with ReLU() function, and it works as fast as first model with no activations. So maybe problem just in ELU()?

Neither ReLU nor ELU have learnable parameters, but they still require compute to execute.
ELU executes the exponential function on all x &gt;= 0. This is computational expensive and so your network is slower.
ReLU is a computational cheap because of the ease of the operation x[x &lt; 0] = 0 and so you don't really see a spike in time. This is one of the reasons why ReLU is a common choice as activation function.
