Activation function vs If else statement
Activation functions in general cannot be implemented with if-else statements.

ReLU is a particularly simple activation function, and can thus easily be implemented with an if-else block. For example (in Python):

def relu(x):
    return x if x &gt; 0 else 0


But this only works because ReLU is linear for $ x \geq 0$.

What about sigmoidal activation functions?



There's no if/else implementation for this function that's analogous to that of ReLU.  In fact, this is a one-liner with no if/else involved:

def approximate_logistic(x):
    return 1 / (1 + 2.71828**-x)


Same goes for lots of other activation functions like Softplus, CELU, Tanh.

If/else blocks simply aren't powerful enough to capture the wide variety of activation functions that we might want to use.