How to create custom Activation functions in Keras / TensorFlow?
First you need to define a function using backend functions. As an example, here is how I implemented the swish activation function:

from keras import backend as K

def swish(x, beta=1.0):
    return x * K.sigmoid(beta * x)


This allows you to add the activation function to your model like this:

model.add(Conv2D(64, (3, 3)))
model.add(Activation(swish))


If you want to use a string as an alias for your custom function you will have to register the custom object with Keras. It can be done like this:

from keras.utils.generic_utils import get_custom_objects

get_custom_objects().update({'swish': Activation(swish)})


This allows you to add the activation directly to layer by name:

model.add(Conv2D(64, (3, 3), activation='swish'))


For more advanced activation functions, with trainable parameters and such, it is best to implement them as a Keras Layer. Here the swish function is used in a layer, allowing beta to be learned while training:

from keras.layers import Layer

class Swish(Layer):

    def __init__(self, beta=1.0, trainable=False, **kwargs):
        super(Swish, self).__init__(**kwargs)
        self.supports_masking = True
        self.beta = beta
        self.trainable = trainable

    def build(self, input_shape):
        self.beta_factor = K.variable(self.beta,
                                      dtype=K.floatx(),
                                      name='beta_factor')
        if self.trainable:
            self._trainable_weights.append(self.beta_factor)

        super(Swish, self).build(input_shape)

    def call(self, inputs, mask=None):
        return swish(inputs, self.beta_factor)

    def get_config(self):
        config = {'beta': self.get_weights()[0] if self.trainable else self.beta,
                  'trainable': self.trainable}
        base_config = super(Swish, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

    def compute_output_shape(self, input_shape):
        return input_shape


You then would add the activation function the same as any other layer:

model.add(Conv2D(64, (3, 3)))
model.add(Swish(beta=1.0, trainable=True))