title: Advanced Activation layers in Keras Functional API
body: <p>When setting up a Neural Network using Keras you can use either the <code>Sequential</code> model, or the <code>Functional API</code>. My understanding is the the former is easy to set up and manage, and operates as a linear stack of layers, and that the functional approach is useful for more complex architectures, particularly those which involve sharing the output of an internal layer. I personally like using the functional API for versatility, however, am having difficulties with advanced activation layers such as LeakyReLU. When using standard activations, in the sequential model one can write:</p>
  
  <pre><code>model = Sequential()
  model.add(Dense(32, activation='relu', input_dim=100))
  model.add(Dense(10, activation='softmax'))
  model.compile(optimizer='rmsprop',
            loss='categorical_crossentropy',
            metrics=['accuracy'])
  </code></pre>
  
  <p>Similarly in the functional API one can write the above as:</p>
  
  <pre><code>inpt = Input(shape = (100,))
  dense_1 = Dense(32, activation ='relu')(inpt)
  out = Dense(10, activation ='softmax')(dense_2)
  model = Model(inpt,out)
  model.compile(optimizer='rmsprop',
            loss='categorical_crossentropy',
            metrics=['accuracy'])
  </code></pre>
  
  <p>However, when using advanced activations like LeakyReLU and PReLU, in that sequential model we write them as separate layers. For example:</p>
  
  <pre><code>model = Sequential()
  model.add(Dense(32, input_dim=100))
  model.add(LeakyReLU(alpha=0.1))
  model.add(Dense(10, activation='softmax'))
  model.compile(optimizer='rmsprop',
            loss='categorical_crossentropy',
            metrics=['accuracy'])
  </code></pre>
  
  <p>Now, I'm assuming one does the equivalent in the functional API approach:</p>
  
  <pre><code>inpt = Input(shape = (100,))
  dense_1 = Dense(32)(inpt)
  LR = LeakyReLU(alpha=0.1)(dense_1)
  out = Dense(10, activation ='softmax')(LR)
  model = Model(inpt,out)
  model.compile(optimizer='rmsprop',
            loss='categorical_crossentropy',
            metrics=['accuracy'])
  </code></pre>
  
  <p>My questions are:</p>
  
  <ol>
  <li>Is this correct syntax in the functional approach?</li>
  <li>Why does Keras require a new layer for these advanced activation functions rather than allowing us to just replace <code>'relu'</code>?</li>
  <li>Is there something fundamentally different about creating a new layer for the activation function, rather than assigning it to an existing layer definition (as in the first examples where we wrote <code>'relu'</code>), as I realise you could always write your activation functions, including standard ones, as new layers, although have read that that should be avoided? </li>
  </ol>