Group neural networks outputs using Keras/Tensorflow I am trying to group the outputs of my neural network, in order to have them perform a separated classification.

Let's take the example where the groups are constituted of two nodes and we previously have four output nodes, then the network should look like this : 



How can I achieve this using Keras or Tensorflow ? I was thinking that I need to implement a custom layer and do the operation inside of it, but I wonder if there is an easier solution using the Keras Functional API (or lower level Tensorflow ?) :)
Using the keras functional API will get you what you need.

I'm assuming that you are currently using the standard keras sequential model API, which is simpler but also restricts you to a single pipeline. When using the functional API, you do need to keep track of inputs and outputs, instead of just defining layers.

For the example in your question:

from keras.layers import Input, Dense
from keras.models import Model

# Left side sub-model:
L1 = Input(shape=(2,))
L2 = Dense(2, activation='softmax')(L1)

# Right side sub-model:
R1 = Input(shape=(2,))
R2 = Dense(2, activation='softmax')(R1)

# Combining them together:
merge = concatenate([L2, R2])

# Some additional layers working on the combined layer:
merged_layer_1 = Dense(4, activation='relu')(merge)

# Output Layer:
output = Dense(2, activation='softmax')(hidden1)

# Defining the model:
my_model = Model(inputs=[L1, R1], outputs=output)


The depths of the layers are made up, but you should get the general idea.