A Neural Network That Learns Bitwise XOR I am trying to build a deep neural network that learns the coordinate-coordinate bitwise XOR of two matrices, but it performs poorly. 

For example, in the 2 bits case, its accuracy stays around 0.5. Here is the code snippet:

from keras.layers import Dense, Activation
from keras.layers import Input
import numpy as np 
from keras.layers.merge import concatenate
from keras.models import Model


size=1
data1 = np.random.choice([0, 1], size=(50000,size,size))
data2 = np.random.choice([0, 1], size=(50000,size,size))
labels  = np.bitwise_xor(data1, data2)
a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(1, activation='sigmoid')(merged)
hidden = Dense(3, activation='sigmoid')(hidden)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(4, activation='sigmoid')(hidden)
hidden = Dense(3, activation='sigmoid')(hidden)
outputs = Dense(1, activation='relu')(hidden)

model = Model(inputs=[a, b], outputs=outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit([data1, data2], np.array(labels), epochs=15, batch_size=32)


What's going on here?

Epoch 1/15
50000/50000 [==============================] - 7s 130us/step - loss: 0.7118 - acc: 0.5044
Epoch 2/15
50000/50000 [==============================] - 4s 78us/step - loss: 0.6933 - acc: 0.5023
Epoch 3/15
50000/50000 [==============================] - 4s 74us/step - loss: 0.6934 - acc: 0.5030
Epoch 4/15
50000/50000 [==============================] - 4s 86us/step - loss: 0.6935 - acc: 0.5002
Epoch 5/15
50000/50000 [==============================] - 4s 79us/step - loss: 0.6934 - acc: 0.5015
Epoch 6/15
50000/50000 [==============================] - 5s 96us/step - loss: 0.6935 - acc: 0.5030
Epoch 7/15
50000/50000 [==============================] - 5s 105us/step - loss: 0.6934 - acc: 0.5026
I think there might be a few things going on.

You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size).

Also, you might have a reason but I don't know why you ran each feature separately through a different layer (each with a single hidden unit), and then merged the outputs before running the merged output through another series of layers:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])


Also, I suspect that running the features through a single hidden unit reduces the information sent through the rest of the network, so the network cannot learn the XOR function.

Here is some code that works for me:

from keras import models

from keras.layers import Dense

import numpy as np


Simulate data:

X_1 = np.random.choice([0, 1], size = (50000, 1))
X_2 = np.random.choice([0, 1], size = (50000, 1))

X = np.concatenate((X_1, X_2), axis = 1)

Y = np.bitwise_xor(X[:, 0], X[:, 1])


FNN Model:

# Define model.

network_fnn = models.Sequential()
network_fnn.add(Dense(4, activation = 'relu', input_shape = (X.shape[1],)))
network_fnn.add(Dense(4, activation = 'relu'))
network_fnn.add(Dense(1, activation = 'sigmoid'))

# Compile model.

network_fnn.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics = ['acc'])

# Fit model.

history_fnn = network_fnn.fit(X, Y, epochs = 5, batch_size = 32, verbose = True)