How to implement this CNN architecture in Keras
In this kind of situation, you can not use sequential API which is generally used in the architecture where you to stack layers on each other.

For this kind of problem use the functional API of keras. Based on references image tried to create architecture &amp; as the author has used RESNET architecture in his paper. So, I have tweaked network according to it rather than replicating architecture give in the image.

input_img = keras.Input(shape = (224, 224, 3))

x1 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(input_img)
x1 = keras.layers.BatchNormalization(axis = 3)(x1)
x1 = keras.layers.Activation('relu')(x1)

x2 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(x1)
x2 = keras.layers.BatchNormalization(axis = 3)(x2)
x2 = keras.layers.Activation('relu')(x2)
x2 = keras.layers.Dropout(0.2)(x2)
x2 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(x2)

merge_x2 = keras.layers.Add()([x1, x2])
x3 = keras.layers.BatchNormalization(axis = 3)(merge_x2)
x3 = keras.layers.Activation('relu')(x3)
x3 = keras.layers.Dropout(0.2)(x3)
x3 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(x3)
x3 = keras.layers.BatchNormalization(axis = 3)(x3)
x3 = keras.layers.Activation('relu')(x3)
x3 = keras.layers.Dropout(0.2)(x3)
x3 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(x3)

conv_merge_x2 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(merge_x2)
merge_x3 = keras.layers.Add()([conv_merge_x2, x3])
x4 = keras.layers.BatchNormalization(axis = 3)(merge_x3)
x4 = keras.layers.Activation('relu')(x4)
x4 = keras.layers.Flatten()(x4)
x4 = keras.layers.Dense(2)(x4)
final = keras.layers.Activation('softmax')(x4)

model = keras.models.Model(input_img, final)
model.compile(loss = "categorical_crossentropy", optimizer = "adam", metrics = ["accuracy"])