Simple neural network implementation in keras I have a binary classification problem(benign/malicious) and I have applied simple neural network with one hidden layer for solving the problem. I have 46 features in my dataset and for the hidden layer I am using 46/2. Also my data is not scaled so for it I am using standardscaler. I am getting accuracy of around 99.79% with the code. However the accuracy was unexpected, I was expecting around 93-94%, I am afraid that I am leaking some data or making some silly mistake.

def create_baseline():
    model = Sequential()
    model.add(Dense(23, input_dim=46, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_baseline, epochs=1000, batch_size=len(X), validation_split=0.15, verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
scores = cross_val_score(pipeline, X, y, cv=kfold)
print("%.2f%%" % (scores[1]*100))


I am using around 5871 malicious samples and 3488 benign samples.

Just to cross check I have implemented without cross validation, a very simple implementation, and I am getting ~99% accuracy in that also. 

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)

scale = StandardScaler(with_mean=0, with_std=1)
new_X_train = scale.fit_transform(X_train)
new_X_test = scale.transform(X_test)

model = Sequential()

model.add(Dense(23, input_dim=46, kernel_initializer="normal"))
model.add(PReLU(alpha_initializer='zero', weights=None))

model.add(Dense(1, kernel_initializer='normal'))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(new_X_train, y_train, epochs=1000, batch_size=len(X_train), validation_split=0.15)

scores = model.evaluate(new_X_test, y_test)
print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

print(history.history.keys())

plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()


The plots I am getting are:
Considering the nature of your problem (disease classification?), probably you have imbalanced data: more negative/benign w.r.t. positive/malicious. In such cases accuracy is not a good metric to assess the quality of a model.

Consider, as example, a dataset with 99 "negative" cases and only one "positive" case, and consider a classifier which predict "negative" all the times. The accuracy of such model is, actually, 99%:



What you want to do in order to deal with unbalanced data is:

1) just use another metric, as ROC-AUC. This can be sufficient if you have a dataset not so much unbalanced, say up to 1:20. This metric is not officially supported in Keras, but you can find a custom callback which calculate the AUC score every end of epoch here.

2) if the imbalance is very big, say 1:1000, you need to balance the dataset. There are lots of techniques, the simplest one is to make copies of the minority class samples, as many as needed to reach the balance. Other smarter techniques exist, and some of then are included in imbalanced-learn for SKLearn.

If you want to have a better overview of techniques and metrics used in such cases, I suggest you this paper.