How to get predictions with predict_generator on streaming test data in Keras? In the Keras blog on training convnets from scratch, the code shows only the network running on training and validation data. What about test data? Is the validation data the same as test data (I think not). If there was a separate test folder on similar lines as the train and validation folders, how do we get a confusion matrix for the test data. I know that we have to use scikit learn or some other package to do this, but how do I get something along the lines of class wise probabilities for test data? I am hoping to use this for the confusion matrix.
To get a confusion matrix from the test data you should go througt two steps:


Make predictions for the test data


For example, use model.predict_generator to predict the first 2000 probabilities from the test generator. 

generator = datagen.flow_from_directory(
        'data/test',
        target_size=(150, 150),
        batch_size=16,
        class_mode=None,  # only data, no labels
        shuffle=False)  # keep data in same order as labels

probabilities = model.predict_generator(generator, 2000)



Compute the confusion matrix based on the label predictions


For example, compare the probabilities with the case that there are 1000 cats and 1000 dogs respectively. 

from sklearn.metrics import confusion_matrix

y_true = np.array([0] * 1000 + [1] * 1000)
y_pred = probabilities &gt; 0.5

confusion_matrix(y_true, y_pred)




Additional note on test and validation data

The Keras documentation uses three different sets of data: training data, validation data and test data. Training data is used to optimize the model parameters. The validation data is used to make choices about the meta-parameters, e.g. the number of epochs. After optimizing a model with optimal meta-parameters the test data is used to get a fair estimate of the model performance.