How to get predictions with predict_generator on streaming test data in Keras?
To get a confusion matrix from the test data you should go througt two steps:


Make predictions for the test data


For example, use model.predict_generator to predict the first 2000 probabilities from the test generator. 

generator = datagen.flow_from_directory(
        'data/test',
        target_size=(150, 150),
        batch_size=16,
        class_mode=None,  # only data, no labels
        shuffle=False)  # keep data in same order as labels

probabilities = model.predict_generator(generator, 2000)



Compute the confusion matrix based on the label predictions


For example, compare the probabilities with the case that there are 1000 cats and 1000 dogs respectively. 

from sklearn.metrics import confusion_matrix

y_true = np.array([0] * 1000 + [1] * 1000)
y_pred = probabilities &gt; 0.5

confusion_matrix(y_true, y_pred)




Additional note on test and validation data

The Keras documentation uses three different sets of data: training data, validation data and test data. Training data is used to optimize the model parameters. The validation data is used to make choices about the meta-parameters, e.g. the number of epochs. After optimizing a model with optimal meta-parameters the test data is used to get a fair estimate of the model performance.