keras plotting loss and MSE Can someone give me a tip on how I could incorporate MSE &amp; loss plots? I have been following some machinelearningmastery posts to plot this but the application is classification and I am attempting regression. Also what is different in my script is I am defining the model thru calling a function, so I am curious if my script could be re-written without the function def wider_model() that defines the model.

This script below works except what is commented out on the bottom for the plt plots. In the machinelearningmastery post, someone does ask this question how to do for regression, and supposedly if you print print(history.history.keys()) two values are returned for dict_keys([‘mean_absolute_error’, ‘loss’])...  

Any tips help, there isn't a lot of wisdom here... Thank you

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import math



# load dataset
dataset = pandas.read_csv("prepdSPdata.csv", index_col='Date', parse_dates=True)

# shuffle dataset
dataset = dataset.sample(frac=1.0)


# split into input (X) and output (Y) variables
X = numpy.array(dataset.drop(['Demand'],1))
Y = numpy.array(dataset['Demand'])

print(dataset.shape)
print(dataset.dtypes)
print(dataset.columns)


def wider_model():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=11, kernel_initializer='normal', activation='relu'))
    model.add(Dense(10, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model


# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=1, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print("RMSE", math.sqrt(results.std()))



# list all data in history
#print(wider_model.wider_model.keys())

# summarize history for MSE
#plt.plot(history.history['acc'])
#plt.plot(history.history['val_acc'])
#plt.title('model MSE')
#plt.ylabel('MSE')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()

# summarize history for loss
#plt.plot(history.history['loss'])
#plt.plot(history.history['val_loss'])
#plt.title('model loss')
#plt.ylabel('loss')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()
cross_val_score does not return the history of the training. You can use fit instead: 

history = model.fit( ...


See this example.

As you mentioned, the history object holds the results of the training for each epoch.

Here is the relevant bit:

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
pyplot.plot(history.history['mean_squared_error'])
pyplot.plot(history.history['mean_absolute_error'])
pyplot.plot(history.history['mean_absolute_percentage_error'])
pyplot.plot(history.history['cosine_proximity'])
pyplot.show()


I was actually working on the same example that you referenced yesterday. I think it is hard to understand because it introduces many functions and concepts: estimators, StandardScaler, KerasRegressor, Pipeline, KFold and cross_val_score.

However, I did like the approach to creating and testing models, and Cross Validation would produce more robust models.