My network optimises for mean_squared_error, but the predictions are useless After going through the tutorials of tensorflow I decided to do my own project. For this I did the following:


I have scraped movie reviews from a polish site called filmweb. I got a dataset o 5400 movie reviews with a rating in range 1-10
I used the “Morfeusz” python bindings to tokenize and lemmatize the text. I have stripped most basic stop words.
I mapped most used 10000 lemmas to numbers and put the data in lists.
Randomly split the data into two equal sets.
My training data is the movie review lemmas padded with zeroes (like in the tutorial). The expected result is the rating.
My model is


    vocab_size = 10000

    model = keras.Sequential()
    model.add(keras.layers.Embedding(vocab_size, 16))
    model.add(keras.layers.GlobalAveragePooling1D())
    model.add(
        keras.layers.Dense(16, activation=tf.nn.relu)
    )
    model.add(keras.layers.Dense(1))

    optimizer = tf.keras.optimizers.RMSprop(0.001)


    model.compile(optimizer=optimizer, 
              loss='mean_squared_error',
              metrics=['accuracy'])


After running the optimization process i see the loss metric dropping quite fast. However the accuracy is well below 0.1 and after I check the predictions of this network, they are useless. The model predicts all the ratings to be in range of 6-7. So I understand that the network “optimized” by simply always guessing the mean value.

I wonder what should be my next step to improve. Is my model wrong for this task? Should I somehow normalize or augment my data? Or is the dataset size simply too small to get any meaningful result?

EDIT: this is the link to my dataset and the script that generated it:
https://www.dropbox.com/s/csyo934hhtbxxu9/fw-data.tar.bz2?dl=0
What you are doing right now is regression on the review score, therefore accuracy is not a suitable metric to assess performance. Your model is optimizing the mean squared error, thus it makes sense that it is predicting the (almost) average review score.

Since you are interested in the accuracy on the rating's prediction, you should set your output layer as a softmax layer with 10 possible outputs, as in:

model.add(keras.layers.Dense(10), activation='softmax')


Then you can optimize the categorical_crossentropy of your model, and report the accuracy. In this way, you'll do classification of the ratings.

model.compile(optimizer=optimizer, 
              loss='categorical_crossentropy',
              metrics=['accuracy'])