Time Series Continuous Classification Intro

I'm quite new to all the subjects in this question. This is my very first shot at Keras, Tensorflow, NNs or Time Series. If you're not a newbie, you'll notice that immediately.

Problem

I have 15 samples from an industrial process that reads temperatures, weights and heatflows every 2 seconds for roughly an hour (1876 rows). Samples look like this:

      temp   weight     heat
t
0  31.9661  9.09606 -0.834733
2  31.9905  9.09503 -0.839906
4  32.0967  9.09374 -0.854042
6  32.3021  9.09315 -0.877126
8  32.5371  9.09390 -0.917037


Each of the 15 samples has been analysed by an expert that has produced QA, rating 5 of them as lousy (0), another 5 as medium (5) and the third lot as perfect (10). My goal is to analyse further samples and produce an equivalent assessment.

Statistical analysis on the data revealed that the only feature showing differences is heatflow, so I'm focusing on that column. The column name is calor (heat in Spanish). I also detected very synchronous activity around the tails but big discrepancies over the middle section, so I'm also focusing on that section (rows 1250 to 3250).

Solution

I'm reading the CSVs into a dictionary and then passing it to a Keras network with the following code:

sequences = [data[k][['calor']].loc[1250:3250].values for k in data.keys()]
final_seq = np.array(sequences)

targets = [5, 5, 5, 5, 5, 0, 0, 0, 0, 0, 10, 10, 10, 10, 10]
groups = [1, 1, 1, 2, 3, 1, 1, 1, 2, 3, 1, 1, 1, 2, 3]

train = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 1])
test = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 2])
validation = np.array([final_seq[i] for i in range(len(groups)) if groups[i] == 3])

train_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 1])
test_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 2])
validation_target = np.array([targets[i] for i in range(len(groups)) if groups[i] == 3])

# Building the model
model = Sequential()
model.add(LSTM(11, input_shape=train[0].shape))
model.add(Dense(11, input_shape=(11,), activation='softmax'))

adam = Adam(lr=0.001)

chk = ModelCheckpoint('best_model.pkl', monitor='val_accuracy', save_best_only=True, mode='max', verbose=1)

model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])

train_binary = to_categorical(train_target)
validation_binary = to_categorical(validation_target)
model.fit(train, train_binary, batch_size=len(train), epochs=100, callbacks=[chk], validation_data=(validation, validation_binary))

model = load_model('best_model.pkl')
test_preds = model.predict_classes(test)
print(f'test_target: {test_target}\ntest_preds: {test_preds}')
acc = accuracy_score(test_target, test_preds)
print(f'Accuracy score: {acc}')


I get different results, but usually around the following:

Model testing...
test_target: [ 5  0 10]
test_preds: [10  0 10]
Accuracy score: 0.6666666666666666


Accuracy of 66% is the best I ever got so far.

Questions

Am I doing this right?

I'm not sure about some hyperparameters. I kind of arbitrated 100 epochs, the batch size as the total sample, the categorical_crossentropy loss function and the groupings as per the above groups list because of the targets (which are fixed).

I'm also not sure about the LSTM thing. My categories are not only numeric but ordinal. I could actually have a regression result here.

Should I even be using NN for this? Basically, any input is welcome.

I'm going to need much more data, correct?

I have 5 samples each. How many more should I have?

Thank you very much, experts!
With only 5 samples per class you probably should not be using any Neural Network. If you can find a way to summarize your long time-series into a couple of numerical features, then you can try with a simple Logistic Regression model. Exploring the data is the key here, as well as understanding the problem domain. What is the physics of the process for example, or how would experts describe the requirements to get a particular score.

Even so I would try to collect some 10x more data.