Classification model using RNN(action detection)
@CapJS,

Instead of going ahead with an RNN, which helps you model the dependencies and relationship between your content, I would suggest you to take a look at 1D convolution networks to achieve the classification of activity. 

There is a nice post talking about something similar : https://machinelearningmastery.com/cnn-models-for-human-activity-recognition-time-series-classification/ 

With 1D convolution, you are by design gaining better processing performance and given that you have just 124 samples per activity, you do not need a deep layered network as the data size is still quite small. 

Given the data size, I would also suggest you to go ahead with something as simple as logistic regression, random forest approach. Hope this helps.