How should continuous action/gesture recognition be performed differently than isolated action recognition
This question actually includes many. I try to answer a couple.

First of all, you need to make sure for your use case to know the non-functional requirements. It is helpful to know that you have a soft real-time case (if harm is done in case of too late predictions it might also be hard real-time)


Latency: how much time may pass after the action was made until the prediction is there?
Stability of prediction: how fast it's the system allowed to change it's prediction? Once every 0.5s?
General hardware, especially memory usage
Optimization metric: maybe accuracy does not tell you enough?
Evaluation setup: likely, you need to have multiple "evaluation points" per video, e.g. Once after the system had one second, once after it had 1.5,... You will also want to consider the stability of the prediction.


Now to the question about how you can do that. If you didn't try it so far, I recommend looking at optical flow.