Are the HOG features of the images good enough to do image classification alone? Why are their variances low?
Definitely not sufficient to detect;

First HoG is not scale invariant - you change resize your image and handle this problem-. 

Second HoG is not rotation invariant - you should include vary poses of action but this makes your solution more complex-

In my opinion; you should choose another kind of feature descriptors which can handle rotation and scale invariance. (SIFT, SURF, ORB ...)

Maybe you can use spatio-temporal information for this problem - It will be very useful-

Thanks.

And one more suggestion is about Neural Networks. If you have enough dataset for your problem, you can use "Neural Networks" which can be more roboust to other kind of feature based methods.