I have to work on task of using hand keypoints as pointer (or touchless mouse).

The main problem here is that the (deep learning) hand keypoints are not perfect (sometime under varied of light, skin colors), therefore the chosen key point are scattering, not smoothly moving like the real mouse we use.

How can I smooth them online (in real-time). Not the solution given array of 2D points and then we smooth on this array. This is the case of new point get in one by one and we have to correct them immediately! to avoid user suffer the scattering mouse.

I'm using opencv and python. Please be nice since I'm new to Computer Vision.
Thanks