Apply Low Pass (Smoothing) Filter on a Set of Matrices and Reject Outliers
The method used was a parametric representation of the matrix $ H $.

For intuition, thing of rotation matrix in $ \mathbb{R}^{2} $, namely $ H \in \mathbb{R}^{2 \times 2} $.
Since we have series of those (In case of video, a series in time) we can define $ H \left( t \right) $.

As a rotation matrix it can be parameterized by a single parameter $ \theta $ which is the rotation value. Since we are dealing with a time series then we have $ \theta \left( t \right) $.

Now, this is just a simple Time Series and regular methods of smoothing and outlier rejection can be applied.

Another idea would be to find the best base to approximate all matrices (Which with respect to that base the can be diagonalized) and then work on the "Singular / EigenValues".