Background averaging not working
I would try techniques related to low-rank approximations of your data $X$. For instance, the paper GoDec: Randomized Low-rank &amp; Sparse Matrix Decomposition in Noisy Case attempts to decompose
$$X = L + S + G$$
with a low-rank part ($\mathrm{rank}(L) \le r$), a sparse part ($\mathrm{card}(S) \le k$) and a residual $G$. 

It has been employed for background modeling, with backgrounds related to the low-rank part, plus sparse en independent objects. 



Some video demos are:


GoDec and Robust PCA for background modeling on Escalator
GoDec and Robust PCA for background modeling on Hall


There are alternatives in the field of robust matrix factorization, and I understand it as an instance of sparse coding.