Histogram operation for improving visual object tracking I'm developping a people tracking with nearest neighbour data association technique and kalman filter for smoothing/predicting.

The tracker works quite good but sometimes it mismatch objects if they cross them-self.. (I think it is a common problem for a tracker).

I really believe that this tracker could be strongly improved with also histogram matching (I'm using OpenCV and I've seen an easy implementation of Bhattacharyya distance that seems quite popular in this field).

The fact is that I'd like to keep an average histogram of the tracked objects (because objects slightly changes their appearance if they rotate for example) and I'm a little bit confused on that: how to continuously adapt the average-histogram-object when new observation arrives?

observations could be different in size.. how can I handle this situation? using the same bins for each observation (even if they are 10 pixels or 100 pixels?) and sum up the histograms? using a num of bins proportional to the number of pixels?  and what about doing the average of two histograms?

I think I need all histograms with the same num of bins, and maybe use a weighted average where the weight is proportional to the num of pixels, but I'd like to ask for some more accurate advice from you!
I don't think you have any choice other than to use the same number of bins for each observation.  Otherwise not only will you not be able to average the histograms, you will also not be able to compare them.

And you definitely need to change the histogram slowly, i. e.
$$h = (1 - \alpha)h + \alpha h_{obs}$$ where $h$ is your "moving average" histogram, $h_{obs}$ is the histogram of the current observation, and $\alpha$ is a small factor, whose value you will have to tune experimentally.

As far as different sizes of observations, the simplest thing to do is to impose a hard constraint saying that the size of the object cannot change too much from frame to frame.