Tracking a multicolor object
For a full proof track you need to combine more than one method...following are some of the hints...

if you have prior knowledge of the object then you can use template matching...but template matching is little process intensive...if you are using GPU then you might have some benefit
from your write up i presume external light varies to a lesser extent...so on that ground you can use goodfeaturestotrack function of opencv and use optical flow to track only those ponits found by goodfeaturestotrack in the next frames of the video
if the background is stable except some brightness variation and the object is moving comparatively more than background then you can subtract the previous frame from the present frame to get the position of the moving object...this is kind of fast and easy change detection technique...
Filtering contours based on area is good idea but try to add some more features to the filtering criteria...i.e. you can try filtering based on ellpticity,aspect ratio of the bounding box etc...
lastly...if you have any prior knowledge about the motion path of the object you can use kalman filter...
if the background is all most not variant or to some extent variant then you can try gaussian mixture model to model the background...while the changing ball is your fore ground...
I want to track a multicolored object(4 colors). Currently, I am parsing the image into HSV and applying multiple color filter ranges on the camera feed and finally adding up filtered images. Then I filter the contours based on area. 
This method is quite stable most of the time but when the external light varies a bit,  the object is not recognized as the hue values are getting messed up and it is getting difficult to track the object. 
Also since I am filtering the contours based on area I often have false positives and the object is not being tracked properly sometimes.
Do you have any suggestion for getting rid of these problems. Could I use some other method to track it instead of filtering individually on colors and then adding up the images and searching for contours?