Tracking a multicolor object
You might try having multiple or an infinite number of models of the object depending upon the light sources available, and then classifying your object as either the object with one of the light sources or not the object.  Note: this is a machine learning-type approach to the problem.
Filtering with a Kalman, extended Kalman filter, or particle filter (depending on your application) would be a good idea, so that you can have a "memory" of the recently tracked features and have expectations for the next tracked color/feature in the near term (i.e. if you just saw the object, there is a high likelihood that it hasn't disappeared in the next frame).
In general, this is a difficult problem that I have run into a few times doing robotics research.  The only robust solution is to learn models and to confirm or deny them with what your system actually sees.  Any number of machine learning approaches should work, but the easiest would probably be support vector machines.  The most robust would probably be something like Gaussian Processes (if you want to do an infinite number of models).  Good luck and don't get too frustrated; this is not an easy problem!
I want to track a multicolored object(4 colors). Currently, I am parsing the image into HSV and applying multiple color filter ranges on the camera feed and finally adding up filtered images. Then I filter the contours based on area. 
This method is quite stable most of the time but when the external light varies a bit,  the object is not recognized as the hue values are getting messed up and it is getting difficult to track the object. 
Also since I am filtering the contours based on area I often have false positives and the object is not being tracked properly sometimes.
Do you have any suggestion for getting rid of these problems. Could I use some other method to track it instead of filtering individually on colors and then adding up the images and searching for contours?