Tracking using color
You can check the area of the same-color region. It should be close to the last one and it should be around some value (you can filter out very small and very big regions).
You can check rectangularity, circularity, convexity, angle, width, height depending on your object to be tracked.
If your false-detected regions are very small, you can try to a bit smooth the image first and then check for values.
Besides that if the false detected objects are very similar to the correct ones, your method seems to be the most feasible way to do it. You can consider making it faster by choosing different platform or optimizations.