Color transform algorithm (that also works on saturation and value/lightness)?
Very often, the Lab color space is used, since it is close to how humans perceive color. As to which distance to apply: This wikipedia article mentions a Delta E distance of 1.0 for a just noticeable difference. There are a lot of different formulas resulting in different thresholds.