How to encode data with a feature having multidimensional features (colors)?
One-hot encoding will give you a sparse matrix, Try LabelEncoding, before converting them into RGB that is.

Also, you could try breaking the RGB values into three features (R, G, B) and try that approach as well.

Hope this helps.