What algorithm to use for training combinations
Your current setup might not be the best one for this.  You need to think about the structure of the model, and the set of features you use.

Model

I would suggest a different approach: see if you can construct a model that captures how well a set of 5 colors match.  In particular, perhaps you can fit a model $f$, where $f(c_1,\dots,c_5)$ describes how well the five colors $c_1,\dots,c_5$ match each other (larger values of $f$ correspond to a better match).  For instance, one plausible model for $f$ is

$$f(c_1,\dots,c_5) = \prod_{i&lt;j} g(c_i,c_j)$$

where $g$ is some function (that you try to fit to the data).  Then, given a training set that describes many different combinations of five colors and how well they match, you could try to fit a model to that (e.g., find $g$ such that $f$ matches the training set as closely as possible).  You can then consider some architecture for $g$, e.g., a small neural network with a few hidden layers, and use gradient descent to train the model.

Features

Constructing a model based solely on the RGB values might be a lot to ask.  I suspect you can do a lot better by choosing some features that are likely to be relevant to color matching.

Often it helps to have some domain knowledge.  Do you know anything about color matching?  Are there any properties of colors that are relevant or informative or affect whether they match?

I would suggest that you pick multiple plausible features, then let the model training process work out which ones are most relevant.  Here are some ideas for you to consider:


Consider adding other color space as additional features.  I would suggest you consider adding the CMYK, HSV, and CIELAB.
It appears that primary colors are best matched with primary colors, and secondary colors best matched with secondary colors, etc.  Therefore, when determining how well two colors $c_i,c_j$ match, perhaps the difference between their hues (modulo 360 degrees) might be relevant.  You could add this as an additional feature that is input to $g$.


Maybe your study of the art of what colors match will suggest other features you can try as well.