How can I determine the mathematical relation between the input and output variables?
There are always a large number of possible functions that can produce a given set of input-output values. The challenge is to find a simplest function (according to whatever criteria you choose) to produce those values.

One approach is to write a general function of the input variables, comprising terms of all order in R, G, and B, with a coefficient for each term, then search for values of the coefficients that A) reproduce the known input-output values accurately and B) leave the largest number of coefficients equal to zero.  

Several different algorithms can be used to do the search efficiently.  My choice would be a genetic algorithm to seek the minimum of the RMS difference between the produced and known I-O values, summed with the product of a gradually increasing parameter and the number of nonzero coefficients.