how to get the RGB value of a color using phototransistor or photodiode?
To understand the results you are getting you have to consider the spectral response curves of your light source, detector, and the object being measured. The final result is a combination of all these 'filters' acting on the signal. And if you want to relate that to the colors you see, then you also have to take into account the spectral response of the human eye.  


  I'm using a Photoshop print on a matte photo print paper as the color


A photographic print is calibrated to show correct colors when viewed by the human eye. This does not mean that it will reflect all the color components in direct proportion to the values in the digital image. The pigments or dyes used in color printers don't produce pure colors. A perfect 'green' ink (which is actually a combination of cyan and yellow) would reflect all wavelengths between 495 and 570nm ('green') while absorbing all other wavelengths in the visible spectrum ('blue' and 'red'). But practical inks do not have such sharp response, so 'green' ink won't reflect all green light, and may also have some blue and red in it.

To compensate for the color being too dark the printer may lay down less ink to let more of the white paper show through. But traces of blue and red in the 'green' could make it look muddy or washed out, which the printer may compensate for by adding black. The printer may also alter color ratios to get a more accurate hue. The end result looks alright, but is not an exact match to the digital RGB value of 0,255,0. What you are seeing is not actually pure green, but a mixture of many different colors that just look like pure green to the human eye.

Another factor to consider is the spectral output of your LEDs. Most green LEDs emit a fairly narrow band of green light, If this doesn't line up with the ink's spectral response then the output could be lower than expected. If you then calibrate to this lower value and call it '255', a pure white image could read higher than expected because the plain white paper reflects more wavelengths of green light than the 'green' ink does.

Finally, the detector's response could be distorting the results. Standard photo-transistors peak in the near infra-red, and response drops off rapidly towards the blue end of the spectrum. This could skew the response so that 'yellow' green produces a higher reading than 'blue' green.  

Bottom line:- the printer modifies the image's RGB values to produce a result that looks right to the human eye, so to get a meaningful result your emitter/detector combination must replicate the spectral response of the human eye. Even then, the results won't exactly match the RGB color values in the digital image (1%? forget about it!).