how to measure RBG values of a color using phototransistor
The visible light spectrum ranges from 380 nm (blue) to 700 nm (red).

The human eye has three types of cones: one is most sensitive to wavelengths around 450 nm, and is called S (for short-wavelength) or blue; the second is most sensitive to wavelengths around 540 nm, and is called M (for medium-wavelength) or green; and the third is most sensitive to wavelengths around 570 nm, and is called L (for long-wavelength) or red, although it is actually most sensitive to yellowish-green rather than red.



I don't think it makes much difference whether you use a photo-diode or photo-transistor, since they are made up of the same material (silicon).  The more important hing to do is get an accurate plot of the sensor you are using, re its response to various visible light frequencies.

A typical photo-diode response, although sensitive to visible light, has its peak response in the middle of the near-infrared region (900 nm), and is not at all flat in the visible spectrum (eight times more sensitive to red than blue):



It is possible to get photo-diodes that more closely mimic the human eye response.  One such diode is the BPW21R.  It's response is broader than the overall human eye response, but it peaks at the same place (approximately 550 nm).



You need to find what wavelengths your R, G, and B filter papers are passing, and then match them up against the color response plot corresponding to your photo-diode or transistor.  Then you can normalize the outputs (either by boosting or attenuating the signal) of each of the three filters to match human eye response.