Using CMOS to detect transmittance in sample from LEDs
You won't be able to separate the light frequencies with just an optical sensor no matter how much processing you do on the output. 

From your question it appears you want to build an optical spectrometer. The below is one way: 



There are also methods involving micromirror (DLP) technology. 

The idea is to optically spread out the light frequencies into a spatial distribution (as with a prism or diffraction grating) and then sense the light intensity at different points.