Microcontroller or arduino alternative for small cell-phone sized device?
When I am working on a new design, once I know what I want it to do I normally start by writing out the basic code, and drawing out a basic schematic. This way I have a good idea of what the microcontroller needs to be able to do. Once you know what parts you are using, you can finish the schematic then the software. And make sure you double check the datasheet for each component, paying attention to size/package, abilities, power and heat requirements!


  I am using TCS230 as my optical sensor.I need to read from only 15mm2
  at a distance of 1cm matte surface.I propose to use a concave lens to
  focus.Is this setup good or is there a better sensor I can use?


I skimmed the datasheet and didn't see anything that makes me think this wont work. You may have to play around with the lens/lenses a bit to get the ratio correct but I think that should work.

If you're not familiar with this ic, make sure you buy the prototype board, not just the chip because it is very small and hard to work with if you don't have an etched pcb.


  How hard is it for me to use a microcontroller. I need to complete this
  in 2 months. Is there any other option for me? Main concern is time arduino I 
  know and can prog quickly but I do not know avr yet.


I'm not a fan of the Arduino's language, it's not too precise and Avr-GCC is easy to get the hang of. You don't need to use assembly. That being said, since this is a project that has a deadline, stick to what you know. You can write it out in the Arduino Framework then if you have time you can port it over to Avr-GCC, it may even help you get the hang of it quicker. 

I think you can program most all the Atmega line with the Arduino Framework, So you can make a custom device and still use the langueage you know.


  Can I reduce LCD pins to fit mini? What LCD can I use? Which
  microcontroller should I use?


If the LCD is just displaying text you can use a mono LCD controlled via serial i2c. This would allow you to use only 2 pins for the LCD, so your total will be 14 pins for everything. This would allow you to use something like the Arduino Pro Mini, as long as it has enough serial lines for everything.

Just google Mono LCD, and see what looks good to you. Make sure it is controllable via serial, and doesn't have any high voltage requirement.


  What type of battery to use Li-ion? How should I decide on
  batterygiven size considerations?


Build the device then you can measure the power draw and figure out how long you want it to run, etc. Then you can pick a battery based on that and the final size of the project. This won't be hard to find something that works.