Classification of Colors using Machine Learning from an Array of Pixels
This is a classical problem for a convolutional neural network. Typically, these work with series of layers using 2d-convolution to find features (e.g., "curved" or "yellow"), then pooling (to borrow from adjacent convolutions), then some combination.

I think the main question is how you are planning to get a good, labeled, dataset (that is, one where a banana is already labeled as "banana", for training your NN).