Is it reliable to use TensorFlow (ML in general) to classify baggage bag tags based on the presence of a green stripe?
The scientific answer would be, it depends. 

In case you are using any kind of Deep net, then 40 images is far too little. 
It might be helpful to describe your problem setting a little bit more in depth.
Are the bags always in the same place, or do they need to be localized first? These kind of details could help other users in their recommendations.

As a first approach, before you try a deep net or any kind of ML I would try a simple baseline first. Do you know what the exact pixel value of your green stripe is? You could then simply check whether this colour is present at all. This is rather coarse, but I would see how far this gets you and it is good to see whether your ML methods can beat this simple baseline. 
Subsequently you could also think of trying to localize the bagtags (in whatever way you like) then cropping it and checking for the presence of this green stripe.