Predict value from image set
Your initial idea seems about right. Before creating your own classifier you might want to try transfer learning, using some pretrained network like VGG16 that is incorporated in most of machine learning frameworks.

As to inference on mobile devices, TensorFlow offers some tutorials in this subject:
https://www.tensorflow.org/mobile/tflite/