Methods for fast image comparison
The problem might not be caused by your loss function. Deep learning models tend to be computationally demanding. Mobile devices are not usually prepared for handling models with high throughput. That being said, you might try to:


Prepare smaller model - less layers, less computation
Mobile models optimization - Google provides some materials on optimizing Tensorflow models for mobile inference:

https://www.tensorflow.org/mobile/prepare_models
https://www.tensorflow.org/mobile/optimizing