Machine Learning is very much in vogue at the moment and there are many Android folks producing some excellent guides to creating and training models. However, I do not feel that in the vast majority of cases Android Developers will be required to train their own models. But that does not mean that using a model that someone else has created is a straightforward task. In this series, we’ll take a look at some of the problems and pain points that I encountered while attempting to incorporate a pre-trained TensorFlow Lite model in to an Android app.
Previously we examined our trained MNIST number recognition model to understand its inputs and outputs which is an absolute requirement before we can implement that model within our app.
I mentioned in the previous article that one of my must-haves was to use MLKit rather than directly invoking the TensorFlow Lite Android runtime. Although MLKit uses this internally, it offers one huge benefit: You can store your TensorFlow Lite model in the cloud and this will get automatically downloaded to the client.
Although it’s not a bad idea to bundle a version of the model in with your APK, it’s really not a good idea to only use that mechanism to distribute the model. The reason for this is if you need to update the model, then it requires a new APK and app update to be made each time the model changes. That can be painful if you have a separate team responsible for updating the model and they wish to make new releases which are not in sync with the app development team’s planned release schedule.
It therefore makes more sense to periodically check a server and download a new model whenever it changes. This will allow the model team to be able to update the model independently of app releases. The only requirement here is that they can refine the internals of the model, and re-train an existing model, but they cannot change the input & output formats – changing these will need to be tied in with a new version of the app which works with the new input & output formats.
While it would not be too difficult to host the model file on a server and poll periodically from the app and download a new model whenever it changes, MLKit offers this functionality for free, and is the reason that I decided that using MLKit was an absolute requirement.
The init block is mainly setting the options for how we obtain the model. We create a couple of conditions sets which will control the circumstances under which we’ll download the model. The initial conditions control how the model will be downloaded the initially, and have little to no restrictions. We want the user to have the model initially. However we are more reserved with the updates conditions which control how updated models will be downloaded. For these we require the device to be charging (so we don’t waste battery), the device to be idle (so we don’t cause the device to slow down while in use), and we require wifi (to avoid using the user’s data).
If we were bundling a copy of the model in the APK, then I would be a bit stricter with the initial conditions, but the app will be pretty useless without a model, so I’ve take this approach.
Once we have defined these, we build and register a FirebaseCloudModelSource. Finally we create the FirebaseModelOptions which we’ll use to obtain Interpreter instances.
The only other thing that we need to do is ensure that we have declared the INTERNET permission in our Manifest. With these relatively few lines of code we’ll get our model downloaded from the cloud and it will periodically check for updated models and download them as necessary.
I should mention that the project source code does not contain the google-services.json file that the Firebase Assistant added to my project. This is because it contains details specific to my Firebase account, so anyone wanting to use the sample code will need to use their own.
I’m not going to give a full explanation of this as the focus of this series is implementing an ML model, but the key thing is that when it receives an ACTION_UP event because the user lifts their finger from the screen, as well as drawing to the screen, we also draw the path to a bitmap and pass this to a drawing listener. The bitmap is actually a 28×28 pixel bitmap to match the training data for our MNIST model, and it is important to match our analysis data format to the training data format. It also makes the data much faster to analyse and categorise if it is relatively small.
Here we have the drawingListener implementation which will pass the bitmap to a NumberClassifier instance which is the ML model implementation. In the final article in this series we’ll take a look at that class and cover a real pain point which I experienced in trying to get it working.
Although we don’t have fully working digit detection app yet, the source code we have so far is available here.