Android application to retrieve car details from car plate numbers

Computer vision is a field in computer science that has had great success due to the increasing popularity of machine learning. Instead of having a human look at images and decide what they depict, we are able to teach computers to recognize patterns of previous images and see the resemblance in new images. In this paper we will get car all information from his number plates through an Android Application which use image preprocessing and YOLO (You only look once) technique to detect the plates and then optimal character recognition to read the plates and Arabic numbers of the cars in Iraq and we have trained it using darkflow. Every Car has its unique Licensed number plate which will be scanned and give the details about the car. Computer vision can also be used to read alphanumeric characters in images and turn them into text. We have implemented API to detect the text and number in the plates. The purpose of this project is to develop a system for our Government and Police So that If police need details about any car he would easily access it. We have achieved the accuracy of 92.23% which means out of 3500 number plates 3220 read correctly.


Introduction
Machine learning, neural networks and artificial intelligence are all concepts which have exploded in popularity the past few years. It allows computers to automatically analyze immense amounts of data and make decisions based on its patters [3]. This can be invaluable because the amounts of data used are often much too large for any human to analyze, comprehend and draw a conclusion from. It can be used almost anywhere, from self-driving cars to brewing the perfect pint of beer. The purpose of this project is to develop a system for our Government and Police So that If police need details about any car he would easily access it, the workflow consists of taking several pictures of the car in question and then opening a text editor to manually add the number of the license plate for the application to download information about it. The idea is to change the workflow so that information about the car is automatically gathered as part of the process of taking pictures. This would be done by creating a system where a computer is able to read the license plate directly off an image taken by the employees, and that is what our project consists of. The project outcome came very close to what we expected while planning it.   2 We have an Android application capable of detecting and cropping license plates as well as an OCR service that is able to read a surprisingly high number of license plates [6]. Our application's performance did not reach the levels we though were necessary before creating it, but as we tested it with a real phone we realized that our estimated requirements were too high. A high-level image of our system is provided in Figure 1.

Problem Statement
In the problem statement our project has had a lot of minor problems as well as a few major. An example of a major problem was when we spent several days trying to port our trained CNN to Android by using TensorFlow Lite, which is a lightweight implementation of TensorFlow specifically made for mobile and embedded systems. This was unsuccessful due to TensorFlow lite requiring another type of file than the one our model was created as and in the end we decided to use regular TensorFlow. Minor problems mostly consisted of being unable to install packages or programs and things not working when they were supposed to. Problems like these were usually solved by either checking the associated GitHub for similar errors or looking them up on Google [13].

Contribution
In the contribution of our work, we have divided our work into three parts, first we have created an application for a smartphone, an OCR service and a database. The application should be an Android application that is able to detect license plates, capture an image of said plates and send them to the second part which is the OCR service [8]. Our OCR service should be a standalone service that is able to read the text of the license plates it receives and send the text to the last part of our system, the database [2]. The database should then use the text received to fetch information about the car and send it back to the phone which requested information in the first place.

METHODS AND PROCEDURES
The computer ran Ubuntu 16.04 with the NVIDIA CUDA and cuDNN libraries installed to take advantage of the GPU. To set up our development environment we used Android studio in which the code was written in java and all commands displayed and were written in the terminal. Android studio was used due to how easy it is to manage virtual environments using the platform. A total of 13 virtual environments were used during a trial and error phase of setting up the most basic object detector before we found the perfect setup. They were used to test different techniques and packages as well as changing things between iterations without ruining our older setups. Every new environment created with android studio needs to restart the Kernel, and our final environment where SDK was install was created. The pipeline of our system will begin with using an application on a smartphone. This application should open up a camera with an embedded object detector that can detect license plates in real-time. This object detector is going to have two purposes. The main purpose should to be to localize the license plate for further usage in our pipeline. Its other purpose is to give the user an idea of how good the picture is going to be. This means that the bounding boxes should be displayed in the camera and not just work in the backend. The reason for this is that a good picture will make is easier for an OCR service to read it. The object detection used in our application should be a CNN trained by us, using data provided by our employer and the YOLO technique. Since it is going to be used in a phone and not a computer our computational resources will be far more limited, and as such YOLO is required to give us the best possible performance. A frame rate of at least 5-10 frames per second is what we believe necessary for the application to feel usable. When a user is certain the picture will be good, the user should take a photo. This will start the cropping process. The location of the license plate and the image should be sent to a backend service in the same application that crops the picture and in turn sends it to a server hosting of our pipeline, the OCR service.

Training
Before training our CNN we had to prepare data to train with. As planned, we received large amounts of data from our employer. The final dataset consisted of about 3500 pictures of vehicles where the license plate was visible to varying degrees. Since Darkflow uses supervised learning for its CNN, all pictures had to be labeled. What this means in object detection is that for every picture in the dataset, a square has to be manually drawn over the object you wish to detect then labeled with what it is representing, as shown in Figure 3. The final task before starting a training session is to download a configuration file and pre-trained weights. The configuration file is a text file specifying exactly how the CNN will be built. It determines how many convolutional layers will exist as well as tuning for each layer. The configuration file also has to be modified to fit the amount of objects the object detector should be able to detect, in our case one. Pre-trained weights are an already trained CNN which we built upon. This is common practice when training a CNN because pre-trained weights are usually trained on huge datasets where the CNN is allowed to learn low level features like edges and corners. The model is then used and trained to recognize specific objects as in our case, license plates. During our training we always used the same pre-trained weights, the Tiny YOLO weights found at the official page for Darknet. To begin training we have shown our database detail in  The model option specifies which configuration file to use. Tiny-yolo-voc-graph.cfg was our modified and final configuration file. Load determines which, if any, already trained CNN to use. Since this was our first run for a new CNN we used the pre-trained weights. In subsequent runs for the same CNN, the load command was used to load the CNN we had trained. Train tells Darkflow to start training Dataset should be used. Dataset and annotations are the file paths to our datasets, allowing Darkflow to find the images and annotations we created before. GPU 0.75 makes Darkflow use GPU instead of CPU and allows it to use 75% of the available VRAM.

RESULTS
The results are based on data the network has been trained with. It was also made sure that there was no overlap in images of Licensed Number plates that were recorded multiple times. Five images were chosen randomly from each of the two sources in the very beginning. We developed our system into two major parts an object detector and an OCR service. The object detector was ported to an Android application created by TensorFlow, and the application was in turn modified to suit our own needs. Our OCR service was developed as a standalone service that receives images taken with our application and reads them. We evaluate what we have created based on our application's performance on different Android devices, accuracy of our object detector as well as both accuracy and performance of our OCR service. We also measure the impact of our image preprocessing. The figure 5 shows the Inspection results.

Fig. 5: Screenshot of inspection results
Performance was a big issue for us from the beginning and one of the key points during the development of our object detector. Using real-time object detection on a phone is not a new concept, but it is demanding even on the hardware of modern smartphones. When using a Samsung S10 the phone was able to complete the process of taking a snapshot, performing the YOLO algorithm and draw a box if any license plate was found in about 0.5 seconds. This results in an update frequency of two times per seconds. While discussing the project before implementation we agreed that a performance of at least 5-10 updates per second was necessary for the application to feel smooth enough. This was proven false when we got a chance to test it live. Two updates per second is not a lot, but the camera preview that ran at full speed made application feel surprisingly smooth. Part of the reason for this is because the application is supposed to be used on cars standing still in a parking lot. Holding the camera still means that the box drawn on the screen does not lag behind as it would when be rotating since the box is drawn at specific coordinates on the screen. When testing it we pointed the camera at a plate, waited a moment for the object detection to calibrate and pressed the camera button. Getting a green box (90% confidence) was very easy and usually happened as soon as the camera was pointing directly at the rear or front of a car. The performance of then taking a photo and sending it was almost negligible. When a box is shown on the camera preview, a cropped version of the license plate is already stored. This means that pressing the camera button immediately ends the activity while the image is sent to our OCR service on another thread without user interference. Since a Samsung S10 is a high-end smartphone we deemed it was necessary to test the application on worse performing devices. We were able to get two more smartphones to test on, a Samsung S8 as well as a Samsung S5 Mini. When using the Samsung S8 our performance was only slightly worse than on a Samsung S10 with about 0.6 seconds spent on each iteration of the YOLO algorithm, which still felt acceptable. The Samsung S5 Mini on the other hand had a performance of five seconds for each iteration. While functional, we consider the application to be too demanding for such a low-end device. For a more detailed comparison of the different devices see Table 2. This table shows that using TensorFlow on an Android is all about the CPU. Extra memory, at least over 4 GB, does not seem to make any difference. To evaluate our final object detector, we took the testing images discussed and used the object detector on them. These pictures were not taken with OCR in mind and it is important to note that they are only used to evaluate the quality of the object detector. We wrote a Android studio script that would use the YOLO algorithm with our model. The figure 6 shows the user interface of our application, which is shown below.

Fig. 6: Screenshot of user interface
On each image in our test set and count every time it found a license plate with at least 50% confidence, which we consider to be the minimum for a successful detection. From a total of 147 images of varying quality and angle where the license plate was shown, 138 plates were found. This gave us an accuracy of about 92% on the type of images it was trained on, which is a very acceptable number. To gather images with the purpose of later testing our OCR service we used the same script but on our original dataset of 3500 images and changed the confidence criteria to 90%. The script was also edited so that any detected license plate fulfilling the criteria was cropped and stored. This was to gather images that would be similar to those taken with the application while the box is green. 1026 images were found and used and evaluate our OCR service. Do note, however, that these images do not prove the object detectors quality as these are the same images as it was trained on, resulting in bias towards them. ake for the OCR service the next time, the next time after that, and so on.
We ran some tests on a set of 342 images, of varying qualities. Here we are evaluating how big of an impact the tweaking of the threshold in binarizing an image actually has and how to regulate it to work as optimal as possible. The optimal state would be hitting either the best binarized image for the OCR service to read as fast as possible, or if this is not possible, reach the all-white image as fast as possible. This, because these two cases are our break points. Maybe we would be better off completely without the reiterations of the linearization.
The number we will tweak with is the value of which the binarization method uses as threshold. We will use a variable that will change according to the attempt (iteration) we are in. And remember, the following will occur if the first adaptive binarization will not succeed and we have to start changing the threshold manually, see Figure 7. Example: we get back a calculated threshold value of 150 from the first iteration (see Figure 7 (threshold value)), then the threshold is 150 in the next iteration. The equation will look like so: 150 + N × x, where N is the current attempt and x is the multiplier. If x = 2, then, every N: th iteration, will look like so: 150 + N × 2. Remember that for every positive value, there is a negative one that will also be applied, and so with this ret becomes: -2, 2, -4, 4, -6, 6..., and will give the threshold 148, 152, 146, 154, 144 Fig. 7: Flowchart of the binarization process We set the value of x to 2, 10, 40 and 100 in the four tests. We are looking for a case where we get a high accuracy but still will not have to wait for as long for the slower images. The slowest image is the image in the set of 342 images that takes the longest to go through the OCR service, see Table 3. This image will almost certainly be one that the OCR software was not able to read; the image had to go through all iterations without any luck, hence the long time. This is the image that the person using the app will have to wait for in worst case. We feel that this was an absolute necessity to keep in mind, because we would not want to wait for 18 seconds (see Table 3) for the OCR service to tell us that it could not find a car with the plate you sent to it. On the other hand, we got quite a good accuracy with x = 2, see Table 3. 18 seconds we deemed not to be worth waiting for, even though this is rare. Therefore, having the multiplier set to 2 was not something to stick with. In Table 3 we can also see that by having x set to 100, we manage to keep iterations down and therefore the average time per image. This however is jeopardizing the precision by a lot for the OCR service. The number are getting a little better when lowering x to 40, where we can see that we still manage to keep the speed up, even when accuracy is raising. Giving x the value of ten, we can see that the numbers are starting to show good progress. The accuracy is really close to the one with multiplier set to two and the average attempt is quite low as well.

CONCLUSION
For the duration of 16 weeks we have gone from neither of us ever touching the field of machine learning to developing an application that is able to detect license plates as well as an associated OCR service that is able to read the plates. Since neither of us had any prior experience with neural networks when we began, our first couple of weeks were spent researching the subject. We read theory about neural networks as well as how it is used practically. Several different tools and techniques were tested during our research phase, walking down the wrong path many times before stumbling upon our final frameworks. The project has had a lot of minor problems as well as a few major. As we went into this project with little to no knowledge of neither the software nor the theory behind ML or the associations to it, there was no thinking ahead of things. Where there was a small knowledge of the language Android studio, there was none in how to apply it, together with computer vision, on images. This lead to the search for it on the internet. Information was found, but not necessarily the right one, let alone the better one. The first major problem was not knowing about the configurability of the OCR software. This lead too many hours of work trying to find an optimal state of the image to be in with regards to resolution and the size of the actual characters on the plate. This problem was later solved discovering how the OCR software could be tweaked to read an image in different ways. Some examples of the configuration are: reading the image, assuming that it was a sentence to be read, treat it as if it was a single word or in our case, a single uniform block of text. After having spent many hours on optimizing the size of the image and the characters with little success and results at the bottom, this discovery help us a lot. Our way of attacking the problem was now miles different from that of before. The major problem was found by trying to efficiently identify the plate in the image. As earlier mentioned, the probability that the program will have to iterate is not high. This is due to the discovery of the HSV model. Before we had one way of working with the identification of the plate, and that was through adaptive binarization. With no HSV colors ranges to work with we could not tell the program to let 2nd International Scientific Conference of Al-Ayen University (ISCAU-2020) IOP Conf. Series: Materials Science and Engineering 928 (2020) 032026 IOP Publishing doi:10.1088/1757-899X/928/3/032026 7 through the necessary ranges of light and not determine what colors should be accepted, full trust was given to the adaptive binarization. With this trust, it was not uncommon that the program let though the country identifier (the left most part of the plate), making the OCR software trying to read that part as a character. Upon finding the HSV colors, we could much more accurately highlight the actual plate. Moreover, before we could utilize HSV-even if there are not many of them-green plates were nearly impossible to read.