Build the Seefood app (not hotdog) from Silicon Valley using Ionic, Firebase, and Google Cloud Vision. We go from zero to a deep learning vision app in about 10 minutes. https://angularfirebase.com/lessons/google-cloud-vision-with-ionic-and-firebase/

- Cloud Vision: https://cloud.google.com/vision/
- Ionic: https://ionicframework.com/docs/
- Cloud Functions: https://firebase.google.com/docs/functions/
- AngularFire2 Storage: https://github.com/angular/angularfire2/blob/master/docs/storage/storage.md
- Silicon Valley: https://youtu.be/ACmydtFDTGs
2024-02-04T07:47:27.192Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
ac9362f6-4f01-4d83-9227-2b0f6f6ab192
0a623912d89d2cd2f0a04f8d293b3223d4393bbc7dd7f3483472808761754b4f
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
Today, I'm going to take you from 0 to a deep learning powered mobile app with Ionic, Firebase, and Google Cloud Vision.
You might have seen this app before.
It's ready.
Please, god.
Just do pizza.
Yeah.
Pizza.
Hey, Zach.
Not hot dog?
Wait.
What the f Not only will our app be able to detect hot dogs, but it can Also, label any image using Google's deep learning infrastructure.
And it's actually a lot more sophisticated than you might think.
As you can see here, if we upload a hotdog, it correctly He labels it as such, but what happens if we try to trick it by sending it a picture of a pug in a hot dog costume?
It's smart enough to know that that's not actual hot dog, and labels it instead as a pug.
If you like building cool apps, make sure to subscribe and follow along with the source code at angularfirebase.com.
Let's take a closer look at how this works.
The user will upload an image from our Ionic app to Firebase Storage.
That's going to invoke a cloud function, which will send the image to the Google Vision API.
It does its deep learning magic and converts the image into a set of labels.
We save those labels in Firestore so they can be read in our front end Ionic app in real time.
To get started with this project, You'll first want to start a new Ionic app using the blank template, then CD into your project and run Firebase init functions.
This will associate your project with Firebase and create a folder called functions in the root of the project.
And make sure to select TypeScript when prompted for that option.
After that, you're going to want to CD into the functions directory, and then we're going to install the Google Cloud Vision library.
Cloud Vision is a library that takes an image as its input and returns a whole bunch of data extracted from that image.
It exposes a high level API to access deep learning algorithms that have been trained on millions of images.
It can also do more advanced things like facial recognition and text extraction, which I'm going to cover in a video for pro members.
Our next step for this app is to go into the storage console and create a brand new bucket for the Cloud Vision function.
The reason we do this is because Firebase functions can't be scoped to specific file locations, So it's best to have a dedicated bucket specifically for invoking the function.
After that, go into the rules tab and make sure to enable read and write access for this bucket.
As you may know, every Firebase project is also a Google Cloud Platform project.
Cloud Vision is not enabled by default, So you need to go into the Cloud Platform console, to the APIs tab, then find Cloud Vision and enable it.
Before we build the actual mobile app in Ionic, I want to build out the Cloud Function just so you know how everything works on the back end.
So go into your index ts file in the functions directory, then import Firebase functions as well as the admin SDK.
Because we have Cloud Vision enabled on the Cloud Platform console, we can just import it and then instantiate a new Vision client.
There's no need to add API keys to the environment or anything like that.
Then we need to set a variable for the name of the storage bucket.
Otherwise, Firebase is going to try and use our default bucket.
From there, I'll export a constant for our actual function which I'm calling image tagger.
It's going to be a storage function that points to our bucket name.
Then we'll call object on change and return an async function which just ensures that We return a promise from this operation.
Now inside the function code, I'll set up a couple variables here for the actual file data and the file name, which is going to be important later on because the file name needs to correspond to the document ID in the Firestore database.
The image URI is the actual stored file location, which is going to be used by Cloud Vision to read the actual raw file.
So we'll upload the file from Ionic, But for right now, all you need to know is that that file path is going to have the same ID as its data in Firestore.
So we'll just split off the JPEG from it and then use it to make a reference to the actual document in Firestore, which is in the photos collection.
At this point, we can do all of the deep learning magic with a single line of code.
We can await the results from our vision client by calling label detection and passing it that image URI.
It responds with a big object of data, but we don't need everything there.
So we're just going to map it down to the label descriptions that we wanna show the user on the front end.
We also wanna know if there's a hotdog in this image.
So to do that, we can just look at this array of strings and use the fancy new includes method to see if there is a hotdog string in this array, which will return true or false.
Now there's only one step left, and that is to save this data in Firestore.
We already have a document reference, so we can just say doc ref set with the corresponding data.
Now you can go ahead and deploy this function and try it out.
Go into Firebase Storage and upload an image that you think might be a hot dog but just aren't 100% sure.
Then jump over to the Firestore database and go into the photos collection, and you should see a document that looks something like this.
We have a hot dog boolean value and then an array of a bunch of strings that label this image.
Our back end is working perfectly.
Let's CD back into Ionic and generate a new page called Vision.
Then we have a couple of dependencies that we need to install on our project.
The first one is the native camera Cordova plugin which will give us access to a camera on a native mobile device.
Then we'll also install AngularFire There too to make Firebase more friendly inside of an Angular environment.
Follow the install instructions for both packages, and then open your app module file.
For AngularFire, you can see we're importing Firestore as well as the storage module.
Import the camera here as well, And then make sure to add your Firebase configuration to this configuration object.
The Firebase modules are going to go in the import section.
The Vision page goes in entry components, and the camera goes in the provider's array.
After that, we just need to go into the app component.
And instead of using the homepage as the root page, we're going to swap that out for the Vision page.
Now, we can switch over to that Vision page and write all of the front end code.
First, we'll bring in the loading controller from Ionic and also a few things here from RXJS and Angular Firestore and Fire Storage and, of course, our Ionic camera.
Then we'll go ahead and set up a few variables here in the page.
The first one is the upload task.
Then we'll declare an observable for the actual results that we get back from Firestore.
We can also set up variables for the loading indicator and the image itself to show those in the front end as well.
From there, we need to add all of our dependencies to the constructor.
Again, those include Firestore, storage, the camera, and the loading controller.
We can also instantiate the loading object in the constructor, which we do by calling loading controller create, and you can customize this as well.
At this point, we need to define a method that will start the actual upload process.
It takes the file as an argument, and then it will present the loading spinner once it's triggered.
Then we'll generate a random ID for our document and our file name.
So the actual file name is going to be the document ID followed by JPEG.
And then we'll use that ID to make a reference to Firestore.
Again, it's important that these two values match because that's how we keep track of the file in storage and its location in the Firestore database.
Then we'll make a reference to it in Firestore even though that document doesn't technically exist at this point.
We call value changes to get it as an observable, And then it's going to emit null initially, so we want to filter that value out.
When we do get a value, we want to dismiss the loading indicator so the user can see the results in the front end.
For that, we can use the tap operator and then call loading, dismiss.
Now we're ready to handle the actual file.
The camera is going to return it in a base 64 format, which is just a big long encrypted string.
But Firebase needs it in a specific format, So all we have to do is append it to this string here.
Then we can just make a reference to that storage location, which, again, should match the document ID.
And then instead of calling upload, we call put string with the actual image itself.
The only missing piece at this point is how do we get the actual image from the camera.
For that part, I'm going to set up an async function because the camera returns the image as a promise, and then it's important that the destination type is a data URL.
Then because this is an async function, we can await the operation which is going to return the actual base 64 string.
So we await the camera to get the picture with our configuration options.
And then after that, we'll go ahead and start the upload.
Now we just have a couple things to set up in the page template.
1st, we wanna give the user a button that they can tap that will start the actual process to capture the image.
After that, I set up an ng if statement so we can unwrap the results observable.
And based on that result, we should know if the image is a hotdog or not.
So if it is, then we'll go ahead and display a green button.
If it's not, we'll display a red button.
Then we also have an array of labels in that result, So we can go ahead and loop over them, and we'll display an Ionic chip for each one.
At this point, we should be good to go.
If we run either an iOS or Android emulator, We should be able to get the Cloud Vision API results in our actual front end Ionic app.
I'll go ahead and run the Android emulator with the Pixel 2, and we'll see how everything turned out.
I've saved a couple images on this emulator, and we'll go ahead and select the hot dog.
And you can see it uploads to storage, And then after a couple seconds or so, we get the results back, and, of course, this is hot dog.
If this video helped you, please like and subscribe.
And if you're serious about building apps, consider becoming a pro member at angularfirebase.com.
You'll get access to all kinds of exclusive content designed to help you build and ship your app faster.
Thanks for watching, and I'll see you soon.

Speaker 0: Today, I'm going to take you from 0 to a deep learning powered mobile app with Ionic, Firebase, and Google Cloud Vision. You might have seen this app before. It's ready. Please, god. Just do pizza.

Yeah. Pizza. Hey, Zach. Not hot dog? Wait.

What the f Not only will our app be able to detect hot dogs, but it can Also, label any image using Google's deep learning infrastructure. And it's actually a lot more sophisticated than you might think. As you can see here, if we upload a hotdog, it correctly He labels it as such, but what happens if we try to trick it by sending it a picture of a pug in a hot dog costume? It's smart enough to know that that's not actual hot dog, and labels it instead as a pug. If you like building cool apps, make sure to subscribe and follow along with the source code at angularfirebase.com.

Let's take a closer look at how this works. The user will upload an image from our Ionic app to Firebase Storage. That's going to invoke a cloud function, which will send the image to the Google Vision API. It does its deep learning magic and converts the image into a set of labels. We save those labels in Firestore so they can be read in our front end Ionic app in real time.

To get started with this project, You'll first want to start a new Ionic app using the blank template, then CD into your project and run Firebase init functions. This will associate your project with Firebase and create a folder called functions in the root of the project. And make sure to select TypeScript when prompted for that option. After that, you're going to want to CD into the functions directory, and then we're going to install the Google Cloud Vision library. Cloud Vision is a library that takes an image as its input and returns a whole bunch of data extracted from that image.

It exposes a high level API to access deep learning algorithms that have been trained on millions of images. It can also do more advanced things like facial recognition and text extraction, which I'm going to cover in a video for pro members. Our next step for this app is to go into the storage console and create a brand new bucket for the Cloud Vision function. The reason we do this is because Firebase functions can't be scoped to specific file locations, So it's best to have a dedicated bucket specifically for invoking the function. After that, go into the rules tab and make sure to enable read and write access for this bucket.

As you may know, every Firebase project is also a Google Cloud Platform project. Cloud Vision is not enabled by default, So you need to go into the Cloud Platform console, to the APIs tab, then find Cloud Vision and enable it. Before we build the actual mobile app in Ionic, I want to build out the Cloud Function just so you know how everything works on the back end. So go into your index ts file in the functions directory, then import Firebase functions as well as the admin SDK. Because we have Cloud Vision enabled on the Cloud Platform console, we can just import it and then instantiate a new Vision client.

There's no need to add API keys to the environment or anything like that. Then we need to set a variable for the name of the storage bucket. Otherwise, Firebase is going to try and use our default bucket. From there, I'll export a constant for our actual function which I'm calling image tagger. It's going to be a storage function that points to our bucket name.

Then we'll call object on change and return an async function which just ensures that We return a promise from this operation. Now inside the function code, I'll set up a couple variables here for the actual file data and the file name, which is going to be important later on because the file name needs to correspond to the document ID in the Firestore database. The image URI is the actual stored file location, which is going to be used by Cloud Vision to read the actual raw file. So we'll upload the file from Ionic, But for right now, all you need to know is that that file path is going to have the same ID as its data in Firestore. So we'll just split off the JPEG from it and then use it to make a reference to the actual document in Firestore, which is in the photos collection.

At this point, we can do all of the deep learning magic with a single line of code. We can await the results from our vision client by calling label detection and passing it that image URI. It responds with a big object of data, but we don't need everything there. So we're just going to map it down to the label descriptions that we wanna show the user on the front end. We also wanna know if there's a hotdog in this image.

So to do that, we can just look at this array of strings and use the fancy new includes method to see if there is a hotdog string in this array, which will return true or false. Now there's only one step left, and that is to save this data in Firestore. We already have a document reference, so we can just say doc ref set with the corresponding data. Now you can go ahead and deploy this function and try it out. Go into Firebase Storage and upload an image that you think might be a hot dog but just aren't 100% sure.

Then jump over to the Firestore database and go into the photos collection, and you should see a document that looks something like this. We have a hot dog boolean value and then an array of a bunch of strings that label this image. Our back end is working perfectly. Let's CD back into Ionic and generate a new page called Vision. Then we have a couple of dependencies that we need to install on our project.

The first one is the native camera Cordova plugin which will give us access to a camera on a native mobile device. Then we'll also install AngularFire There too to make Firebase more friendly inside of an Angular environment. Follow the install instructions for both packages, and then open your app module file. For AngularFire, you can see we're importing Firestore as well as the storage module. Import the camera here as well, And then make sure to add your Firebase configuration to this configuration object.

The Firebase modules are going to go in the import section. The Vision page goes in entry components, and the camera goes in the provider's array. After that, we just need to go into the app component. And instead of using the homepage as the root page, we're going to swap that out for the Vision page. Now, we can switch over to that Vision page and write all of the front end code.

First, we'll bring in the loading controller from Ionic and also a few things here from RXJS and Angular Firestore and Fire Storage and, of course, our Ionic camera. Then we'll go ahead and set up a few variables here in the page. The first one is the upload task. Then we'll declare an observable for the actual results that we get back from Firestore. We can also set up variables for the loading indicator and the image itself to show those in the front end as well.

From there, we need to add all of our dependencies to the constructor. Again, those include Firestore, storage, the camera, and the loading controller. We can also instantiate the loading object in the constructor, which we do by calling loading controller create, and you can customize this as well. At this point, we need to define a method that will start the actual upload process. It takes the file as an argument, and then it will present the loading spinner once it's triggered.

Then we'll generate a random ID for our document and our file name. So the actual file name is going to be the document ID followed by JPEG. And then we'll use that ID to make a reference to Firestore. Again, it's important that these two values match because that's how we keep track of the file in storage and its location in the Firestore database. Then we'll make a reference to it in Firestore even though that document doesn't technically exist at this point.

We call value changes to get it as an observable, And then it's going to emit null initially, so we want to filter that value out. When we do get a value, we want to dismiss the loading indicator so the user can see the results in the front end. For that, we can use the tap operator and then call loading, dismiss. Now we're ready to handle the actual file. The camera is going to return it in a base 64 format, which is just a big long encrypted string.

But Firebase needs it in a specific format, So all we have to do is append it to this string here. Then we can just make a reference to that storage location, which, again, should match the document ID. And then instead of calling upload, we call put string with the actual image itself. The only missing piece at this point is how do we get the actual image from the camera. For that part, I'm going to set up an async function because the camera returns the image as a promise, and then it's important that the destination type is a data URL.

Then because this is an async function, we can await the operation which is going to return the actual base 64 string. So we await the camera to get the picture with our configuration options. And then after that, we'll go ahead and start the upload. Now we just have a couple things to set up in the page template. 1st, we wanna give the user a button that they can tap that will start the actual process to capture the image.

After that, I set up an ng if statement so we can unwrap the results observable. And based on that result, we should know if the image is a hotdog or not. So if it is, then we'll go ahead and display a green button. If it's not, we'll display a red button. Then we also have an array of labels in that result, So we can go ahead and loop over them, and we'll display an Ionic chip for each one.

At this point, we should be good to go. If we run either an iOS or Android emulator, We should be able to get the Cloud Vision API results in our actual front end Ionic app. I'll go ahead and run the Android emulator with the Pixel 2, and we'll see how everything turned out. I've saved a couple images on this emulator, and we'll go ahead and select the hot dog. And you can see it uploads to storage, And then after a couple seconds or so, we get the results back, and, of course, this is hot dog.

If this video helped you, please like and subscribe. And if you're serious about building apps, consider becoming a pro member at angularfirebase.com. You'll get access to all kinds of exclusive content designed to help you build and ship your app faster. Thanks for watching, and I'll see you soon.
Today, I'm going to take you from 0 to a deep learning powered mobile app with Ionic, Firebase, and Google Cloud Vision. You might have seen this app before. It's ready. Please, god. Just do pizza. Yeah. Pizza. Hey, Zach. Not hot dog? Wait. What the f Not only will our app be able to detect hot dogs, but it can Also, label any image using Google's deep learning infrastructure. And it's actually a lot more sophisticated than you might think. As you can see here, if we upload a hotdog, it correctly He labels it as such, but what happens if we try to trick it by sending it a picture of a pug in a hot dog costume? It's smart enough to know that that's not actual hot dog, and labels it instead as a pug. If you like building cool apps, make sure to subscribe and follow along with the source code at angularfirebase.com. Let's take a closer look at how this works. The user will upload an image from our Ionic app to Firebase Storage. That's going to invoke a cloud function, which will send the image to the Google Vision API. It does its deep learning magic and converts the image into a set of labels. We save those labels in Firestore so they can be read in our front end Ionic app in real time. To get started with this project, You'll first want to start a new Ionic app using the blank template, then CD into your project and run Firebase init functions. This will associate your project with Firebase and create a folder called functions in the root of the project. And make sure to select TypeScript when prompted for that option. After that, you're going to want to CD into the functions directory, and then we're going to install the Google Cloud Vision library. Cloud Vision is a library that takes an image as its input and returns a whole bunch of data extracted from that image.
It exposes a high level API to access deep learning algorithms that have been trained on millions of images. It can also do more advanced things like facial recognition and text extraction, which I'm going to cover in a video for pro members. Our next step for this app is to go into the storage console and create a brand new bucket for the Cloud Vision function.
deep learning
machine learning
visualization
artificial intelligence
deepgram
The reason we do this is because Firebase functions can't be scoped to specific file locations, So it's best to have a dedicated bucket specifically for invoking the function. After that, go into the rules tab and make sure to enable read and write access for this bucket. As you may know, every Firebase project is also a Google Cloud Platform project. Cloud Vision is not enabled by default, So you need to go into the Cloud Platform console, to the APIs tab, then find Cloud Vision and enable it. Before we build the actual mobile app in Ionic, I want to build out the Cloud Function just so you know how everything works on the back end. So go into your index ts file in the functions directory, then import Firebase functions as well as the admin SDK. Because we have Cloud Vision enabled on the Cloud Platform console, we can just import it and then instantiate a new Vision client. There's no need to add API keys to the environment or anything like that. Then we need to set a variable for the name of the storage bucket. Otherwise, Firebase is going to try and use our default bucket. From there, I'll export a constant for our actual function which I'm calling image tagger. It's going to be a storage function that points to our bucket name. Then we'll call object on change and return an async function which just ensures that We return a promise from this operation. Now inside the function code, I'll set up a couple variables here for the actual file data and the file name, which is going to be important later on because the file name needs to correspond to the document ID in the Firestore database. The image URI is the actual stored file location, which is going to be used by Cloud Vision to read the actual raw file. So we'll upload the file from Ionic, But for right now, all you need to know is that that file path is going to have the same ID as its data in Firestore.
So we'll just split off the JPEG from it and then use it to make a reference to the actual document in Firestore, which is in the photos collection. At this point, we can do all of the deep learning magic with a single line of code. We can await the results from our vision client by calling label detection and passing it that image URI. It responds with a big object of data, but we don't need everything there.
deep learning
visualization
So we're just going to map it down to the label descriptions that we wanna show the user on the front end. We also wanna know if there's a hotdog in this image. So to do that, we can just look at this array of strings and use the fancy new includes method to see if there is a hotdog string in this array, which will return true or false. Now there's only one step left, and that is to save this data in Firestore. We already have a document reference, so we can just say doc ref set with the corresponding data. Now you can go ahead and deploy this function and try it out. Go into Firebase Storage and upload an image that you think might be a hot dog but just aren't 100% sure. Then jump over to the Firestore database and go into the photos collection, and you should see a document that looks something like this. We have a hot dog boolean value and then an array of a bunch of strings that label this image. Our back end is working perfectly. Let's CD back into Ionic and generate a new page called Vision. Then we have a couple of dependencies that we need to install on our project. The first one is the native camera Cordova plugin which will give us access to a camera on a native mobile device. Then we'll also install AngularFire There too to make Firebase more friendly inside of an Angular environment. Follow the install instructions for both packages, and then open your app module file. For AngularFire, you can see we're importing Firestore as well as the storage module. Import the camera here as well, And then make sure to add your Firebase configuration to this configuration object. The Firebase modules are going to go in the import section.
The Vision page goes in entry components, and the camera goes in the provider's array. After that, we just need to go into the app component. And instead of using the homepage as the root page, we're going to swap that out for the Vision page. Now, we can switch over to that Vision page and write all of the front end code.
visualization
First, we'll bring in the loading controller from Ionic and also a few things here from RXJS and Angular Firestore and Fire Storage and, of course, our Ionic camera. Then we'll go ahead and set up a few variables here in the page. The first one is the upload task. Then we'll declare an observable for the actual results that we get back from Firestore. We can also set up variables for the loading indicator and the image itself to show those in the front end as well. From there, we need to add all of our dependencies to the constructor. Again, those include Firestore, storage, the camera, and the loading controller. We can also instantiate the loading object in the constructor, which we do by calling loading controller create, and you can customize this as well. At this point, we need to define a method that will start the actual upload process. It takes the file as an argument, and then it will present the loading spinner once it's triggered. Then we'll generate a random ID for our document and our file name. So the actual file name is going to be the document ID followed by JPEG. And then we'll use that ID to make a reference to Firestore. Again, it's important that these two values match because that's how we keep track of the file in storage and its location in the Firestore database. Then we'll make a reference to it in Firestore even though that document doesn't technically exist at this point. We call value changes to get it as an observable, And then it's going to emit null initially, so we want to filter that value out. When we do get a value, we want to dismiss the loading indicator so the user can see the results in the front end. For that, we can use the tap operator and then call loading, dismiss. Now we're ready to handle the actual file. The camera is going to return it in a base 64 format, which is just a big long encrypted string. But Firebase needs it in a specific format, So all we have to do is append it to this string here. Then we can just make a reference to that storage location, which, again, should match the document ID.
And then instead of calling upload, we call put string with the actual image itself. The only missing piece at this point is how do we get the actual image from the camera. For that part, I'm going to set up an async function because the camera returns the image as a promise, and then it's important that the destination type is a data URL.
photography
Then because this is an async function, we can await the operation which is going to return the actual base 64 string. So we await the camera to get the picture with our configuration options.
And then after that, we'll go ahead and start the upload. Now we just have a couple things to set up in the page template. 1st, we wanna give the user a button that they can tap that will start the actual process to capture the image. After that, I set up an ng if statement so we can unwrap the results observable. And based on that result, we should know if the image is a hotdog or not.
visualization
modeling
So if it is, then we'll go ahead and display a green button. If it's not, we'll display a red button. Then we also have an array of labels in that result, So we can go ahead and loop over them, and we'll display an Ionic chip for each one. At this point, we should be good to go. If we run either an iOS or Android emulator, We should be able to get the Cloud Vision API results in our actual front end Ionic app. I'll go ahead and run the Android emulator with the Pixel 2, and we'll see how everything turned out. I've saved a couple images on this emulator, and we'll go ahead and select the hot dog. And you can see it uploads to storage, And then after a couple seconds or so, we get the results back, and, of course, this is hot dog.
If this video helped you, please like and subscribe. And if you're serious about building apps, consider becoming a pro member at angularfirebase.com. You'll get access to all kinds of exclusive content designed to help you build and ship your app faster. Thanks for watching, and I'll see you soon.
youtube
seasons
Today, I'm going to take you from 0 to a deep learning powered mobile app with Ionic, Firebase, and Google Cloud Vision. You might have seen this app before. It's ready. Please, god. Just do pizza. Yeah. Pizza. Hey, Zach. Not hot dog? Wait. What the f Not only will our app be able to detect hot dogs, but it can Also, label any image using Google's deep learning infrastructure. And it's actually a lot more sophisticated than you might think. As you can see here, if we upload a hotdog, it correctly He labels it as such, but what happens if we try to trick it by sending it a picture of a pug in a hot dog costume? It's smart enough to know that that's not actual hot dog, and labels it instead as a pug. If you like building cool apps, make sure to subscribe and follow along with the source code at angularfirebase.com. Let's take a closer look at how this works. The user will upload an image from our Ionic app to Firebase Storage. That's going to invoke a cloud function, which will send the image to the Google Vision API. It does its deep learning magic and converts the image into a set of labels. We save those labels in Firestore so they can be read in our front end Ionic app in real time. To get started with this project, You'll first want to start a new Ionic app using the blank template, then CD into your project and run Firebase init functions. This will associate your project with Firebase and create a folder called functions in the root of the project. And make sure to select TypeScript when prompted for that option. After that, you're going to want to CD into the functions directory, and then we're going to install the Google Cloud Vision library. Cloud Vision is a library that takes an image as its input and returns a whole bunch of data extracted from that image. It exposes a high level API to access deep learning algorithms that have been trained on millions of images. It can also do more advanced things like facial recognition and text extraction, which I'm going to cover in a video for pro members. Our next step for this app is to go into the storage console and create a brand new bucket for the Cloud Vision function. The reason we do this is because Firebase functions can't be scoped to specific file locations, So it's best to have a dedicated bucket specifically for invoking the function. After that, go into the rules tab and make sure to enable read and write access for this bucket. As you may know, every Firebase project is also a Google Cloud Platform project. Cloud Vision is not enabled by default, So you need to go into the Cloud Platform console, to the APIs tab, then find Cloud Vision and enable it. Before we build the actual mobile app in Ionic, I want to build out the Cloud Function just so you know how everything works on the back end. So go into your index ts file in the functions directory, then import Firebase functions as well as the admin SDK. Because we have Cloud Vision enabled on the Cloud Platform console, we can just import it and then instantiate a new Vision client. There's no need to add API keys to the environment or anything like that. Then we need to set a variable for the name of the storage bucket. Otherwise, Firebase is going to try and use our default bucket. From there, I'll export a constant for our actual function which I'm calling image tagger. It's going to be a storage function that points to our bucket name. Then we'll call object on change and return an async function which just ensures that We return a promise from this operation. Now inside the function code, I'll set up a couple variables here for the actual file data and the file name, which is going to be important later on because the file name needs to correspond to the document ID in the Firestore database. The image URI is the actual stored file location, which is going to be used by Cloud Vision to read the actual raw file. So we'll upload the file from Ionic, But for right now, all you need to know is that that file path is going to have the same ID as its data in Firestore. So we'll just split off the JPEG from it and then use it to make a reference to the actual document in Firestore, which is in the photos collection. At this point, we can do all of the deep learning magic with a single line of code. We can await the results from our vision client by calling label detection and passing it that image URI. It responds with a big object of data, but we don't need everything there. So we're just going to map it down to the label descriptions that we wanna show the user on the front end. We also wanna know if there's a hotdog in this image. So to do that, we can just look at this array of strings and use the fancy new includes method to see if there is a hotdog string in this array, which will return true or false. Now there's only one step left, and that is to save this data in Firestore. We already have a document reference, so we can just say doc ref set with the corresponding data. Now you can go ahead and deploy this function and try it out. Go into Firebase Storage and upload an image that you think might be a hot dog but just aren't 100% sure. Then jump over to the Firestore database and go into the photos collection, and you should see a document that looks something like this. We have a hot dog boolean value and then an array of a bunch of strings that label this image. Our back end is working perfectly. Let's CD back into Ionic and generate a new page called Vision. Then we have a couple of dependencies that we need to install on our project. The first one is the native camera Cordova plugin which will give us access to a camera on a native mobile device. Then we'll also install AngularFire There too to make Firebase more friendly inside of an Angular environment. Follow the install instructions for both packages, and then open your app module file. For AngularFire, you can see we're importing Firestore as well as the storage module. Import the camera here as well, And then make sure to add your Firebase configuration to this configuration object. The Firebase modules are going to go in the import section. The Vision page goes in entry components, and the camera goes in the provider's array. After that, we just need to go into the app component. And instead of using the homepage as the root page, we're going to swap that out for the Vision page. Now, we can switch over to that Vision page and write all of the front end code. First, we'll bring in the loading controller from Ionic and also a few things here from RXJS and Angular Firestore and Fire Storage and, of course, our Ionic camera. Then we'll go ahead and set up a few variables here in the page. The first one is the upload task. Then we'll declare an observable for the actual results that we get back from Firestore. We can also set up variables for the loading indicator and the image itself to show those in the front end as well. From there, we need to add all of our dependencies to the constructor. Again, those include Firestore, storage, the camera, and the loading controller. We can also instantiate the loading object in the constructor, which we do by calling loading controller create, and you can customize this as well. At this point, we need to define a method that will start the actual upload process. It takes the file as an argument, and then it will present the loading spinner once it's triggered. Then we'll generate a random ID for our document and our file name. So the actual file name is going to be the document ID followed by JPEG. And then we'll use that ID to make a reference to Firestore. Again, it's important that these two values match because that's how we keep track of the file in storage and its location in the Firestore database. Then we'll make a reference to it in Firestore even though that document doesn't technically exist at this point. We call value changes to get it as an observable, And then it's going to emit null initially, so we want to filter that value out. When we do get a value, we want to dismiss the loading indicator so the user can see the results in the front end. For that, we can use the tap operator and then call loading, dismiss. Now we're ready to handle the actual file. The camera is going to return it in a base 64 format, which is just a big long encrypted string. But Firebase needs it in a specific format, So all we have to do is append it to this string here. Then we can just make a reference to that storage location, which, again, should match the document ID. And then instead of calling upload, we call put string with the actual image itself. The only missing piece at this point is how do we get the actual image from the camera. For that part, I'm going to set up an async function because the camera returns the image as a promise, and then it's important that the destination type is a data URL. Then because this is an async function, we can await the operation which is going to return the actual base 64 string. So we await the camera to get the picture with our configuration options. And then after that, we'll go ahead and start the upload. Now we just have a couple things to set up in the page template. 1st, we wanna give the user a button that they can tap that will start the actual process to capture the image. After that, I set up an ng if statement so we can unwrap the results observable. And based on that result, we should know if the image is a hotdog or not. So if it is, then we'll go ahead and display a green button. If it's not, we'll display a red button. Then we also have an array of labels in that result, So we can go ahead and loop over them, and we'll display an Ionic chip for each one. At this point, we should be good to go. If we run either an iOS or Android emulator, We should be able to get the Cloud Vision API results in our actual front end Ionic app. I'll go ahead and run the Android emulator with the Pixel 2, and we'll see how everything turned out. I've saved a couple images on this emulator, and we'll go ahead and select the hot dog. And you can see it uploads to storage, And then after a couple seconds or so, we get the results back, and, of course, this is hot dog. If this video helped you, please like and subscribe. And if you're serious about building apps, consider becoming a pro member at angularfirebase.com. You'll get access to all kinds of exclusive content designed to help you build and ship your app faster. Thanks for watching, and I'll see you soon.
Today,
today
I'm
i'm
going
going
to
to
take
take
you
you
from
from
0
0
to
to
a
a
deep
deep
learning
learning
powered
powered
mobile
mobile
app
app
with
with
Ionic,
ionic
Firebase,
firebase
and
and
Google
google
Cloud
cloud
Vision.
vision
You
you
might
might
have
have
seen
seen
this
this
app
app
before.
before
It's
it's
ready.
ready
Please,
please
god.
god
Just
just
do
do
pizza.
pizza
Yeah.
yeah
Pizza.
pizza
Hey,
hey
Zach.
zach
Not
not
hot
hot
dog?
dog
Wait.
wait
What
what
the
the
f
f
Not
not
only
only
will
will
our
our
app
app
be
be
able
able
to
to
detect
detect
hot
hot
dogs,
dogs
but
but
it
it
can
can
Also,
also
label
label
any
any
image
image
using
using
Google's
google's
deep
deep
learning
learning
infrastructure.
infrastructure
And
and
it's
it's
actually
actually
a
a
lot
lot
more
more
sophisticated
sophisticated
than
than
you
you
might
might
think.
think
As
as
you
you
can
can
see
see
here,
here
if
if
we
we
upload
upload
a
a
hotdog,
hotdog
it
it
correctly
correctly
He
he
labels
labels
it
it
as
as
such,
such
but
but
what
what
happens
happens
if
if
we
we
try
try
to
to
trick
trick
it
it
by
by
sending
sending
it
it
a
a
picture
picture
of
of
a
a
pug
pug
in
in
a
a
hot
hot
dog
dog
costume?
costume
It's
it's
smart
smart
enough
enough
to
to
know
know
that
that
that's
that's
not
not
actual
actual
hot
hot
dog,
dog
and
and
labels
labels
it
it
instead
instead
as
as
a
a
pug.
pug
If
if
you
you
like
like
building
building
cool
cool
apps,
apps
make
make
sure
sure
to
to
subscribe
subscribe
and
and
follow
follow
along
along
with
with
the
the
source
source
code
code
at
at
angularfirebase.com.
angularfirebase.com
Let's
let's
take
take
a
a
closer
closer
look
look
at
at
how
how
this
this
works.
works
The
the
user
user
will
will
upload
upload
an
an
image
image
from
from
our
our
Ionic
ionic
app
app
to
to
Firebase
firebase
Storage.
storage
That's
that's
going
going
to
to
invoke
invoke
a
a
cloud
cloud
function,
function
which
which
will
will
send
send
the
the
image
image
to
to
the
the
Google
google
Vision
vision
API.
api
It
it
does
does
its
its
deep
deep
learning
learning
magic
magic
and
and
converts
converts
the
the
image
image
into
into
a
a
set
set
of
of
labels.
labels
We
we
save
save
those
those
labels
labels
in
in
Firestore
firestore
so
so
they
they
can
can
be
be
read
read
in
in
our
our
front
front
end
end
Ionic
ionic
app
app
in
in
real
real
time.
time
To
to
get
get
started
started
with
with
this
this
project,
project
You'll
you'll
first
first
want
want
to
to
start
start
a
a
new
new
Ionic
ionic
app
app
using
using
the
the
blank
blank
template,
template
then
then
CD
cd
into
into
your
your
project
project
and
and
run
run
Firebase
firebase
init
init
functions.
functions
This
this
will
will
associate
associate
your
your
project
project
with
with
Firebase
firebase
and
and
create
create
a
a
folder
folder
called
called
functions
functions
in
in
the
the
root
root
of
of
the
the
project.
project
And
and
make
make
sure
sure
to
to
select
select
TypeScript
typescript
when
when
prompted
prompted
for
for
that
that
option.
option
After
after
that,
that
you're
you're
going
going
to
to
want
want
to
to
CD
cd
into
into
the
the
functions
functions
directory,
directory
and
and
then
then
we're
we're
going
going
to
to
install
install
the
the
Google
google
Cloud
cloud
Vision
vision
library.
library
Cloud
cloud
Vision
vision
is
is
a
a
library
library
that
that
takes
takes
an
an
image
image
as
as
its
its
input
input
and
and
returns
returns
a
a
whole
whole
bunch
bunch
of
of
data
data
extracted
extracted
from
from
that
that
image.
image
It
it
exposes
exposes
a
a
high
high
level
level
API
api
to
to
access
access
deep
deep
learning
learning
algorithms
algorithms
that
that
have
have
been
been
trained
trained
on
on
millions
millions
of
of
images.
images
It
it
can
can
also
also
do
do
more
more
advanced
advanced
things
things
like
like
facial
facial
recognition
recognition
and
and
text
text
extraction,
extraction
which
which
I'm
i'm
going
going
to
to
cover
cover
in
in
a
a
video
video
for
for
pro
pro
members.
members
Our
our
next
next
step
step
for
for
this
this
app
app
is
is
to
to
go
go
into
into
the
the
storage
storage
console
console
and
and
create
create
a
a
brand
brand
new
new
bucket
bucket
for
for
the
the
Cloud
cloud
Vision
vision
function.
function
The
the
reason
reason
we
we
do
do
this
this
is
is
because
because
Firebase
firebase
functions
functions
can't
can't
be
be
scoped
scoped
to
to
specific
specific
file
file
locations,
locations
So
so
it's
it's
best
best
to
to
have
have
a
a
dedicated
dedicated
bucket
bucket
specifically
specifically
for
for
invoking
invoking
the
the
function.
function
After
after
that,
that
go
go
into
into
the
the
rules
rules
tab
tab
and
and
make
make
sure
sure
to
to
enable
enable
read
read
and
and
write
write
access
access
for
for
this
this
bucket.
bucket
As
as
you
you
may
may
know,
know
every
every
Firebase
firebase
project
project
is
is
also
also
a
a
Google
google
Cloud
cloud
Platform
platform
project.
project
Cloud
cloud
Vision
vision
is
is
not
not
enabled
enabled
by
by
default,
default
So
so
you
you
need
need
to
to
go
go
into
into
the
the
Cloud
cloud
Platform
platform
console,
console
to
to
the
the
APIs
apis
tab,
tab
then
then
find
find
Cloud
cloud
Vision
vision
and
and
enable
enable
it.
it
Before
before
we
we
build
build
the
the
actual
actual
mobile
mobile
app
app
in
in
Ionic,
ionic
I
i
want
want
to
to
build
build
out
out
the
the
Cloud
cloud
Function
function
just
just
so
so
you
you
know
know
how
how
everything
everything
works
works
on
on
the
the
back
back
end.
end
So
so
go
go
into
into
your
your
index
index
ts
ts
file
file
in
in
the
the
functions
functions
directory,
directory
then
then
import
import
Firebase
firebase
functions
functions
as
as
well
well
as
as
the
the
admin
admin
SDK.
sdk
Because
because
we
we
have
have
Cloud
cloud
Vision
vision
enabled
enabled
on
on
the
the
Cloud
cloud
Platform
platform
console,
console
we
we
can
can
just
just
import
import
it
it
and
and
then
then
instantiate
instantiate
a
a
new
new
Vision
vision
client.
client
There's
there's
no
no
need
need
to
to
add
add
API
api
keys
keys
to
to
the
the
environment
environment
or
or
anything
anything
like
like
that.
that
Then
then
we
we
need
need
to
to
set
set
a
a
variable
variable
for
for
the
the
name
name
of
of
the
the
storage
storage
bucket.
bucket
Otherwise,
otherwise
Firebase
firebase
is
is
going
going
to
to
try
try
and
and
use
use
our
our
default
default
bucket.
bucket
From
from
there,
there
I'll
i'll
export
export
a
a
constant
constant
for
for
our
our
actual
actual
function
function
which
which
I'm
i'm
calling
calling
image
image
tagger.
tagger
It's
it's
going
going
to
to
be
be
a
a
storage
storage
function
function
that
that
points
points
to
to
our
our
bucket
bucket
name.
name
Then
then
we'll
we'll
call
call
object
object
on
on
change
change
and
and
return
return
an
an
async
async
function
function
which
which
just
just
ensures
ensures
that
that
We
we
return
return
a
a
promise
promise
from
from
this
this
operation.
operation
Now
now
inside
inside
the
the
function
function
code,
code
I'll
i'll
set
set
up
up
a
a
couple
couple
variables
variables
here
here
for
for
the
the
actual
actual
file
file
data
data
and
and
the
the
file
file
name,
name
which
which
is
is
going
going
to
to
be
be
important
important
later
later
on
on
because
because
the
the
file
file
name
name
needs
needs
to
to
correspond
correspond
to
to
the
the
document
document
ID
id
in
in
the
the
Firestore
firestore
database.
database
The
the
image
image
URI
uri
is
is
the
the
actual
actual
stored
stored
file
file
location,
location
which
which
is
is
going
going
to
to
be
be
used
used
by
by
Cloud
cloud
Vision
vision
to
to
read
read
the
the
actual
actual
raw
raw
file.
file
So
so
we'll
we'll
upload
upload
the
the
file
file
from
from
Ionic,
ionic
But
but
for
for
right
right
now,
now
all
all
you
you
need
need
to
to
know
know
is
is
that
that
that
that
file
file
path
path
is
is
going
going
to
to
have
have
the
the
same
same
ID
id
as
as
its
its
data
data
in
in
Firestore.
firestore
So
so
we'll
we'll
just
just
split
split
off
off
the
the
JPEG
jpeg
from
from
it
it
and
and
then
then
use
use
it
it
to
to
make
make
a
a
reference
reference
to
to
the
the
actual
actual
document
document
in
in
Firestore,
firestore
which
which
is
is
in
in
the
the
photos
photos
collection.
collection
At
at
this
this
point,
point
we
we
can
can
do
do
all
all
of
of
the
the
deep
deep
learning
learning
magic
magic
with
with
a
a
single
single
line
line
of
of
code.
code
We
we
can
can
await
await
the
the
results
results
from
from
our
our
vision
vision
client
client
by
by
calling
calling
label
label
detection
detection
and
and
passing
passing
it
it
that
that
image
image
URI.
uri
It
it
responds
responds
with
with
a
a
big
big
object
object
of
of
data,
data
but
but
we
we
don't
don't
need
need
everything
everything
there.
there
So
so
we're
we're
just
just
going
going
to
to
map
map
it
it
down
down
to
to
the
the
label
label
descriptions
descriptions
that
that
we
we
wanna
wanna
show
show
the
the
user
user
on
on
the
the
front
front
end.
end
We
we
also
also
wanna
wanna
know
know
if
if
there's
there's
a
a
hotdog
hotdog
in
in
this
this
image.
image
So
so
to
to
do
do
that,
that
we
we
can
can
just
just
look
look
at
at
this
this
array
array
of
of
strings
strings
and
and
use
use
the
the
fancy
fancy
new
new
includes
includes
method
method
to
to
see
see
if
if
there
there
is
is
a
a
hotdog
hotdog
string
string
in
in
this
this
array,
array
which
which
will
will
return
return
true
true
or
or
false.
false
Now
now
there's
there's
only
only
one
one
step
step
left,
left
and
and
that
that
is
is
to
to
save
save
this
this
data
data
in
in
Firestore.
firestore
We
we
already
already
have
have
a
a
document
document
reference,
reference
so
so
we
we
can
can
just
just
say
say
doc
doc
ref
ref
set
set
with
with
the
the
corresponding
corresponding
data.
data
Now
now
you
you
can
can
go
go
ahead
ahead
and
and
deploy
deploy
this
this
function
function
and
and
try
try
it
it
out.
out
Go
go
into
into
Firebase
firebase
Storage
storage
and
and
upload
upload
an
an
image
image
that
that
you
you
think
think
might
might
be
be
a
a
hot
hot
dog
dog
but
but
just
just
aren't
aren't
100%
100%
sure.
sure
Then
then
jump
jump
over
over
to
to
the
the
Firestore
firestore
database
database
and
and
go
go
into
into
the
the
photos
photos
collection,
collection
and
and
you
you
should
should
see
see
a
a
document
document
that
that
looks
looks
something
something
like
like
this.
this
We
we
have
have
a
a
hot
hot
dog
dog
boolean
boolean
value
value
and
and
then
then
an
an
array
array
of
of
a
a
bunch
bunch
of
of
strings
strings
that
that
label
label
this
this
image.
image
Our
our
back
back
end
end
is
is
working
working
perfectly.
perfectly
Let's
let's
CD
cd
back
back
into
into
Ionic
ionic
and
and
generate
generate
a
a
new
new
page
page
called
called
Vision.
vision
Then
then
we
we
have
have
a
a
couple
couple
of
of
dependencies
dependencies
that
that
we
we
need
need
to
to
install
install
on
on
our
our
project.
project
The
the
first
first
one
one
is
is
the
the
native
native
camera
camera
Cordova
cordova
plugin
plugin
which
which
will
will
give
give
us
us
access
access
to
to
a
a
camera
camera
on
on
a
a
native
native
mobile
mobile
device.
device
Then
then
we'll
we'll
also
also
install
install
AngularFire
angularfire
There
there
too
too
to
to
make
make
Firebase
firebase
more
more
friendly
friendly
inside
inside
of
of
an
an
Angular
angular
environment.
environment
Follow
follow
the
the
install
install
instructions
instructions
for
for
both
both
packages,
packages
and
and
then
then
open
open
your
your
app
app
module
module
file.
file
For
for
AngularFire,
angularfire
you
you
can
can
see
see
we're
we're
importing
importing
Firestore
firestore
as
as
well
well
as
as
the
the
storage
storage
module.
module
Import
import
the
the
camera
camera
here
here
as
as
well,
well
And
and
then
then
make
make
sure
sure
to
to
add
add
your
your
Firebase
firebase
configuration
configuration
to
to
this
this
configuration
configuration
object.
object
The
the
Firebase
firebase
modules
modules
are
are
going
going
to
to
go
go
in
in
the
the
import
import
section.
section
The
the
Vision
vision
page
page
goes
goes
in
in
entry
entry
components,
components
and
and
the
the
camera
camera
goes
goes
in
in
the
the
provider's
provider's
array.
array
After
after
that,
that
we
we
just
just
need
need
to
to
go
go
into
into
the
the
app
app
component.
component
And
and
instead
instead
of
of
using
using
the
the
homepage
homepage
as
as
the
the
root
root
page,
page
we're
we're
going
going
to
to
swap
swap
that
that
out
out
for
for
the
the
Vision
vision
page.
page
Now,
now
we
we
can
can
switch
switch
over
over
to
to
that
that
Vision
vision
page
page
and
and
write
write
all
all
of
of
the
the
front
front
end
end
code.
code
First,
first
we'll
we'll
bring
bring
in
in
the
the
loading
loading
controller
controller
from
from
Ionic
ionic
and
and
also
also
a
a
few
few
things
things
here
here
from
from
RXJS
rxjs
and
and
Angular
angular
Firestore
firestore
and
and
Fire
fire
Storage
storage
and,
and
of
of
course,
course
our
our
Ionic
ionic
camera.
camera
Then
then
we'll
we'll
go
go
ahead
ahead
and
and
set
set
up
up
a
a
few
few
variables
variables
here
here
in
in
the
the
page.
page
The
the
first
first
one
one
is
is
the
the
upload
upload
task.
task
Then
then
we'll
we'll
declare
declare
an
an
observable
observable
for
for
the
the
actual
actual
results
results
that
that
we
we
get
get
back
back
from
from
Firestore.
firestore
We
we
can
can
also
also
set
set
up
up
variables
variables
for
for
the
the
loading
loading
indicator
indicator
and
and
the
the
image
image
itself
itself
to
to
show
show
those
those
in
in
the
the
front
front
end
end
as
as
well.
well
From
from
there,
there
we
we
need
need
to
to
add
add
all
all
of
of
our
our
dependencies
dependencies
to
to
the
the
constructor.
constructor
Again,
again
those
those
include
include
Firestore,
firestore
storage,
storage
the
the
camera,
camera
and
and
the
the
loading
loading
controller.
controller
We
we
can
can
also
also
instantiate
instantiate
the
the
loading
loading
object
object
in
in
the
the
constructor,
constructor
which
which
we
we
do
do
by
by
calling
calling
loading
loading
controller
controller
create,
create
and
and
you
you
can
can
customize
customize
this
this
as
as
well.
well
At
at
this
this
point,
point
we
we
need
need
to
to
define
define
a
a
method
method
that
that
will
will
start
start
the
the
actual
actual
upload
upload
process.
process
It
it
takes
takes
the
the
file
file
as
as
an
an
argument,
argument
and
and
then
then
it
it
will
will
present
present
the
the
loading
loading
spinner
spinner
once
once
it's
it's
triggered.
triggered
Then
then
we'll
we'll
generate
generate
a
a
random
random
ID
id
for
for
our
our
document
document
and
and
our
our
file
file
name.
name
So
so
the
the
actual
actual
file
file
name
name
is
is
going
going
to
to
be
be
the
the
document
document
ID
id
followed
followed
by
by
JPEG.
jpeg
And
and
then
then
we'll
we'll
use
use
that
that
ID
id
to
to
make
make
a
a
reference
reference
to
to
Firestore.
firestore
Again,
again
it's
it's
important
important
that
that
these
these
two
two
values
values
match
match
because
because
that's
that's
how
how
we
we
keep
keep
track
track
of
of
the
the
file
file
in
in
storage
storage
and
and
its
its
location
location
in
in
the
the
Firestore
firestore
database.
database
Then
then
we'll
we'll
make
make
a
a
reference
reference
to
to
it
it
in
in
Firestore
firestore
even
even
though
though
that
that
document
document
doesn't
doesn't
technically
technically
exist
exist
at
at
this
this
point.
point
We
we
call
call
value
value
changes
changes
to
to
get
get
it
it
as
as
an
an
observable,
observable
And
and
then
then
it's
it's
going
going
to
to
emit
emit
null
null
initially,
initially
so
so
we
we
want
want
to
to
filter
filter
that
that
value
value
out.
out
When
when
we
we
do
do
get
get
a
a
value,
value
we
we
want
want
to
to
dismiss
dismiss
the
the
loading
loading
indicator
indicator
so
so
the
the
user
user
can
can
see
see
the
the
results
results
in
in
the
the
front
front
end.
end
For
for
that,
that
we
we
can
can
use
use
the
the
tap
tap
operator
operator
and
and
then
then
call
call
loading,
loading
dismiss.
dismiss
Now
now
we're
we're
ready
ready
to
to
handle
handle
the
the
actual
actual
file.
file
The
the
camera
camera
is
is
going
going
to
to
return
return
it
it
in
in
a
a
base
base
64
64
format,
format
which
which
is
is
just
just
a
a
big
big
long
long
encrypted
encrypted
string.
string
But
but
Firebase
firebase
needs
needs
it
it
in
in
a
a
specific
specific
format,
format
So
so
all
all
we
we
have
have
to
to
do
do
is
is
append
append
it
it
to
to
this
this
string
string
here.
here
Then
then
we
we
can
can
just
just
make
make
a
a
reference
reference
to
to
that
that
storage
storage
location,
location
which,
which
again,
again
should
should
match
match
the
the
document
document
ID.
id
And
and
then
then
instead
instead
of
of
calling
calling
upload,
upload
we
we
call
call
put
put
string
string
with
with
the
the
actual
actual
image
image
itself.
itself
The
the
only
only
missing
missing
piece
piece
at
at
this
this
point
point
is
is
how
how
do
do
we
we
get
get
the
the
actual
actual
image
image
from
from
the
the
camera.
camera
For
for
that
that
part,
part
I'm
i'm
going
going
to
to
set
set
up
up
an
an
async
async
function
function
because
because
the
the
camera
camera
returns
returns
the
the
image
image
as
as
a
a
promise,
promise
and
and
then
then
it's
it's
important
important
that
that
the
the
destination
destination
type
type
is
is
a
a
data
data
URL.
url
Then
then
because
because
this
this
is
is
an
an
async
async
function,
function
we
we
can
can
await
await
the
the
operation
operation
which
which
is
is
going
going
to
to
return
return
the
the
actual
actual
base
base
64
64
string.
string
So
so
we
we
await
await
the
the
camera
camera
to
to
get
get
the
the
picture
picture
with
with
our
our
configuration
configuration
options.
options
And
and
then
then
after
after
that,
that
we'll
we'll
go
go
ahead
ahead
and
and
start
start
the
the
upload.
upload
Now
now
we
we
just
just
have
have
a
a
couple
couple
things
things
to
to
set
set
up
up
in
in
the
the
page
page
template.
template
1st,
1st
we
we
wanna
wanna
give
give
the
the
user
user
a
a
button
button
that
that
they
they
can
can
tap
tap
that
that
will
will
start
start
the
the
actual
actual
process
process
to
to
capture
capture
the
the
image.
image
After
after
that,
that
I
i
set
set
up
up
an
an
ng
ng
if
if
statement
statement
so
so
we
we
can
can
unwrap
unwrap
the
the
results
results
observable.
observable
And
and
based
based
on
on
that
that
result,
result
we
we
should
should
know
know
if
if
the
the
image
image
is
is
a
a
hotdog
hotdog
or
or
not.
not
So
so
if
if
it
it
is,
is
then
then
we'll
we'll
go
go
ahead
ahead
and
and
display
display
a
a
green
green
button.
button
If
if
it's
it's
not,
not
we'll
we'll
display
display
a
a
red
red
button.
button
Then
then
we
we
also
also
have
have
an
an
array
array
of
of
labels
labels
in
in
that
that
result,
result
So
so
we
we
can
can
go
go
ahead
ahead
and
and
loop
loop
over
over
them,
them
and
and
we'll
we'll
display
display
an
an
Ionic
ionic
chip
chip
for
for
each
each
one.
one
At
at
this
this
point,
point
we
we
should
should
be
be
good
good
to
to
go.
go
If
if
we
we
run
run
either
either
an
an
iOS
ios
or
or
Android
android
emulator,
emulator
We
we
should
should
be
be
able
able
to
to
get
get
the
the
Cloud
cloud
Vision
vision
API
api
results
results
in
in
our
our
actual
actual
front
front
end
end
Ionic
ionic
app.
app
I'll
i'll
go
go
ahead
ahead
and
and
run
run
the
the
Android
android
emulator
emulator
with
with
the
the
Pixel
pixel
2,
2
and
and
we'll
we'll
see
see
how
how
everything
everything
turned
turned
out.
out
I've
i've
saved
saved
a
a
couple
couple
images
images
on
on
this
this
emulator,
emulator
and
and
we'll
we'll
go
go
ahead
ahead
and
and
select
select
the
the
hot
hot
dog.
dog
And
and
you
you
can
can
see
see
it
it
uploads
uploads
to
to
storage,
storage
And
and
then
then
after
after
a
a
couple
couple
seconds
seconds
or
or
so,
so
we
we
get
get
the
the
results
results
back,
back
and,
and
of
of
course,
course
this
this
is
is
hot
hot
dog.
dog
If
if
this
this
video
video
helped
helped
you,
you
please
please
like
like
and
and
subscribe.
subscribe
And
and
if
if
you're
you're
serious
serious
about
about
building
building
apps,
apps
consider
consider
becoming
becoming
a
a
pro
pro
member
member
at
at
angularfirebase.com.
angularfirebase.com
You'll
you'll
get
get
access
access
to
to
all
all
kinds
kinds
of
of
exclusive
exclusive
content
content
designed
designed
to
to
help
help
you
you
build
build
and
and
ship
ship
your
your
app
app
faster.
faster
Thanks
thanks
for
for
watching,
watching
and
and
I'll
i'll
see
see
you
you
soon.
soon
success
The Ionic app is designed to detect hot dogs and label images using Google's deep learning infrastructure and uploads images to AWS. The Cloud Function app allows for the creation of a new page called Vision, which involves installing the camera and the front end Ionic app. The speakers emphasize the importance of setting up variables for the front end and the need to define a method for uploading files. They also discuss how to access the Cloud Vision API results in the front end Ionic app and generate a hotdog string.
Seefood - Ionic + Firebase + Cloud Vision
Today, I'm going to take you from 0 to a deep learning powered mobile app with Ionic, Firebase, and Google Cloud Vision. You might have seen this app before. It's ready. Please, god. Just do pizza. Yeah. Pizza. Hey, Zach. Not hot dog? Wait. What the f Not only will our app be able to detect hot dogs, but it can Also, label any image using Google's deep learning infrastructure. And it's actually a lot more sophisticated than you might think. As you can see here, if we upload a hotdog, it correctly He labels it as such, but what happens if we try to trick it by sending it a picture of a pug in a hot dog costume? It's smart enough to know that that's not actual hot dog, and labels it instead as a pug. If you like building cool apps, make sure to subscribe and follow along with the source code at angularfirebase.com. Let's take a closer look at how this works. The user will upload an image from our Ionic app to Firebase Storage. That's going to invoke a cloud function, which will send the image to the Google Vision API. It does its deep learning magic and converts the image into a set of labels. We save those labels in Firestore so they can be read in our front end Ionic app in real time. To get started with this project, You'll first want to start a new Ionic app using the blank template, then CD into your project and run Firebase init functions. This will associate your project with Firebase and create a folder called functions in the root of the project. And make sure to select TypeScript when prompted for that option. After that, you're going to want to CD into the functions directory, and then we're going to install the Google Cloud Vision library. Cloud Vision is a library that takes an image as its input and returns a whole bunch of data extracted from that image. It exposes a high level API to access deep learning algorithms that have been trained on millions of images. It can also do more advanced things like facial recognition and text extraction, which I'm going to cover in a video for pro members. Our next step for this app is to go into the storage console and create a brand new bucket for the Cloud Vision function. The reason we do this is because Firebase functions can't be scoped to specific file locations, So it's best to have a dedicated bucket specifically for invoking the function. After that, go into the rules tab and make sure to enable read and write access for this bucket. As you may know, every Firebase project is also a Google Cloud Platform project. Cloud Vision is not enabled by default, So you need to go into the Cloud Platform console, to the APIs tab, then find Cloud Vision and enable it. Before we build the actual mobile app in Ionic, I want to build out the Cloud Function just so you know how everything works on the back end. So go into your index ts file in the functions directory, then import Firebase functions as well as the admin SDK. Because we have Cloud Vision enabled on the Cloud Platform console, we can just import it and then instantiate a new Vision client. There's no need to add API keys to the environment or anything like that. Then we need to set a variable for the name of the storage bucket. Otherwise, Firebase is going to try and use our default bucket. From there, I'll export a constant for our actual function which I'm calling image tagger. It's going to be a storage function that points to our bucket name. Then we'll call object on change and return an async function which just ensures that We return a promise from this operation. Now inside the function code, I'll set up a couple variables here for the actual file data and the file name, which is going to be important later on because the file name needs to correspond to the document ID in the Firestore database. The image URI is the actual stored file location, which is going to be used by Cloud Vision to read the actual raw file. So we'll upload the file from Ionic, But for right now, all you need to know is that that file path is going to have the same ID as its data in Firestore. So we'll just split off the JPEG from it and then use it to make a reference to the actual document in Firestore, which is in the photos collection. At this point, we can do all of the deep learning magic with a single line of code. We can await the results from our vision client by calling label detection and passing it that image URI. It responds with a big object of data, but we don't need everything there. So we're just going to map it down to the label descriptions that we wanna show the user on the front end. We also wanna know if there's a hotdog in this image. So to do that, we can just look at this array of strings and use the fancy new includes method to see if there is a hotdog string in this array, which will return true or false. Now there's only one step left, and that is to save this data in Firestore. We already have a document reference, so we can just say doc ref set with the corresponding data. Now you can go ahead and deploy this function and try it out. Go into Firebase Storage and upload an image that you think might be a hot dog but just aren't 100% sure. Then jump over to the Firestore database and go into the photos collection, and you should see a document that looks something like this. We have a hot dog boolean value and then an array of a bunch of strings that label this image. Our back end is working perfectly. Let's CD back into Ionic and generate a new page called Vision. Then we have a couple of dependencies that we need to install on our project. The first one is the native camera Cordova plugin which will give us access to a camera on a native mobile device. Then we'll also install AngularFire There too to make Firebase more friendly inside of an Angular environment. Follow the install instructions for both packages, and then open your app module file. For AngularFire, you can see we're importing Firestore as well as the storage module. Import the camera here as well, And then make sure to add your Firebase configuration to this configuration object. The Firebase modules are going to go in the import section. The Vision page goes in entry components, and the camera goes in the provider's array. After that, we just need to go into the app component. And instead of using the homepage as the root page, we're going to swap that out for the Vision page. Now, we can switch over to that Vision page and write all of the front end code. First, we'll bring in the loading controller from Ionic and also a few things here from RXJS and Angular Firestore and Fire Storage and, of course, our Ionic camera. Then we'll go ahead and set up a few variables here in the page. The first one is the upload task. Then we'll declare an observable for the actual results that we get back from Firestore. We can also set up variables for the loading indicator and the image itself to show those in the front end as well. From there, we need to add all of our dependencies to the constructor. Again, those include Firestore, storage, the camera, and the loading controller. We can also instantiate the loading object in the constructor, which we do by calling loading controller create, and you can customize this as well. At this point, we need to define a method that will start the actual upload process. It takes the file as an argument, and then it will present the loading spinner once it's triggered. Then we'll generate a random ID for our document and our file name. So the actual file name is going to be the document ID followed by JPEG. And then we'll use that ID to make a reference to Firestore. Again, it's important that these two values match because that's how we keep track of the file in storage and its location in the Firestore database. Then we'll make a reference to it in Firestore even though that document doesn't technically exist at this point. We call value changes to get it as an observable, And then it's going to emit null initially, so we want to filter that value out. When we do get a value, we want to dismiss the loading indicator so the user can see the results in the front end. For that, we can use the tap operator and then call loading, dismiss. Now we're ready to handle the actual file. The camera is going to return it in a base 64 format, which is just a big long encrypted string. But Firebase needs it in a specific format, So all we have to do is append it to this string here. Then we can just make a reference to that storage location, which, again, should match the document ID. And then instead of calling upload, we call put string with the actual image itself. The only missing piece at this point is how do we get the actual image from the camera. For that part, I'm going to set up an async function because the camera returns the image as a promise, and then it's important that the destination type is a data URL. Then because this is an async function, we can await the operation which is going to return the actual base 64 string. So we await the camera to get the picture with our configuration options. And then after that, we'll go ahead and start the upload. Now we just have a couple things to set up in the page template. 1st, we wanna give the user a button that they can tap that will start the actual process to capture the image. After that, I set up an ng if statement so we can unwrap the results observable. And based on that result, we should know if the image is a hotdog or not. So if it is, then we'll go ahead and display a green button. If it's not, we'll display a red button. Then we also have an array of labels in that result, So we can go ahead and loop over them, and we'll display an Ionic chip for each one. At this point, we should be good to go. If we run either an iOS or Android emulator, We should be able to get the Cloud Vision API results in our actual front end Ionic app. I'll go ahead and run the Android emulator with the Pixel 2, and we'll see how everything turned out. I've saved a couple images on this emulator, and we'll go ahead and select the hot dog. And you can see it uploads to storage, And then after a couple seconds or so, we get the results back, and, of course, this is hot dog. If this video helped you, please like and subscribe. And if you're serious about building apps, consider becoming a pro member at angularfirebase.com. You'll get access to all kinds of exclusive content designed to help you build and ship your app faster. Thanks for watching, and I'll see you soon.
The Ionic app is designed to detect hot dogs and label images using Google's deep learning infrastructure and uploads images to AWS. The Cloud Function app allows for the creation of a new page called Vision, which involves installing the camera and the front end Ionic app. The speakers emphasize the importance of setting up variables for the front end and the need to define a method for uploading files. They also discuss how to access the Cloud Vision API results in the front end Ionic app and generate a hotdog string.