Learn how to build and deploy a machine learning model to Cloud ML Engine, then make it available to the world via Firebase Cloud Functions. https://angularfirebase.com/lessons/serverless-machine-learning-with-python-and-firebase-cloud-functions/

- Radi Cho's TensorFlow Repo https://github.com/radi-cho/tfjs-firebase
- ML Engine https://cloud.google.com/ml-engine/
- Datalab https://cloud.google.com/datalab/
2024-02-04T07:12:53.634Z
nova-2
2-general-nova
2024-01-09.29447
30089e05-99d1-4376-b32e-c263170674af
433118d6-1120-4433-b3d5-693cf1b486f6
37486a1ef83905cce59f290980a57208830d4f8c790874aeaf3c6ad683041749
67875a7f-c9c4-48a0-aa55-5bdb8a91c34a
deprecated
Today, we'll start from 0 and finish the video with our own custom machine learning model deployed to the cloud as an API.
We will pull some actual user data that we have saved in the Firestore database, then we'll export it to Datalab so we can train a Python based machine learning model with virtually unlimited compute resources.
Once we have a solid model in place we'll deploy it to ML engine so we can version it and analyze it in production, and finally we will Expose it to the universe by making it available in a Firebase cloud function.
But before you do that, make sure to like and subscribe, and I'll be giving away one of a kind Python Brain t shirt.
All you have to do is leave a comment below, then we'll pick a random winner via livestream next week.
This video was made in collaboration with Roddy Cholakov, who is, get this, only in the 7th grade and already doing crazy stuff with TensorFlow, Firebase, and a bunch of other tech.
In addition to this video, you'll also want to check out his GitHub repo that shows you how to train a TensorFlow model directly in a Cloud Function.
Now we're ready to Started and we'll do something that we've never done on this channel before and that's write Python code.
Python is the most commonly used language for machine learning problems and it's pretty easy to learn if you already know script.
If you wanna get inspired by Python, head over to Kaggle and check out some of the amazing work done by real data scientists.
In today's video, you'll learn not only how to and train a machine learning model, but also how to deploy it to the Cloud as a real product that people can use.
The first step in any machine learning problem is the underlying data.
A practical use case for a Firebase developer would be to save some information about a user in the Firestore database, Then train that data to build a predictive model that adds some value to the user experience.
If we look in our database, you can see here we have a collection of developer records and each one will tell us what Tools this developer uses and their overall happiness as a developer.
This is just synthetic data that I generated for this demo, but you can think of it as being information that you might collect from a user survey or something along those lines.
Machine learning problems generally require a huge amount of data and also a huge amount of compute resources to run operations on that data.
We're going to use a service on GCP called Datalab.
It allows us to stream and process our data in the Cloud, which means you can spin up a GPU powered supercomputer from your top and then just shut it off when you're done.
To get going with Datalab, you'll need to have a Firebase or GCP project, and then we have a couple of things to run here from the command line.
Basically, you're just installing Datalab and then calling Datalab create with the name that you want to give it.
What this does is give you an interactive TIFF Python or Jupyter notebook in the Cloud that you can use to run Python code.
And you can allocate as much CPU or memory that you need and also attach GPUs to it.
The nice thing about this notebook is that it's tightly integrated with all of your other GCP resources such as storage buckets, BigQuery, and everything else.
It should give you a link to localhost that will show a notebook that looks like this.
The first thing I'm going to do is install Firebase Admin by running exclamation pip install Firebase Admin.
Now if you're not familiar with Jupyter Notebooks, they're basically just a nice UI for executing Python code.
We can put each code block in its own cell and then Execute those 1 by 1 by clicking the play button.
You'll notice that there are a ton of examples in the Datalab environment, so definitely check those out, especially if you're not super comfortable with Python.
The notebook we're building today does 4 main things.
1st, it brings in our data from Firestore.
Then we do an exploratory data analysis.
Then we split the data up and train an algorithm.
And then finally, export it to a Cloud storage bucket.
1st, we'll need access to Firestore.
So we'll go into the Firebase project settings and generate a new private key.
That'll give you a JSON file that you can save in the root of the Datalab.
There are 2 dependencies that you'll see commonly in machine learning problems.
1 is Pandas, which allows us to put data in a DataFrame.
And the second one is NumPy, which helps us make scientific calculations on arrays.
What we'll be doing in just a second here is taking our Firestore data and putting it into a Pandas data frame.
And NumPy will help us generate some random synthetic data that we can use for this demo.
From there, we have a library called pyplot that will help us generate some charts, then we'll import Firebase Admin and initialize it with our credentials.
The process is basically identical to Node.
Js or any other language for that matter.
This next part shows how I generated the synthetic data and I'm not going to go through it line by line, but just Explain what's going on at a high level.
Almost all machine learning problems boil down to finding a signal that's hidden within some noise.
So the data I've generated below is slightly random, but also has some structure to it as well.
The value that we're trying to predict is developer happiness.
So I started by giving every user a random happiness between negative 2525.
Then I have a dictionary with 6 different programming languages in it, each one that will give the developer a different level of happiness.
And I also gave each developer a value of Firebase, which when true makes the developer much happier than when false.
But the general idea here is that the data is partially random and partially fixed.
Our goal is to create an algorithm that can detect which features are important and which aren't.
In this example, Firebase should be the most important followed by programming language and then age shouldn't be important at all.
If you have your own data in Firestore, you can just skip this step.
And then in the next cell, I will actually retrieve that data from the database and then save it to a CSV file, we don't have to reread it each time and get charged for additional reads in Firestore.
Conveniently, Pandas has a 2 CSV method that we can use to Just quickly save our data to a CSV file.
So now if we come back to our notebook, we can just run pandas read CSV.
Then the first thing I'll do is just sample 10 items from that data frame to see what the data looks like.
Machine learning is not magic.
And the first thing you should do once you have your data is run an exploratory analysis.
The idea is to uncover some signals here that you can use to nudge your algorithm in the right direction.
And the reason we put our data in a Pandas DataFrame is because it gives us a whole bunch of methods to help analyze and also clean the data before it goes into an algorithm.
So here we're running DataFrame sample to just get 10 random rows out of the dataset.
And a DataFrame is treated very much like an array.
You can see here that I'm doing DataFrame with the happiness key on there.
And then I'm gonna run plot histogram to give us a visual idea of what the actual label looks like.
So you can see here we have a pretty normal distribution of happiness values.
And this is what we want our algorithm to be able to predict based on the other features in the data.
Another thing you can do is run DataFrame described to get common statistics on the dataset such as the mean, standard deviation, and things like that.
So now that we've explored our data, it's time to introduce a new library called scikit learn.
You've probably heard a lot of hype surrounding TensorFlow and deep neural networks and those Tools are great, but they're often overkill for a lot of problems.
Scikit learn contains a huge collection of tools for tackling machine learning problems.
The first thing we'll do is use it to clean our data to make sure that all the values are numeric, which is required for most machine learning algorithms.
Currently, you'll notice that our programming language value is a string value.
So what we need to do is get all of the unique elements in that array and convert them to a number, Then assign each value to the corresponding number.
We can do this easily with scikit learn by creating a label encoder and then having it fit transform that column to the numeric classes.
If we sample the data frame again, you'll see that our language column is now numeric and no longer a string.
The next thing scikit learn will help us with is slicing the data into different chunks that we can use to train an algorithm.
We'll define a new variable named x which is all of our features in the data minus the one we're trying to predict which is happiness.
Then we'll define y as the variable that we are trying to predict.
Now we need to separate these into sets for training and then another one for testing or validation.
So I'm using this TrainTest Split method, passing it our x y values and then making the test size be 1 third of the total dataset size, which means we'll be training on 66% of the data.
The reason you don't train all the data is because then you have no way to validate whether or not the model is any good, and you want a model that can kind of generalize the overall dataset.
So now we're going to train an algorithm that's usually very good out of the box, which is a random forest.
In our case, we have a value between negative 100 and positive 100.
And the predicted value can be anything in between there.
So we have an infinite number of possible predictions, which means we're dealing with a regression problem.
This differs from a classification problem where you might be predicting different labels like whether or not a picture is a dog or a cat.
The code in this last cell is very simple.
We just have the Random Forest Regressor, which we fit to our x and y data.
And then we predict on the features in the test set of that data.
Yeah.
There are a bunch of different options you can pass to the regressor to try to fine tune it.
That's called hyperparameter tuning.
And then you need to have a metric to analyze its performance.
In this case, we're going to use Mean Absolute Error which means that our Random Forest Model is about 18 points wrong on average.
So how good is that exactly?
Well, the best place to start is just by comparing it to a random set of predictions.
In the next cell, we can see that completely random predictions produce A mean absolute error of about 56 which means that our algorithm is relatively effective.
Now the last thing we wanna do is figure out which features in the data were most important the decision making process.
As you can see here, the age was not important at all because it was completely random.
And then Firebase and programming languages were the most important.
So at this point, we have a predictive model, but we have no way for anybody to use it.
If you're building a product, you most likely want to serve that up as an API so it can be consumed by some kind of front end application like Angular or iOS or whatever.
And that's where ML Engine comes in.
The last cell in the notebook that you're looking Here is taking the model that we just trained, dumping it as a job lib file, and then saving it in our Firebase storage bucket.
ML Engine will be able to read the model file in the bucket and then serve out predictions in a highly performant way with predictions being served over the network in just a matter of a few 100 milliseconds.
So before we head over to ML Engine, go into your Firebase Storage Bucket and make sure that you have your job load file saved in its own individual folder.
From there, you'll wanna go back to the GCP console and then go to enable APIs.
You'll need to enable the Cloud Build API because your models will should be versioned.
And, of course, we'll need the ML Engine API as well.
From there, you can navigate over to the Cloud ML Engine dashboard.
And the first thing I wanna do is create a model and give that model a name.
You'll need this name later when we get to the Cloud Function.
The great thing about this system is that Your models can have multiple versions, so you can see how their performance changes over time.
It's important that you fill out things correctly here like our Python version as 2.7 in the environment And we're using the scikit learn framework.
And the version is 19 and the runtime version is 1.9.
From there, you'll need to select the folder that has this model in it which is that job load file that we saved earlier.
It'll take a few minutes to save.
But once that's done, you'll be able to either make predictions from the command line or from other Google APIs such as a Cloud function as we'll see here next.
Once the model goes into production, we'll be able to get some actual performance metrics about how it's handling its workload.
So now we're on to the final step and that's writing a Cloud Function that can consume this model.
For that, we'll switch over to Versus Code.
And first, we'll go into the command line, run Firebase init functions, Then we'll c d into the functions directory and install Google APIs.
The Google APIs package for Node will allow us to authenticate the Cloud Function and then make requests to the ML API without the need for an HTTP library.
I'm using TypeScript here, but you could just as easily use vanilla JS.
We'll import Google from Google APIs, then reference ML version 1.
The function itself is called predict happiness, and it's just a regular HTTPS function.
But you could use any kind of Cloud Function here you want.
A good alternative would be to run a prediction every time a new record is added to the Firestore database.
We're going to use this single Cloud function to make requests to potentially multiple models.
So we'll have the request body define which model should make the prediction.
And then the instances will be the actual data that we're trying to predict, which is typically an array with the same shape in which you train your actual Machine Learning model.
So now that we have the request body, we can format our request to the Google API.
This request requires authentication, which we can get by calling Google auth get application default.
And that will connect our Firebase credentials with the Google API.
From there, we'll set up a variable for the model name, which needs to have your project ID in it as well as the name of the actual model that you gave it in the ML Engine console.
And now everything is in place to make a prediction call to the ML Engine API.
We call projects predict, pass in our auth credentials, the name of the model, and then also the instances that we're trying to predict.
This can actually be an array of arrays if we want to make multiple predictions from a single Cloud Function call.
And then TypeScript gets a little confused right here, so we need to add as any after the argument to predict.
So that Should resolve with the predictions from our model, and then we can send that back out as the response from this HTTP function.
At this point, you could deploy this Cloud function and you essentially an API endpoint with your machine learning model, but we're just going to run it locally.
And then we'll test it with an application called Insomnia.
Insomnia is a free desktop app that you can install and it allows you to make HTTP calls in this nice user interface.
So first in the JSON body, we'll declare the model name, which is developer happiness.
For the instances, we pass an array of arrays where each array is going to be a prediction that we're trying to make.
So these values match the features in our original dataset.
For example, column 1 would be the user's age, column 2, whether or not they use Firebase, and column 3, their encoded programming language.
Now if we send this request to our function, we get a response back in about 400 milliseconds that gives us the prediction from those instances.
So each set of features sent from the request here on the left got a different happiness prediction from the response here on the right.
In other words, we started with basically nothing, And now we have an endpoint in the Cloud for our custom machine learning model.
I'm gonna go ahead and wrap things up there.
If this video helped you, please like and subscribe and you can find the full source code on angularfirebase.com.
If you're serious about building a project with Firebase or Google Cloud, consider upgrading to a Pro membership at angularfirebase.com.
You'll get access to a whole bunch of exclusive content designed to help you build and ship your app faster.
Thanks for watching and I'll talk to you soon.

Speaker 0: Today, we'll start from 0 and finish the video with our own custom machine learning model deployed to the cloud as an API. We will pull some actual user data that we have saved in the Firestore database, then we'll export it to Datalab so we can train a Python based machine learning model with virtually unlimited compute resources. Once we have a solid model in place we'll deploy it to ML engine so we can version it and analyze it in production, and finally we will Expose it to the universe by making it available in a Firebase cloud function. But before you do that, make sure to like and subscribe, and I'll be giving away one of a kind Python Brain t shirt. All you have to do is leave a comment below, then we'll pick a random winner via livestream next week.

This video was made in collaboration with Roddy Cholakov, who is, get this, only in the 7th grade and already doing crazy stuff with TensorFlow, Firebase, and a bunch of other tech. In addition to this video, you'll also want to check out his GitHub repo that shows you how to train a TensorFlow model directly in a Cloud Function. Now we're ready to Started and we'll do something that we've never done on this channel before and that's write Python code. Python is the most commonly used language for machine learning problems and it's pretty easy to learn if you already know script. If you wanna get inspired by Python, head over to Kaggle and check out some of the amazing work done by real data scientists.

In today's video, you'll learn not only how to and train a machine learning model, but also how to deploy it to the Cloud as a real product that people can use. The first step in any machine learning problem is the underlying data. A practical use case for a Firebase developer would be to save some information about a user in the Firestore database, Then train that data to build a predictive model that adds some value to the user experience. If we look in our database, you can see here we have a collection of developer records and each one will tell us what Tools this developer uses and their overall happiness as a developer. This is just synthetic data that I generated for this demo, but you can think of it as being information that you might collect from a user survey or something along those lines.

Machine learning problems generally require a huge amount of data and also a huge amount of compute resources to run operations on that data. We're going to use a service on GCP called Datalab. It allows us to stream and process our data in the Cloud, which means you can spin up a GPU powered supercomputer from your top and then just shut it off when you're done. To get going with Datalab, you'll need to have a Firebase or GCP project, and then we have a couple of things to run here from the command line. Basically, you're just installing Datalab and then calling Datalab create with the name that you want to give it.

What this does is give you an interactive TIFF Python or Jupyter notebook in the Cloud that you can use to run Python code. And you can allocate as much CPU or memory that you need and also attach GPUs to it. The nice thing about this notebook is that it's tightly integrated with all of your other GCP resources such as storage buckets, BigQuery, and everything else. It should give you a link to localhost that will show a notebook that looks like this. The first thing I'm going to do is install Firebase Admin by running exclamation pip install Firebase Admin.

Now if you're not familiar with Jupyter Notebooks, they're basically just a nice UI for executing Python code. We can put each code block in its own cell and then Execute those 1 by 1 by clicking the play button. You'll notice that there are a ton of examples in the Datalab environment, so definitely check those out, especially if you're not super comfortable with Python. The notebook we're building today does 4 main things. 1st, it brings in our data from Firestore.

Then we do an exploratory data analysis. Then we split the data up and train an algorithm. And then finally, export it to a Cloud storage bucket. 1st, we'll need access to Firestore. So we'll go into the Firebase project settings and generate a new private key.

That'll give you a JSON file that you can save in the root of the Datalab. There are 2 dependencies that you'll see commonly in machine learning problems. 1 is Pandas, which allows us to put data in a DataFrame. And the second one is NumPy, which helps us make scientific calculations on arrays. What we'll be doing in just a second here is taking our Firestore data and putting it into a Pandas data frame.

And NumPy will help us generate some random synthetic data that we can use for this demo. From there, we have a library called pyplot that will help us generate some charts, then we'll import Firebase Admin and initialize it with our credentials. The process is basically identical to Node. Js or any other language for that matter. This next part shows how I generated the synthetic data and I'm not going to go through it line by line, but just Explain what's going on at a high level.

Almost all machine learning problems boil down to finding a signal that's hidden within some noise. So the data I've generated below is slightly random, but also has some structure to it as well. The value that we're trying to predict is developer happiness. So I started by giving every user a random happiness between negative 2525. Then I have a dictionary with 6 different programming languages in it, each one that will give the developer a different level of happiness.

And I also gave each developer a value of Firebase, which when true makes the developer much happier than when false. But the general idea here is that the data is partially random and partially fixed. Our goal is to create an algorithm that can detect which features are important and which aren't. In this example, Firebase should be the most important followed by programming language and then age shouldn't be important at all. If you have your own data in Firestore, you can just skip this step.

And then in the next cell, I will actually retrieve that data from the database and then save it to a CSV file, we don't have to reread it each time and get charged for additional reads in Firestore. Conveniently, Pandas has a 2 CSV method that we can use to Just quickly save our data to a CSV file. So now if we come back to our notebook, we can just run pandas read CSV. Then the first thing I'll do is just sample 10 items from that data frame to see what the data looks like. Machine learning is not magic.

And the first thing you should do once you have your data is run an exploratory analysis. The idea is to uncover some signals here that you can use to nudge your algorithm in the right direction. And the reason we put our data in a Pandas DataFrame is because it gives us a whole bunch of methods to help analyze and also clean the data before it goes into an algorithm. So here we're running DataFrame sample to just get 10 random rows out of the dataset. And a DataFrame is treated very much like an array.

You can see here that I'm doing DataFrame with the happiness key on there. And then I'm gonna run plot histogram to give us a visual idea of what the actual label looks like. So you can see here we have a pretty normal distribution of happiness values. And this is what we want our algorithm to be able to predict based on the other features in the data. Another thing you can do is run DataFrame described to get common statistics on the dataset such as the mean, standard deviation, and things like that.

So now that we've explored our data, it's time to introduce a new library called scikit learn. You've probably heard a lot of hype surrounding TensorFlow and deep neural networks and those Tools are great, but they're often overkill for a lot of problems. Scikit learn contains a huge collection of tools for tackling machine learning problems. The first thing we'll do is use it to clean our data to make sure that all the values are numeric, which is required for most machine learning algorithms. Currently, you'll notice that our programming language value is a string value.

So what we need to do is get all of the unique elements in that array and convert them to a number, Then assign each value to the corresponding number. We can do this easily with scikit learn by creating a label encoder and then having it fit transform that column to the numeric classes. If we sample the data frame again, you'll see that our language column is now numeric and no longer a string. The next thing scikit learn will help us with is slicing the data into different chunks that we can use to train an algorithm. We'll define a new variable named x which is all of our features in the data minus the one we're trying to predict which is happiness.

Then we'll define y as the variable that we are trying to predict. Now we need to separate these into sets for training and then another one for testing or validation. So I'm using this TrainTest Split method, passing it our x y values and then making the test size be 1 third of the total dataset size, which means we'll be training on 66% of the data. The reason you don't train all the data is because then you have no way to validate whether or not the model is any good, and you want a model that can kind of generalize the overall dataset. So now we're going to train an algorithm that's usually very good out of the box, which is a random forest.

In our case, we have a value between negative 100 and positive 100. And the predicted value can be anything in between there. So we have an infinite number of possible predictions, which means we're dealing with a regression problem. This differs from a classification problem where you might be predicting different labels like whether or not a picture is a dog or a cat. The code in this last cell is very simple.

We just have the Random Forest Regressor, which we fit to our x and y data. And then we predict on the features in the test set of that data. Yeah. There are a bunch of different options you can pass to the regressor to try to fine tune it. That's called hyperparameter tuning.

And then you need to have a metric to analyze its performance. In this case, we're going to use Mean Absolute Error which means that our Random Forest Model is about 18 points wrong on average. So how good is that exactly? Well, the best place to start is just by comparing it to a random set of predictions. In the next cell, we can see that completely random predictions produce A mean absolute error of about 56 which means that our algorithm is relatively effective.

Now the last thing we wanna do is figure out which features in the data were most important the decision making process. As you can see here, the age was not important at all because it was completely random. And then Firebase and programming languages were the most important. So at this point, we have a predictive model, but we have no way for anybody to use it. If you're building a product, you most likely want to serve that up as an API so it can be consumed by some kind of front end application like Angular or iOS or whatever.

And that's where ML Engine comes in. The last cell in the notebook that you're looking Here is taking the model that we just trained, dumping it as a job lib file, and then saving it in our Firebase storage bucket. ML Engine will be able to read the model file in the bucket and then serve out predictions in a highly performant way with predictions being served over the network in just a matter of a few 100 milliseconds. So before we head over to ML Engine, go into your Firebase Storage Bucket and make sure that you have your job load file saved in its own individual folder. From there, you'll wanna go back to the GCP console and then go to enable APIs.

You'll need to enable the Cloud Build API because your models will should be versioned. And, of course, we'll need the ML Engine API as well. From there, you can navigate over to the Cloud ML Engine dashboard. And the first thing I wanna do is create a model and give that model a name. You'll need this name later when we get to the Cloud Function.

The great thing about this system is that Your models can have multiple versions, so you can see how their performance changes over time. It's important that you fill out things correctly here like our Python version as 2.7 in the environment And we're using the scikit learn framework. And the version is 19 and the runtime version is 1.9. From there, you'll need to select the folder that has this model in it which is that job load file that we saved earlier. It'll take a few minutes to save.

But once that's done, you'll be able to either make predictions from the command line or from other Google APIs such as a Cloud function as we'll see here next. Once the model goes into production, we'll be able to get some actual performance metrics about how it's handling its workload. So now we're on to the final step and that's writing a Cloud Function that can consume this model. For that, we'll switch over to Versus Code. And first, we'll go into the command line, run Firebase init functions, Then we'll c d into the functions directory and install Google APIs.

The Google APIs package for Node will allow us to authenticate the Cloud Function and then make requests to the ML API without the need for an HTTP library. I'm using TypeScript here, but you could just as easily use vanilla JS. We'll import Google from Google APIs, then reference ML version 1. The function itself is called predict happiness, and it's just a regular HTTPS function. But you could use any kind of Cloud Function here you want.

A good alternative would be to run a prediction every time a new record is added to the Firestore database. We're going to use this single Cloud function to make requests to potentially multiple models. So we'll have the request body define which model should make the prediction. And then the instances will be the actual data that we're trying to predict, which is typically an array with the same shape in which you train your actual Machine Learning model. So now that we have the request body, we can format our request to the Google API.

This request requires authentication, which we can get by calling Google auth get application default. And that will connect our Firebase credentials with the Google API. From there, we'll set up a variable for the model name, which needs to have your project ID in it as well as the name of the actual model that you gave it in the ML Engine console. And now everything is in place to make a prediction call to the ML Engine API. We call projects predict, pass in our auth credentials, the name of the model, and then also the instances that we're trying to predict.

This can actually be an array of arrays if we want to make multiple predictions from a single Cloud Function call. And then TypeScript gets a little confused right here, so we need to add as any after the argument to predict. So that Should resolve with the predictions from our model, and then we can send that back out as the response from this HTTP function. At this point, you could deploy this Cloud function and you essentially an API endpoint with your machine learning model, but we're just going to run it locally. And then we'll test it with an application called Insomnia.

Insomnia is a free desktop app that you can install and it allows you to make HTTP calls in this nice user interface. So first in the JSON body, we'll declare the model name, which is developer happiness. For the instances, we pass an array of arrays where each array is going to be a prediction that we're trying to make. So these values match the features in our original dataset. For example, column 1 would be the user's age, column 2, whether or not they use Firebase, and column 3, their encoded programming language.

Now if we send this request to our function, we get a response back in about 400 milliseconds that gives us the prediction from those instances. So each set of features sent from the request here on the left got a different happiness prediction from the response here on the right. In other words, we started with basically nothing, And now we have an endpoint in the Cloud for our custom machine learning model. I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe and you can find the full source code on angularfirebase.com.

If you're serious about building a project with Firebase or Google Cloud, consider upgrading to a Pro membership at angularfirebase.com. You'll get access to a whole bunch of exclusive content designed to help you build and ship your app faster. Thanks for watching and I'll talk to you soon.
Today, we'll start from 0 and finish the video with our own custom machine learning model deployed to the cloud as an API. We will pull some actual user data that we have saved in the Firestore database, then we'll export it to Datalab so we can train a Python based machine learning model with virtually unlimited compute resources. Once we have a solid model in place we'll deploy it to ML engine so we can version it and analyze it in production, and finally we will Expose it to the universe by making it available in a Firebase cloud function. But before you do that, make sure to like and subscribe, and I'll be giving away one of a kind Python Brain t shirt. All you have to do is leave a comment below, then we'll pick a random winner via livestream next week. This video was made in collaboration with Roddy Cholakov, who is, get this, only in the 7th grade and already doing crazy stuff with TensorFlow, Firebase, and a bunch of other tech. In addition to this video, you'll also want to check out his GitHub repo that shows you how to train a TensorFlow model directly in a Cloud Function. Now we're ready to Started and we'll do something that we've never done on this channel before and that's write Python code. Python is the most commonly used language for machine learning problems and it's pretty easy to learn if you already know script. If you wanna get inspired by Python, head over to Kaggle and check out some of the amazing work done by real data scientists. In today's video, you'll learn not only how to and train a machine learning model, but also how to deploy it to the Cloud as a real product that people can use. The first step in any machine learning problem is the underlying data. A practical use case for a Firebase developer would be to save some information about a user in the Firestore database, Then train that data to build a predictive model that adds some value to the user experience. If we look in our database, you can see here we have a collection of developer records and each one will tell us what Tools this developer uses and their overall happiness as a developer. This is just synthetic data that I generated for this demo, but you can think of it as being information that you might collect from a user survey or something along those lines. Machine learning problems generally require a huge amount of data and also a huge amount of compute resources to run operations on that data. We're going to use a service on GCP called Datalab. It allows us to stream and process our data in the Cloud, which means you can spin up a GPU powered supercomputer from your top and then just shut it off when you're done.
databases
machine learning
artificial intelligence
data
data management
deep learning
software development
youtube
data analysis
podcasts
neural networks
programming
software
To get going with Datalab, you'll need to have a Firebase or GCP project, and then we have a couple of things to run here from the command line.
Basically, you're just installing Datalab and then calling Datalab create with the name that you want to give it. What this does is give you an interactive TIFF Python or Jupyter notebook in the Cloud that you can use to run Python code. And you can allocate as much CPU or memory that you need and also attach GPUs to it. The nice thing about this notebook is that it's tightly integrated with all of your other GCP resources such as storage buckets, BigQuery, and everything else. It should give you a link to localhost that will show a notebook that looks like this.
computers
data management
software
software development
The first thing I'm going to do is install Firebase Admin by running exclamation pip install Firebase Admin.
Now if you're not familiar with Jupyter Notebooks, they're basically just a nice UI for executing Python code. We can put each code block in its own cell and then Execute those 1 by 1 by clicking the play button. You'll notice that there are a ton of examples in the Datalab environment, so definitely check those out, especially if you're not super comfortable with Python. The notebook we're building today does 4 main things. 1st, it brings in our data from Firestore. Then we do an exploratory data analysis. Then we split the data up and train an algorithm. And then finally, export it to a Cloud storage bucket. 1st, we'll need access to Firestore.
data analysis
data
programming
data management
databases
computer science
computers
So we'll go into the Firebase project settings and generate a new private key.
That'll give you a JSON file that you can save in the root of the Datalab. There are 2 dependencies that you'll see commonly in machine learning problems. 1 is Pandas, which allows us to put data in a DataFrame. And the second one is NumPy, which helps us make scientific calculations on arrays. What we'll be doing in just a second here is taking our Firestore data and putting it into a Pandas data frame. And NumPy will help us generate some random synthetic data that we can use for this demo. From there, we have a library called pyplot that will help us generate some charts, then we'll import Firebase Admin and initialize it with our credentials.
data
machine learning
data analysis
databases
The process is basically identical to Node. Js or any other language for that matter. This next part shows how I generated the synthetic data and I'm not going to go through it line by line, but just Explain what's going on at a high level. Almost all machine learning problems boil down to finding a signal that's hidden within some noise. So the data I've generated below is slightly random, but also has some structure to it as well. The value that we're trying to predict is developer happiness. So I started by giving every user a random happiness between negative 2525.
machine learning
artificial intelligence
probability
neural networks
Then I have a dictionary with 6 different programming languages in it, each one that will give the developer a different level of happiness. And I also gave each developer a value of Firebase, which when true makes the developer much happier than when false. But the general idea here is that the data is partially random and partially fixed. Our goal is to create an algorithm that can detect which features are important and which aren't. In this example, Firebase should be the most important followed by programming language and then age shouldn't be important at all. If you have your own data in Firestore, you can just skip this step. And then in the next cell, I will actually retrieve that data from the database and then save it to a CSV file, we don't have to reread it each time and get charged for additional reads in Firestore. Conveniently, Pandas has a 2 CSV method that we can use to Just quickly save our data to a CSV file. So now if we come back to our notebook, we can just run pandas read CSV.
Then the first thing I'll do is just sample 10 items from that data frame to see what the data looks like. Machine learning is not magic. And the first thing you should do once you have your data is run an exploratory analysis. The idea is to uncover some signals here that you can use to nudge your algorithm in the right direction. And the reason we put our data in a Pandas DataFrame is because it gives us a whole bunch of methods to help analyze and also clean the data before it goes into an algorithm. So here we're running DataFrame sample to just get 10 random rows out of the dataset. And a DataFrame is treated very much like an array. You can see here that I'm doing DataFrame with the happiness key on there. And then I'm gonna run plot histogram to give us a visual idea of what the actual label looks like.
machine learning
data
artificial intelligence
data analysis
data management
deep learning
So you can see here we have a pretty normal distribution of happiness values. And this is what we want our algorithm to be able to predict based on the other features in the data. Another thing you can do is run DataFrame described to get common statistics on the dataset such as the mean, standard deviation, and things like that. So now that we've explored our data, it's time to introduce a new library called scikit learn. You've probably heard a lot of hype surrounding TensorFlow and deep neural networks and those Tools are great, but they're often overkill for a lot of problems. Scikit learn contains a huge collection of tools for tackling machine learning problems. The first thing we'll do is use it to clean our data to make sure that all the values are numeric, which is required for most machine learning algorithms. Currently, you'll notice that our programming language value is a string value. So what we need to do is get all of the unique elements in that array and convert them to a number, Then assign each value to the corresponding number. We can do this easily with scikit learn by creating a label encoder and then having it fit transform that column to the numeric classes. If we sample the data frame again, you'll see that our language column is now numeric and no longer a string. The next thing scikit learn will help us with is slicing the data into different chunks that we can use to train an algorithm. We'll define a new variable named x which is all of our features in the data minus the one we're trying to predict which is happiness. Then we'll define y as the variable that we are trying to predict. Now we need to separate these into sets for training and then another one for testing or validation. So I'm using this TrainTest Split method, passing it our x y values and then making the test size be 1 third of the total dataset size, which means we'll be training on 66% of the data. The reason you don't train all the data is because then you have no way to validate whether or not the model is any good, and you want a model that can kind of generalize the overall dataset. So now we're going to train an algorithm that's usually very good out of the box, which is a random forest. In our case, we have a value between negative 100 and positive 100. And the predicted value can be anything in between there. So we have an infinite number of possible predictions, which means we're dealing with a regression problem. This differs from a classification problem where you might be predicting different labels like whether or not a picture is a dog or a cat.
statistics
machine learning
deep learning
programming
data analysis
data
probability
neural networks
computer science
artificial intelligence
The code in this last cell is very simple. We just have the Random Forest Regressor, which we fit to our x and y data.
And then we predict on the features in the test set of that data. Yeah. There are a bunch of different options you can pass to the regressor to try to fine tune it. That's called hyperparameter tuning. And then you need to have a metric to analyze its performance. In this case, we're going to use Mean Absolute Error which means that our Random Forest Model is about 18 points wrong on average. So how good is that exactly? Well, the best place to start is just by comparing it to a random set of predictions. In the next cell, we can see that completely random predictions produce A mean absolute error of about 56 which means that our algorithm is relatively effective.
probability
machine learning
artificial intelligence
neural networks
forests
measurement
Now the last thing we wanna do is figure out which features in the data were most important the decision making process. As you can see here, the age was not important at all because it was completely random. And then Firebase and programming languages were the most important. So at this point, we have a predictive model, but we have no way for anybody to use it. If you're building a product, you most likely want to serve that up as an API so it can be consumed by some kind of front end application like Angular or iOS or whatever.
And that's where ML Engine comes in. The last cell in the notebook that you're looking Here is taking the model that we just trained, dumping it as a job lib file, and then saving it in our Firebase storage bucket. ML Engine will be able to read the model file in the bucket and then serve out predictions in a highly performant way with predictions being served over the network in just a matter of a few 100 milliseconds. So before we head over to ML Engine, go into your Firebase Storage Bucket and make sure that you have your job load file saved in its own individual folder.
machine learning
neural networks
From there, you'll wanna go back to the GCP console and then go to enable APIs. You'll need to enable the Cloud Build API because your models will should be versioned. And, of course, we'll need the ML Engine API as well. From there, you can navigate over to the Cloud ML Engine dashboard. And the first thing I wanna do is create a model and give that model a name. You'll need this name later when we get to the Cloud Function. The great thing about this system is that Your models can have multiple versions, so you can see how their performance changes over time. It's important that you fill out things correctly here like our Python version as 2.7 in the environment And we're using the scikit learn framework. And the version is 19 and the runtime version is 1.9. From there, you'll need to select the folder that has this model in it which is that job load file that we saved earlier.
systems
software
machine learning
software development
microsoft
It'll take a few minutes to save. But once that's done, you'll be able to either make predictions from the command line or from other Google APIs such as a Cloud function as we'll see here next. Once the model goes into production, we'll be able to get some actual performance metrics about how it's handling its workload. So now we're on to the final step and that's writing a Cloud Function that can consume this model.
For that, we'll switch over to Versus Code. And first, we'll go into the command line, run Firebase init functions, Then we'll c d into the functions directory and install Google APIs. The Google APIs package for Node will allow us to authenticate the Cloud Function and then make requests to the ML API without the need for an HTTP library. I'm using TypeScript here, but you could just as easily use vanilla JS. We'll import Google from Google APIs, then reference ML version 1. The function itself is called predict happiness, and it's just a regular HTTPS function. But you could use any kind of Cloud Function here you want. A good alternative would be to run a prediction every time a new record is added to the Firestore database. We're going to use this single Cloud function to make requests to potentially multiple models. So we'll have the request body define which model should make the prediction. And then the instances will be the actual data that we're trying to predict, which is typically an array with the same shape in which you train your actual Machine Learning model. So now that we have the request body, we can format our request to the Google API. This request requires authentication, which we can get by calling Google auth get application default. And that will connect our Firebase credentials with the Google API. From there, we'll set up a variable for the model name, which needs to have your project ID in it as well as the name of the actual model that you gave it in the ML Engine console.
machine learning
modeling
google
artificial intelligence
neural networks
And now everything is in place to make a prediction call to the ML Engine API. We call projects predict, pass in our auth credentials, the name of the model, and then also the instances that we're trying to predict. This can actually be an array of arrays if we want to make multiple predictions from a single Cloud Function call.
machine learning
neural networks
modeling
artificial intelligence
And then TypeScript gets a little confused right here, so we need to add as any after the argument to predict.
So that Should resolve with the predictions from our model, and then we can send that back out as the response from this HTTP function. At this point, you could deploy this Cloud function and you essentially an API endpoint with your machine learning model, but we're just going to run it locally.
machine learning
And then we'll test it with an application called Insomnia. Insomnia is a free desktop app that you can install and it allows you to make HTTP calls in this nice user interface.
So first in the JSON body, we'll declare the model name, which is developer happiness. For the instances, we pass an array of arrays where each array is going to be a prediction that we're trying to make. So these values match the features in our original dataset. For example, column 1 would be the user's age, column 2, whether or not they use Firebase, and column 3, their encoded programming language. Now if we send this request to our function, we get a response back in about 400 milliseconds that gives us the prediction from those instances. So each set of features sent from the request here on the left got a different happiness prediction from the response here on the right. In other words, we started with basically nothing, And now we have an endpoint in the Cloud for our custom machine learning model. I'm gonna go ahead and wrap things up there.
machine learning
data
probability
data analysis
databases
If this video helped you, please like and subscribe and you can find the full source code on angularfirebase.com. If you're serious about building a project with Firebase or Google Cloud, consider upgrading to a Pro membership at angularfirebase.com. You'll get access to a whole bunch of exclusive content designed to help you build and ship your app faster. Thanks for watching and I'll talk to you soon.
streaming
software
multimedia
internet
software development
youtube
Today, we'll start from 0 and finish the video with our own custom machine learning model deployed to the cloud as an API. We will pull some actual user data that we have saved in the Firestore database, then we'll export it to Datalab so we can train a Python based machine learning model with virtually unlimited compute resources. Once we have a solid model in place we'll deploy it to ML engine so we can version it and analyze it in production, and finally we will Expose it to the universe by making it available in a Firebase cloud function. But before you do that, make sure to like and subscribe, and I'll be giving away one of a kind Python Brain t shirt. All you have to do is leave a comment below, then we'll pick a random winner via livestream next week. This video was made in collaboration with Roddy Cholakov, who is, get this, only in the 7th grade and already doing crazy stuff with TensorFlow, Firebase, and a bunch of other tech. In addition to this video, you'll also want to check out his GitHub repo that shows you how to train a TensorFlow model directly in a Cloud Function. Now we're ready to Started and we'll do something that we've never done on this channel before and that's write Python code. Python is the most commonly used language for machine learning problems and it's pretty easy to learn if you already know script. If you wanna get inspired by Python, head over to Kaggle and check out some of the amazing work done by real data scientists. In today's video, you'll learn not only how to and train a machine learning model, but also how to deploy it to the Cloud as a real product that people can use. The first step in any machine learning problem is the underlying data. A practical use case for a Firebase developer would be to save some information about a user in the Firestore database, Then train that data to build a predictive model that adds some value to the user experience. If we look in our database, you can see here we have a collection of developer records and each one will tell us what Tools this developer uses and their overall happiness as a developer. This is just synthetic data that I generated for this demo, but you can think of it as being information that you might collect from a user survey or something along those lines. Machine learning problems generally require a huge amount of data and also a huge amount of compute resources to run operations on that data. We're going to use a service on GCP called Datalab. It allows us to stream and process our data in the Cloud, which means you can spin up a GPU powered supercomputer from your top and then just shut it off when you're done. To get going with Datalab, you'll need to have a Firebase or GCP project, and then we have a couple of things to run here from the command line. Basically, you're just installing Datalab and then calling Datalab create with the name that you want to give it. What this does is give you an interactive TIFF Python or Jupyter notebook in the Cloud that you can use to run Python code. And you can allocate as much CPU or memory that you need and also attach GPUs to it. The nice thing about this notebook is that it's tightly integrated with all of your other GCP resources such as storage buckets, BigQuery, and everything else. It should give you a link to localhost that will show a notebook that looks like this. The first thing I'm going to do is install Firebase Admin by running exclamation pip install Firebase Admin. Now if you're not familiar with Jupyter Notebooks, they're basically just a nice UI for executing Python code. We can put each code block in its own cell and then Execute those 1 by 1 by clicking the play button. You'll notice that there are a ton of examples in the Datalab environment, so definitely check those out, especially if you're not super comfortable with Python. The notebook we're building today does 4 main things. 1st, it brings in our data from Firestore. Then we do an exploratory data analysis. Then we split the data up and train an algorithm. And then finally, export it to a Cloud storage bucket. 1st, we'll need access to Firestore. So we'll go into the Firebase project settings and generate a new private key. That'll give you a JSON file that you can save in the root of the Datalab. There are 2 dependencies that you'll see commonly in machine learning problems. 1 is Pandas, which allows us to put data in a DataFrame. And the second one is NumPy, which helps us make scientific calculations on arrays. What we'll be doing in just a second here is taking our Firestore data and putting it into a Pandas data frame. And NumPy will help us generate some random synthetic data that we can use for this demo. From there, we have a library called pyplot that will help us generate some charts, then we'll import Firebase Admin and initialize it with our credentials. The process is basically identical to Node. Js or any other language for that matter. This next part shows how I generated the synthetic data and I'm not going to go through it line by line, but just Explain what's going on at a high level. Almost all machine learning problems boil down to finding a signal that's hidden within some noise. So the data I've generated below is slightly random, but also has some structure to it as well. The value that we're trying to predict is developer happiness. So I started by giving every user a random happiness between negative 2525. Then I have a dictionary with 6 different programming languages in it, each one that will give the developer a different level of happiness. And I also gave each developer a value of Firebase, which when true makes the developer much happier than when false. But the general idea here is that the data is partially random and partially fixed. Our goal is to create an algorithm that can detect which features are important and which aren't. In this example, Firebase should be the most important followed by programming language and then age shouldn't be important at all. If you have your own data in Firestore, you can just skip this step. And then in the next cell, I will actually retrieve that data from the database and then save it to a CSV file, we don't have to reread it each time and get charged for additional reads in Firestore. Conveniently, Pandas has a 2 CSV method that we can use to Just quickly save our data to a CSV file. So now if we come back to our notebook, we can just run pandas read CSV. Then the first thing I'll do is just sample 10 items from that data frame to see what the data looks like. Machine learning is not magic. And the first thing you should do once you have your data is run an exploratory analysis. The idea is to uncover some signals here that you can use to nudge your algorithm in the right direction. And the reason we put our data in a Pandas DataFrame is because it gives us a whole bunch of methods to help analyze and also clean the data before it goes into an algorithm. So here we're running DataFrame sample to just get 10 random rows out of the dataset. And a DataFrame is treated very much like an array. You can see here that I'm doing DataFrame with the happiness key on there. And then I'm gonna run plot histogram to give us a visual idea of what the actual label looks like. So you can see here we have a pretty normal distribution of happiness values. And this is what we want our algorithm to be able to predict based on the other features in the data. Another thing you can do is run DataFrame described to get common statistics on the dataset such as the mean, standard deviation, and things like that. So now that we've explored our data, it's time to introduce a new library called scikit learn. You've probably heard a lot of hype surrounding TensorFlow and deep neural networks and those Tools are great, but they're often overkill for a lot of problems. Scikit learn contains a huge collection of tools for tackling machine learning problems. The first thing we'll do is use it to clean our data to make sure that all the values are numeric, which is required for most machine learning algorithms. Currently, you'll notice that our programming language value is a string value. So what we need to do is get all of the unique elements in that array and convert them to a number, Then assign each value to the corresponding number. We can do this easily with scikit learn by creating a label encoder and then having it fit transform that column to the numeric classes. If we sample the data frame again, you'll see that our language column is now numeric and no longer a string. The next thing scikit learn will help us with is slicing the data into different chunks that we can use to train an algorithm. We'll define a new variable named x which is all of our features in the data minus the one we're trying to predict which is happiness. Then we'll define y as the variable that we are trying to predict. Now we need to separate these into sets for training and then another one for testing or validation. So I'm using this TrainTest Split method, passing it our x y values and then making the test size be 1 third of the total dataset size, which means we'll be training on 66% of the data. The reason you don't train all the data is because then you have no way to validate whether or not the model is any good, and you want a model that can kind of generalize the overall dataset. So now we're going to train an algorithm that's usually very good out of the box, which is a random forest. In our case, we have a value between negative 100 and positive 100. And the predicted value can be anything in between there. So we have an infinite number of possible predictions, which means we're dealing with a regression problem. This differs from a classification problem where you might be predicting different labels like whether or not a picture is a dog or a cat. The code in this last cell is very simple. We just have the Random Forest Regressor, which we fit to our x and y data. And then we predict on the features in the test set of that data. Yeah. There are a bunch of different options you can pass to the regressor to try to fine tune it. That's called hyperparameter tuning. And then you need to have a metric to analyze its performance. In this case, we're going to use Mean Absolute Error which means that our Random Forest Model is about 18 points wrong on average. So how good is that exactly? Well, the best place to start is just by comparing it to a random set of predictions. In the next cell, we can see that completely random predictions produce A mean absolute error of about 56 which means that our algorithm is relatively effective. Now the last thing we wanna do is figure out which features in the data were most important the decision making process. As you can see here, the age was not important at all because it was completely random. And then Firebase and programming languages were the most important. So at this point, we have a predictive model, but we have no way for anybody to use it. If you're building a product, you most likely want to serve that up as an API so it can be consumed by some kind of front end application like Angular or iOS or whatever. And that's where ML Engine comes in. The last cell in the notebook that you're looking Here is taking the model that we just trained, dumping it as a job lib file, and then saving it in our Firebase storage bucket. ML Engine will be able to read the model file in the bucket and then serve out predictions in a highly performant way with predictions being served over the network in just a matter of a few 100 milliseconds. So before we head over to ML Engine, go into your Firebase Storage Bucket and make sure that you have your job load file saved in its own individual folder. From there, you'll wanna go back to the GCP console and then go to enable APIs. You'll need to enable the Cloud Build API because your models will should be versioned. And, of course, we'll need the ML Engine API as well. From there, you can navigate over to the Cloud ML Engine dashboard. And the first thing I wanna do is create a model and give that model a name. You'll need this name later when we get to the Cloud Function. The great thing about this system is that Your models can have multiple versions, so you can see how their performance changes over time. It's important that you fill out things correctly here like our Python version as 2.7 in the environment And we're using the scikit learn framework. And the version is 19 and the runtime version is 1.9. From there, you'll need to select the folder that has this model in it which is that job load file that we saved earlier. It'll take a few minutes to save. But once that's done, you'll be able to either make predictions from the command line or from other Google APIs such as a Cloud function as we'll see here next. Once the model goes into production, we'll be able to get some actual performance metrics about how it's handling its workload. So now we're on to the final step and that's writing a Cloud Function that can consume this model. For that, we'll switch over to Versus Code. And first, we'll go into the command line, run Firebase init functions, Then we'll c d into the functions directory and install Google APIs. The Google APIs package for Node will allow us to authenticate the Cloud Function and then make requests to the ML API without the need for an HTTP library. I'm using TypeScript here, but you could just as easily use vanilla JS. We'll import Google from Google APIs, then reference ML version 1. The function itself is called predict happiness, and it's just a regular HTTPS function. But you could use any kind of Cloud Function here you want. A good alternative would be to run a prediction every time a new record is added to the Firestore database. We're going to use this single Cloud function to make requests to potentially multiple models. So we'll have the request body define which model should make the prediction. And then the instances will be the actual data that we're trying to predict, which is typically an array with the same shape in which you train your actual Machine Learning model. So now that we have the request body, we can format our request to the Google API. This request requires authentication, which we can get by calling Google auth get application default. And that will connect our Firebase credentials with the Google API. From there, we'll set up a variable for the model name, which needs to have your project ID in it as well as the name of the actual model that you gave it in the ML Engine console. And now everything is in place to make a prediction call to the ML Engine API. We call projects predict, pass in our auth credentials, the name of the model, and then also the instances that we're trying to predict. This can actually be an array of arrays if we want to make multiple predictions from a single Cloud Function call. And then TypeScript gets a little confused right here, so we need to add as any after the argument to predict. So that Should resolve with the predictions from our model, and then we can send that back out as the response from this HTTP function. At this point, you could deploy this Cloud function and you essentially an API endpoint with your machine learning model, but we're just going to run it locally. And then we'll test it with an application called Insomnia. Insomnia is a free desktop app that you can install and it allows you to make HTTP calls in this nice user interface. So first in the JSON body, we'll declare the model name, which is developer happiness. For the instances, we pass an array of arrays where each array is going to be a prediction that we're trying to make. So these values match the features in our original dataset. For example, column 1 would be the user's age, column 2, whether or not they use Firebase, and column 3, their encoded programming language. Now if we send this request to our function, we get a response back in about 400 milliseconds that gives us the prediction from those instances. So each set of features sent from the request here on the left got a different happiness prediction from the response here on the right. In other words, we started with basically nothing, And now we have an endpoint in the Cloud for our custom machine learning model. I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe and you can find the full source code on angularfirebase.com. If you're serious about building a project with Firebase or Google Cloud, consider upgrading to a Pro membership at angularfirebase.com. You'll get access to a whole bunch of exclusive content designed to help you build and ship your app faster. Thanks for watching and I'll talk to you soon.
Today,
today
we'll
we'll
start
start
from
from
0
0
and
and
finish
finish
the
the
video
video
with
with
our
our
own
own
custom
custom
machine
machine
learning
learning
model
model
deployed
deployed
to
to
the
the
cloud
cloud
as
as
an
an
API.
api
We
we
will
will
pull
pull
some
some
actual
actual
user
user
data
data
that
that
we
we
have
have
saved
saved
in
in
the
the
Firestore
firestore
database,
database
then
then
we'll
we'll
export
export
it
it
to
to
Datalab
datalab
so
so
we
we
can
can
train
train
a
a
Python
python
based
based
machine
machine
learning
learning
model
model
with
with
virtually
virtually
unlimited
unlimited
compute
compute
resources.
resources
Once
once
we
we
have
have
a
a
solid
solid
model
model
in
in
place
place
we'll
we'll
deploy
deploy
it
it
to
to
ML
ml
engine
engine
so
so
we
we
can
can
version
version
it
it
and
and
analyze
analyze
it
it
in
in
production,
production
and
and
finally
finally
we
we
will
will
Expose
expose
it
it
to
to
the
the
universe
universe
by
by
making
making
it
it
available
available
in
in
a
a
Firebase
firebase
cloud
cloud
function.
function
But
but
before
before
you
you
do
do
that,
that
make
make
sure
sure
to
to
like
like
and
and
subscribe,
subscribe
and
and
I'll
i'll
be
be
giving
giving
away
away
one
one
of
of
a
a
kind
kind
Python
python
Brain
brain
t
t
shirt.
shirt
All
all
you
you
have
have
to
to
do
do
is
is
leave
leave
a
a
comment
comment
below,
below
then
then
we'll
we'll
pick
pick
a
a
random
random
winner
winner
via
via
livestream
livestream
next
next
week.
week
This
this
video
video
was
was
made
made
in
in
collaboration
collaboration
with
with
Roddy
roddy
Cholakov,
cholakov
who
who
is,
is
get
get
this,
this
only
only
in
in
the
the
7th
7th
grade
grade
and
and
already
already
doing
doing
crazy
crazy
stuff
stuff
with
with
TensorFlow,
tensorflow
Firebase,
firebase
and
and
a
a
bunch
bunch
of
of
other
other
tech.
tech
In
in
addition
addition
to
to
this
this
video,
video
you'll
you'll
also
also
want
want
to
to
check
check
out
out
his
his
GitHub
github
repo
repo
that
that
shows
shows
you
you
how
how
to
to
train
train
a
a
TensorFlow
tensorflow
model
model
directly
directly
in
in
a
a
Cloud
cloud
Function.
function
Now
now
we're
we're
ready
ready
to
to
Started
started
and
and
we'll
we'll
do
do
something
something
that
that
we've
we've
never
never
done
done
on
on
this
this
channel
channel
before
before
and
and
that's
that's
write
write
Python
python
code.
code
Python
python
is
is
the
the
most
most
commonly
commonly
used
used
language
language
for
for
machine
machine
learning
learning
problems
problems
and
and
it's
it's
pretty
pretty
easy
easy
to
to
learn
learn
if
if
you
you
already
already
know
know
script.
script
If
if
you
you
wanna
wanna
get
get
inspired
inspired
by
by
Python,
python
head
head
over
over
to
to
Kaggle
kaggle
and
and
check
check
out
out
some
some
of
of
the
the
amazing
amazing
work
work
done
done
by
by
real
real
data
data
scientists.
scientists
In
in
today's
today's
video,
video
you'll
you'll
learn
learn
not
not
only
only
how
how
to
to
and
and
train
train
a
a
machine
machine
learning
learning
model,
model
but
but
also
also
how
how
to
to
deploy
deploy
it
it
to
to
the
the
Cloud
cloud
as
as
a
a
real
real
product
product
that
that
people
people
can
can
use.
use
The
the
first
first
step
step
in
in
any
any
machine
machine
learning
learning
problem
problem
is
is
the
the
underlying
underlying
data.
data
A
a
practical
practical
use
use
case
case
for
for
a
a
Firebase
firebase
developer
developer
would
would
be
be
to
to
save
save
some
some
information
information
about
about
a
a
user
user
in
in
the
the
Firestore
firestore
database,
database
Then
then
train
train
that
that
data
data
to
to
build
build
a
a
predictive
predictive
model
model
that
that
adds
adds
some
some
value
value
to
to
the
the
user
user
experience.
experience
If
if
we
we
look
look
in
in
our
our
database,
database
you
you
can
can
see
see
here
here
we
we
have
have
a
a
collection
collection
of
of
developer
developer
records
records
and
and
each
each
one
one
will
will
tell
tell
us
us
what
what
Tools
tools
this
this
developer
developer
uses
uses
and
and
their
their
overall
overall
happiness
happiness
as
as
a
a
developer.
developer
This
this
is
is
just
just
synthetic
synthetic
data
data
that
that
I
i
generated
generated
for
for
this
this
demo,
demo
but
but
you
you
can
can
think
think
of
of
it
it
as
as
being
being
information
information
that
that
you
you
might
might
collect
collect
from
from
a
a
user
user
survey
survey
or
or
something
something
along
along
those
those
lines.
lines
Machine
machine
learning
learning
problems
problems
generally
generally
require
require
a
a
huge
huge
amount
amount
of
of
data
data
and
and
also
also
a
a
huge
huge
amount
amount
of
of
compute
compute
resources
resources
to
to
run
run
operations
operations
on
on
that
that
data.
data
We're
we're
going
going
to
to
use
use
a
a
service
service
on
on
GCP
gcp
called
called
Datalab.
datalab
It
it
allows
allows
us
us
to
to
stream
stream
and
and
process
process
our
our
data
data
in
in
the
the
Cloud,
cloud
which
which
means
means
you
you
can
can
spin
spin
up
up
a
a
GPU
gpu
powered
powered
supercomputer
supercomputer
from
from
your
your
top
top
and
and
then
then
just
just
shut
shut
it
it
off
off
when
when
you're
you're
done.
done
To
to
get
get
going
going
with
with
Datalab,
datalab
you'll
you'll
need
need
to
to
have
have
a
a
Firebase
firebase
or
or
GCP
gcp
project,
project
and
and
then
then
we
we
have
have
a
a
couple
couple
of
of
things
things
to
to
run
run
here
here
from
from
the
the
command
command
line.
line
Basically,
basically
you're
you're
just
just
installing
installing
Datalab
datalab
and
and
then
then
calling
calling
Datalab
datalab
create
create
with
with
the
the
name
name
that
that
you
you
want
want
to
to
give
give
it.
it
What
what
this
this
does
does
is
is
give
give
you
you
an
an
interactive
interactive
TIFF
tiff
Python
python
or
or
Jupyter
jupyter
notebook
notebook
in
in
the
the
Cloud
cloud
that
that
you
you
can
can
use
use
to
to
run
run
Python
python
code.
code
And
and
you
you
can
can
allocate
allocate
as
as
much
much
CPU
cpu
or
or
memory
memory
that
that
you
you
need
need
and
and
also
also
attach
attach
GPUs
gpus
to
to
it.
it
The
the
nice
nice
thing
thing
about
about
this
this
notebook
notebook
is
is
that
that
it's
it's
tightly
tightly
integrated
integrated
with
with
all
all
of
of
your
your
other
other
GCP
gcp
resources
resources
such
such
as
as
storage
storage
buckets,
buckets
BigQuery,
bigquery
and
and
everything
everything
else.
else
It
it
should
should
give
give
you
you
a
a
link
link
to
to
localhost
localhost
that
that
will
will
show
show
a
a
notebook
notebook
that
that
looks
looks
like
like
this.
this
The
the
first
first
thing
thing
I'm
i'm
going
going
to
to
do
do
is
is
install
install
Firebase
firebase
Admin
admin
by
by
running
running
exclamation
exclamation
pip
pip
install
install
Firebase
firebase
Admin.
admin
Now
now
if
if
you're
you're
not
not
familiar
familiar
with
with
Jupyter
jupyter
Notebooks,
notebooks
they're
they're
basically
basically
just
just
a
a
nice
nice
UI
ui
for
for
executing
executing
Python
python
code.
code
We
we
can
can
put
put
each
each
code
code
block
block
in
in
its
its
own
own
cell
cell
and
and
then
then
Execute
execute
those
those
1
1
by
by
1
1
by
by
clicking
clicking
the
the
play
play
button.
button
You'll
you'll
notice
notice
that
that
there
there
are
are
a
a
ton
ton
of
of
examples
examples
in
in
the
the
Datalab
datalab
environment,
environment
so
so
definitely
definitely
check
check
those
those
out,
out
especially
especially
if
if
you're
you're
not
not
super
super
comfortable
comfortable
with
with
Python.
python
The
the
notebook
notebook
we're
we're
building
building
today
today
does
does
4
4
main
main
things.
things
1st,
1st
it
it
brings
brings
in
in
our
our
data
data
from
from
Firestore.
firestore
Then
then
we
we
do
do
an
an
exploratory
exploratory
data
data
analysis.
analysis
Then
then
we
we
split
split
the
the
data
data
up
up
and
and
train
train
an
an
algorithm.
algorithm
And
and
then
then
finally,
finally
export
export
it
it
to
to
a
a
Cloud
cloud
storage
storage
bucket.
bucket
1st,
1st
we'll
we'll
need
need
access
access
to
to
Firestore.
firestore
So
so
we'll
we'll
go
go
into
into
the
the
Firebase
firebase
project
project
settings
settings
and
and
generate
generate
a
a
new
new
private
private
key.
key
That'll
that'll
give
give
you
you
a
a
JSON
json
file
file
that
that
you
you
can
can
save
save
in
in
the
the
root
root
of
of
the
the
Datalab.
datalab
There
there
are
are
2
2
dependencies
dependencies
that
that
you'll
you'll
see
see
commonly
commonly
in
in
machine
machine
learning
learning
problems.
problems
1
1
is
is
Pandas,
pandas
which
which
allows
allows
us
us
to
to
put
put
data
data
in
in
a
a
DataFrame.
dataframe
And
and
the
the
second
second
one
one
is
is
NumPy,
numpy
which
which
helps
helps
us
us
make
make
scientific
scientific
calculations
calculations
on
on
arrays.
arrays
What
what
we'll
we'll
be
be
doing
doing
in
in
just
just
a
a
second
second
here
here
is
is
taking
taking
our
our
Firestore
firestore
data
data
and
and
putting
putting
it
it
into
into
a
a
Pandas
pandas
data
data
frame.
frame
And
and
NumPy
numpy
will
will
help
help
us
us
generate
generate
some
some
random
random
synthetic
synthetic
data
data
that
that
we
we
can
can
use
use
for
for
this
this
demo.
demo
From
from
there,
there
we
we
have
have
a
a
library
library
called
called
pyplot
pyplot
that
that
will
will
help
help
us
us
generate
generate
some
some
charts,
charts
then
then
we'll
we'll
import
import
Firebase
firebase
Admin
admin
and
and
initialize
initialize
it
it
with
with
our
our
credentials.
credentials
The
the
process
process
is
is
basically
basically
identical
identical
to
to
Node.
node
Js
js
or
or
any
any
other
other
language
language
for
for
that
that
matter.
matter
This
this
next
next
part
part
shows
shows
how
how
I
i
generated
generated
the
the
synthetic
synthetic
data
data
and
and
I'm
i'm
not
not
going
going
to
to
go
go
through
through
it
it
line
line
by
by
line,
line
but
but
just
just
Explain
explain
what's
what's
going
going
on
on
at
at
a
a
high
high
level.
level
Almost
almost
all
all
machine
machine
learning
learning
problems
problems
boil
boil
down
down
to
to
finding
finding
a
a
signal
signal
that's
that's
hidden
hidden
within
within
some
some
noise.
noise
So
so
the
the
data
data
I've
i've
generated
generated
below
below
is
is
slightly
slightly
random,
random
but
but
also
also
has
has
some
some
structure
structure
to
to
it
it
as
as
well.
well
The
the
value
value
that
that
we're
we're
trying
trying
to
to
predict
predict
is
is
developer
developer
happiness.
happiness
So
so
I
i
started
started
by
by
giving
giving
every
every
user
user
a
a
random
random
happiness
happiness
between
between
negative
negative
2525.
2525
Then
then
I
i
have
have
a
a
dictionary
dictionary
with
with
6
6
different
different
programming
programming
languages
languages
in
in
it,
it
each
each
one
one
that
that
will
will
give
give
the
the
developer
developer
a
a
different
different
level
level
of
of
happiness.
happiness
And
and
I
i
also
also
gave
gave
each
each
developer
developer
a
a
value
value
of
of
Firebase,
firebase
which
which
when
when
true
true
makes
makes
the
the
developer
developer
much
much
happier
happier
than
than
when
when
false.
false
But
but
the
the
general
general
idea
idea
here
here
is
is
that
that
the
the
data
data
is
is
partially
partially
random
random
and
and
partially
partially
fixed.
fixed
Our
our
goal
goal
is
is
to
to
create
create
an
an
algorithm
algorithm
that
that
can
can
detect
detect
which
which
features
features
are
are
important
important
and
and
which
which
aren't.
aren't
In
in
this
this
example,
example
Firebase
firebase
should
should
be
be
the
the
most
most
important
important
followed
followed
by
by
programming
programming
language
language
and
and
then
then
age
age
shouldn't
shouldn't
be
be
important
important
at
at
all.
all
If
if
you
you
have
have
your
your
own
own
data
data
in
in
Firestore,
firestore
you
you
can
can
just
just
skip
skip
this
this
step.
step
And
and
then
then
in
in
the
the
next
next
cell,
cell
I
i
will
will
actually
actually
retrieve
retrieve
that
that
data
data
from
from
the
the
database
database
and
and
then
then
save
save
it
it
to
to
a
a
CSV
csv
file,
file
we
we
don't
don't
have
have
to
to
reread
reread
it
it
each
each
time
time
and
and
get
get
charged
charged
for
for
additional
additional
reads
reads
in
in
Firestore.
firestore
Conveniently,
conveniently
Pandas
pandas
has
has
a
a
2
2
CSV
csv
method
method
that
that
we
we
can
can
use
use
to
to
Just
just
quickly
quickly
save
save
our
our
data
data
to
to
a
a
CSV
csv
file.
file
So
so
now
now
if
if
we
we
come
come
back
back
to
to
our
our
notebook,
notebook
we
we
can
can
just
just
run
run
pandas
pandas
read
read
CSV.
csv
Then
then
the
the
first
first
thing
thing
I'll
i'll
do
do
is
is
just
just
sample
sample
10
10
items
items
from
from
that
that
data
data
frame
frame
to
to
see
see
what
what
the
the
data
data
looks
looks
like.
like
Machine
machine
learning
learning
is
is
not
not
magic.
magic
And
and
the
the
first
first
thing
thing
you
you
should
should
do
do
once
once
you
you
have
have
your
your
data
data
is
is
run
run
an
an
exploratory
exploratory
analysis.
analysis
The
the
idea
idea
is
is
to
to
uncover
uncover
some
some
signals
signals
here
here
that
that
you
you
can
can
use
use
to
to
nudge
nudge
your
your
algorithm
algorithm
in
in
the
the
right
right
direction.
direction
And
and
the
the
reason
reason
we
we
put
put
our
our
data
data
in
in
a
a
Pandas
pandas
DataFrame
dataframe
is
is
because
because
it
it
gives
gives
us
us
a
a
whole
whole
bunch
bunch
of
of
methods
methods
to
to
help
help
analyze
analyze
and
and
also
also
clean
clean
the
the
data
data
before
before
it
it
goes
goes
into
into
an
an
algorithm.
algorithm
So
so
here
here
we're
we're
running
running
DataFrame
dataframe
sample
sample
to
to
just
just
get
get
10
10
random
random
rows
rows
out
out
of
of
the
the
dataset.
dataset
And
and
a
a
DataFrame
dataframe
is
is
treated
treated
very
very
much
much
like
like
an
an
array.
array
You
you
can
can
see
see
here
here
that
that
I'm
i'm
doing
doing
DataFrame
dataframe
with
with
the
the
happiness
happiness
key
key
on
on
there.
there
And
and
then
then
I'm
i'm
gonna
gonna
run
run
plot
plot
histogram
histogram
to
to
give
give
us
us
a
a
visual
visual
idea
idea
of
of
what
what
the
the
actual
actual
label
label
looks
looks
like.
like
So
so
you
you
can
can
see
see
here
here
we
we
have
have
a
a
pretty
pretty
normal
normal
distribution
distribution
of
of
happiness
happiness
values.
values
And
and
this
this
is
is
what
what
we
we
want
want
our
our
algorithm
algorithm
to
to
be
be
able
able
to
to
predict
predict
based
based
on
on
the
the
other
other
features
features
in
in
the
the
data.
data
Another
another
thing
thing
you
you
can
can
do
do
is
is
run
run
DataFrame
dataframe
described
described
to
to
get
get
common
common
statistics
statistics
on
on
the
the
dataset
dataset
such
such
as
as
the
the
mean,
mean
standard
standard
deviation,
deviation
and
and
things
things
like
like
that.
that
So
so
now
now
that
that
we've
we've
explored
explored
our
our
data,
data
it's
it's
time
time
to
to
introduce
introduce
a
a
new
new
library
library
called
called
scikit
scikit
learn.
learn
You've
you've
probably
probably
heard
heard
a
a
lot
lot
of
of
hype
hype
surrounding
surrounding
TensorFlow
tensorflow
and
and
deep
deep
neural
neural
networks
networks
and
and
those
those
Tools
tools
are
are
great,
great
but
but
they're
they're
often
often
overkill
overkill
for
for
a
a
lot
lot
of
of
problems.
problems
Scikit
scikit
learn
learn
contains
contains
a
a
huge
huge
collection
collection
of
of
tools
tools
for
for
tackling
tackling
machine
machine
learning
learning
problems.
problems
The
the
first
first
thing
thing
we'll
we'll
do
do
is
is
use
use
it
it
to
to
clean
clean
our
our
data
data
to
to
make
make
sure
sure
that
that
all
all
the
the
values
values
are
are
numeric,
numeric
which
which
is
is
required
required
for
for
most
most
machine
machine
learning
learning
algorithms.
algorithms
Currently,
currently
you'll
you'll
notice
notice
that
that
our
our
programming
programming
language
language
value
value
is
is
a
a
string
string
value.
value
So
so
what
what
we
we
need
need
to
to
do
do
is
is
get
get
all
all
of
of
the
the
unique
unique
elements
elements
in
in
that
that
array
array
and
and
convert
convert
them
them
to
to
a
a
number,
number
Then
then
assign
assign
each
each
value
value
to
to
the
the
corresponding
corresponding
number.
number
We
we
can
can
do
do
this
this
easily
easily
with
with
scikit
scikit
learn
learn
by
by
creating
creating
a
a
label
label
encoder
encoder
and
and
then
then
having
having
it
it
fit
fit
transform
transform
that
that
column
column
to
to
the
the
numeric
numeric
classes.
classes
If
if
we
we
sample
sample
the
the
data
data
frame
frame
again,
again
you'll
you'll
see
see
that
that
our
our
language
language
column
column
is
is
now
now
numeric
numeric
and
and
no
no
longer
longer
a
a
string.
string
The
the
next
next
thing
thing
scikit
scikit
learn
learn
will
will
help
help
us
us
with
with
is
is
slicing
slicing
the
the
data
data
into
into
different
different
chunks
chunks
that
that
we
we
can
can
use
use
to
to
train
train
an
an
algorithm.
algorithm
We'll
we'll
define
define
a
a
new
new
variable
variable
named
named
x
x
which
which
is
is
all
all
of
of
our
our
features
features
in
in
the
the
data
data
minus
minus
the
the
one
one
we're
we're
trying
trying
to
to
predict
predict
which
which
is
is
happiness.
happiness
Then
then
we'll
we'll
define
define
y
y
as
as
the
the
variable
variable
that
that
we
we
are
are
trying
trying
to
to
predict.
predict
Now
now
we
we
need
need
to
to
separate
separate
these
these
into
into
sets
sets
for
for
training
training
and
and
then
then
another
another
one
one
for
for
testing
testing
or
or
validation.
validation
So
so
I'm
i'm
using
using
this
this
TrainTest
traintest
Split
split
method,
method
passing
passing
it
it
our
our
x
x
y
y
values
values
and
and
then
then
making
making
the
the
test
test
size
size
be
be
1
1
third
third
of
of
the
the
total
total
dataset
dataset
size,
size
which
which
means
means
we'll
we'll
be
be
training
training
on
on
66%
66%
of
of
the
the
data.
data
The
the
reason
reason
you
you
don't
don't
train
train
all
all
the
the
data
data
is
is
because
because
then
then
you
you
have
have
no
no
way
way
to
to
validate
validate
whether
whether
or
or
not
not
the
the
model
model
is
is
any
any
good,
good
and
and
you
you
want
want
a
a
model
model
that
that
can
can
kind
kind
of
of
generalize
generalize
the
the
overall
overall
dataset.
dataset
So
so
now
now
we're
we're
going
going
to
to
train
train
an
an
algorithm
algorithm
that's
that's
usually
usually
very
very
good
good
out
out
of
of
the
the
box,
box
which
which
is
is
a
a
random
random
forest.
forest
In
in
our
our
case,
case
we
we
have
have
a
a
value
value
between
between
negative
negative
100
100
and
and
positive
positive
100.
100
And
and
the
the
predicted
predicted
value
value
can
can
be
be
anything
anything
in
in
between
between
there.
there
So
so
we
we
have
have
an
an
infinite
infinite
number
number
of
of
possible
possible
predictions,
predictions
which
which
means
means
we're
we're
dealing
dealing
with
with
a
a
regression
regression
problem.
problem
This
this
differs
differs
from
from
a
a
classification
classification
problem
problem
where
where
you
you
might
might
be
be
predicting
predicting
different
different
labels
labels
like
like
whether
whether
or
or
not
not
a
a
picture
picture
is
is
a
a
dog
dog
or
or
a
a
cat.
cat
The
the
code
code
in
in
this
this
last
last
cell
cell
is
is
very
very
simple.
simple
We
we
just
just
have
have
the
the
Random
random
Forest
forest
Regressor,
regressor
which
which
we
we
fit
fit
to
to
our
our
x
x
and
and
y
y
data.
data
And
and
then
then
we
we
predict
predict
on
on
the
the
features
features
in
in
the
the
test
test
set
set
of
of
that
that
data.
data
Yeah.
yeah
There
there
are
are
a
a
bunch
bunch
of
of
different
different
options
options
you
you
can
can
pass
pass
to
to
the
the
regressor
regressor
to
to
try
try
to
to
fine
fine
tune
tune
it.
it
That's
that's
called
called
hyperparameter
hyperparameter
tuning.
tuning
And
and
then
then
you
you
need
need
to
to
have
have
a
a
metric
metric
to
to
analyze
analyze
its
its
performance.
performance
In
in
this
this
case,
case
we're
we're
going
going
to
to
use
use
Mean
mean
Absolute
absolute
Error
error
which
which
means
means
that
that
our
our
Random
random
Forest
forest
Model
model
is
is
about
about
18
18
points
points
wrong
wrong
on
on
average.
average
So
so
how
how
good
good
is
is
that
that
exactly?
exactly
Well,
well
the
the
best
best
place
place
to
to
start
start
is
is
just
just
by
by
comparing
comparing
it
it
to
to
a
a
random
random
set
set
of
of
predictions.
predictions
In
in
the
the
next
next
cell,
cell
we
we
can
can
see
see
that
that
completely
completely
random
random
predictions
predictions
produce
produce
A
a
mean
mean
absolute
absolute
error
error
of
of
about
about
56
56
which
which
means
means
that
that
our
our
algorithm
algorithm
is
is
relatively
relatively
effective.
effective
Now
now
the
the
last
last
thing
thing
we
we
wanna
wanna
do
do
is
is
figure
figure
out
out
which
which
features
features
in
in
the
the
data
data
were
were
most
most
important
important
the
the
decision
decision
making
making
process.
process
As
as
you
you
can
can
see
see
here,
here
the
the
age
age
was
was
not
not
important
important
at
at
all
all
because
because
it
it
was
was
completely
completely
random.
random
And
and
then
then
Firebase
firebase
and
and
programming
programming
languages
languages
were
were
the
the
most
most
important.
important
So
so
at
at
this
this
point,
point
we
we
have
have
a
a
predictive
predictive
model,
model
but
but
we
we
have
have
no
no
way
way
for
for
anybody
anybody
to
to
use
use
it.
it
If
if
you're
you're
building
building
a
a
product,
product
you
you
most
most
likely
likely
want
want
to
to
serve
serve
that
that
up
up
as
as
an
an
API
api
so
so
it
it
can
can
be
be
consumed
consumed
by
by
some
some
kind
kind
of
of
front
front
end
end
application
application
like
like
Angular
angular
or
or
iOS
ios
or
or
whatever.
whatever
And
and
that's
that's
where
where
ML
ml
Engine
engine
comes
comes
in.
in
The
the
last
last
cell
cell
in
in
the
the
notebook
notebook
that
that
you're
you're
looking
looking
Here
here
is
is
taking
taking
the
the
model
model
that
that
we
we
just
just
trained,
trained
dumping
dumping
it
it
as
as
a
a
job
job
lib
lib
file,
file
and
and
then
then
saving
saving
it
it
in
in
our
our
Firebase
firebase
storage
storage
bucket.
bucket
ML
ml
Engine
engine
will
will
be
be
able
able
to
to
read
read
the
the
model
model
file
file
in
in
the
the
bucket
bucket
and
and
then
then
serve
serve
out
out
predictions
predictions
in
in
a
a
highly
highly
performant
performant
way
way
with
with
predictions
predictions
being
being
served
served
over
over
the
the
network
network
in
in
just
just
a
a
matter
matter
of
of
a
a
few
few
100
100
milliseconds.
milliseconds
So
so
before
before
we
we
head
head
over
over
to
to
ML
ml
Engine,
engine
go
go
into
into
your
your
Firebase
firebase
Storage
storage
Bucket
bucket
and
and
make
make
sure
sure
that
that
you
you
have
have
your
your
job
job
load
load
file
file
saved
saved
in
in
its
its
own
own
individual
individual
folder.
folder
From
from
there,
there
you'll
you'll
wanna
wanna
go
go
back
back
to
to
the
the
GCP
gcp
console
console
and
and
then
then
go
go
to
to
enable
enable
APIs.
apis
You'll
you'll
need
need
to
to
enable
enable
the
the
Cloud
cloud
Build
build
API
api
because
because
your
your
models
models
will
will
should
should
be
be
versioned.
versioned
And,
and
of
of
course,
course
we'll
we'll
need
need
the
the
ML
ml
Engine
engine
API
api
as
as
well.
well
From
from
there,
there
you
you
can
can
navigate
navigate
over
over
to
to
the
the
Cloud
cloud
ML
ml
Engine
engine
dashboard.
dashboard
And
and
the
the
first
first
thing
thing
I
i
wanna
wanna
do
do
is
is
create
create
a
a
model
model
and
and
give
give
that
that
model
model
a
a
name.
name
You'll
you'll
need
need
this
this
name
name
later
later
when
when
we
we
get
get
to
to
the
the
Cloud
cloud
Function.
function
The
the
great
great
thing
thing
about
about
this
this
system
system
is
is
that
that
Your
your
models
models
can
can
have
have
multiple
multiple
versions,
versions
so
so
you
you
can
can
see
see
how
how
their
their
performance
performance
changes
changes
over
over
time.
time
It's
it's
important
important
that
that
you
you
fill
fill
out
out
things
things
correctly
correctly
here
here
like
like
our
our
Python
python
version
version
as
as
2.7
2.7
in
in
the
the
environment
environment
And
and
we're
we're
using
using
the
the
scikit
scikit
learn
learn
framework.
framework
And
and
the
the
version
version
is
is
19
19
and
and
the
the
runtime
runtime
version
version
is
is
1.9.
1.9
From
from
there,
there
you'll
you'll
need
need
to
to
select
select
the
the
folder
folder
that
that
has
has
this
this
model
model
in
in
it
it
which
which
is
is
that
that
job
job
load
load
file
file
that
that
we
we
saved
saved
earlier.
earlier
It'll
it'll
take
take
a
a
few
few
minutes
minutes
to
to
save.
save
But
but
once
once
that's
that's
done,
done
you'll
you'll
be
be
able
able
to
to
either
either
make
make
predictions
predictions
from
from
the
the
command
command
line
line
or
or
from
from
other
other
Google
google
APIs
apis
such
such
as
as
a
a
Cloud
cloud
function
function
as
as
we'll
we'll
see
see
here
here
next.
next
Once
once
the
the
model
model
goes
goes
into
into
production,
production
we'll
we'll
be
be
able
able
to
to
get
get
some
some
actual
actual
performance
performance
metrics
metrics
about
about
how
how
it's
it's
handling
handling
its
its
workload.
workload
So
so
now
now
we're
we're
on
on
to
to
the
the
final
final
step
step
and
and
that's
that's
writing
writing
a
a
Cloud
cloud
Function
function
that
that
can
can
consume
consume
this
this
model.
model
For
for
that,
that
we'll
we'll
switch
switch
over
over
to
to
Versus
versus
Code.
code
And
and
first,
first
we'll
we'll
go
go
into
into
the
the
command
command
line,
line
run
run
Firebase
firebase
init
init
functions,
functions
Then
then
we'll
we'll
c
c
d
d
into
into
the
the
functions
functions
directory
directory
and
and
install
install
Google
google
APIs.
apis
The
the
Google
google
APIs
apis
package
package
for
for
Node
node
will
will
allow
allow
us
us
to
to
authenticate
authenticate
the
the
Cloud
cloud
Function
function
and
and
then
then
make
make
requests
requests
to
to
the
the
ML
ml
API
api
without
without
the
the
need
need
for
for
an
an
HTTP
http
library.
library
I'm
i'm
using
using
TypeScript
typescript
here,
here
but
but
you
you
could
could
just
just
as
as
easily
easily
use
use
vanilla
vanilla
JS.
js
We'll
we'll
import
import
Google
google
from
from
Google
google
APIs,
apis
then
then
reference
reference
ML
ml
version
version
1.
1
The
the
function
function
itself
itself
is
is
called
called
predict
predict
happiness,
happiness
and
and
it's
it's
just
just
a
a
regular
regular
HTTPS
https
function.
function
But
but
you
you
could
could
use
use
any
any
kind
kind
of
of
Cloud
cloud
Function
function
here
here
you
you
want.
want
A
a
good
good
alternative
alternative
would
would
be
be
to
to
run
run
a
a
prediction
prediction
every
every
time
time
a
a
new
new
record
record
is
is
added
added
to
to
the
the
Firestore
firestore
database.
database
We're
we're
going
going
to
to
use
use
this
this
single
single
Cloud
cloud
function
function
to
to
make
make
requests
requests
to
to
potentially
potentially
multiple
multiple
models.
models
So
so
we'll
we'll
have
have
the
the
request
request
body
body
define
define
which
which
model
model
should
should
make
make
the
the
prediction.
prediction
And
and
then
then
the
the
instances
instances
will
will
be
be
the
the
actual
actual
data
data
that
that
we're
we're
trying
trying
to
to
predict,
predict
which
which
is
is
typically
typically
an
an
array
array
with
with
the
the
same
same
shape
shape
in
in
which
which
you
you
train
train
your
your
actual
actual
Machine
machine
Learning
learning
model.
model
So
so
now
now
that
that
we
we
have
have
the
the
request
request
body,
body
we
we
can
can
format
format
our
our
request
request
to
to
the
the
Google
google
API.
api
This
this
request
request
requires
requires
authentication,
authentication
which
which
we
we
can
can
get
get
by
by
calling
calling
Google
google
auth
auth
get
get
application
application
default.
default
And
and
that
that
will
will
connect
connect
our
our
Firebase
firebase
credentials
credentials
with
with
the
the
Google
google
API.
api
From
from
there,
there
we'll
we'll
set
set
up
up
a
a
variable
variable
for
for
the
the
model
model
name,
name
which
which
needs
needs
to
to
have
have
your
your
project
project
ID
id
in
in
it
it
as
as
well
well
as
as
the
the
name
name
of
of
the
the
actual
actual
model
model
that
that
you
you
gave
gave
it
it
in
in
the
the
ML
ml
Engine
engine
console.
console
And
and
now
now
everything
everything
is
is
in
in
place
place
to
to
make
make
a
a
prediction
prediction
call
call
to
to
the
the
ML
ml
Engine
engine
API.
api
We
we
call
call
projects
projects
predict,
predict
pass
pass
in
in
our
our
auth
auth
credentials,
credentials
the
the
name
name
of
of
the
the
model,
model
and
and
then
then
also
also
the
the
instances
instances
that
that
we're
we're
trying
trying
to
to
predict.
predict
This
this
can
can
actually
actually
be
be
an
an
array
array
of
of
arrays
arrays
if
if
we
we
want
want
to
to
make
make
multiple
multiple
predictions
predictions
from
from
a
a
single
single
Cloud
cloud
Function
function
call.
call
And
and
then
then
TypeScript
typescript
gets
gets
a
a
little
little
confused
confused
right
right
here,
here
so
so
we
we
need
need
to
to
add
add
as
as
any
any
after
after
the
the
argument
argument
to
to
predict.
predict
So
so
that
that
Should
should
resolve
resolve
with
with
the
the
predictions
predictions
from
from
our
our
model,
model
and
and
then
then
we
we
can
can
send
send
that
that
back
back
out
out
as
as
the
the
response
response
from
from
this
this
HTTP
http
function.
function
At
at
this
this
point,
point
you
you
could
could
deploy
deploy
this
this
Cloud
cloud
function
function
and
and
you
you
essentially
essentially
an
an
API
api
endpoint
endpoint
with
with
your
your
machine
machine
learning
learning
model,
model
but
but
we're
we're
just
just
going
going
to
to
run
run
it
it
locally.
locally
And
and
then
then
we'll
we'll
test
test
it
it
with
with
an
an
application
application
called
called
Insomnia.
insomnia
Insomnia
insomnia
is
is
a
a
free
free
desktop
desktop
app
app
that
that
you
you
can
can
install
install
and
and
it
it
allows
allows
you
you
to
to
make
make
HTTP
http
calls
calls
in
in
this
this
nice
nice
user
user
interface.
interface
So
so
first
first
in
in
the
the
JSON
json
body,
body
we'll
we'll
declare
declare
the
the
model
model
name,
name
which
which
is
is
developer
developer
happiness.
happiness
For
for
the
the
instances,
instances
we
we
pass
pass
an
an
array
array
of
of
arrays
arrays
where
where
each
each
array
array
is
is
going
going
to
to
be
be
a
a
prediction
prediction
that
that
we're
we're
trying
trying
to
to
make.
make
So
so
these
these
values
values
match
match
the
the
features
features
in
in
our
our
original
original
dataset.
dataset
For
for
example,
example
column
column
1
1
would
would
be
be
the
the
user's
user's
age,
age
column
column
2,
2
whether
whether
or
or
not
not
they
they
use
use
Firebase,
firebase
and
and
column
column
3,
3
their
their
encoded
encoded
programming
programming
language.
language
Now
now
if
if
we
we
send
send
this
this
request
request
to
to
our
our
function,
function
we
we
get
get
a
a
response
response
back
back
in
in
about
about
400
400
milliseconds
milliseconds
that
that
gives
gives
us
us
the
the
prediction
prediction
from
from
those
those
instances.
instances
So
so
each
each
set
set
of
of
features
features
sent
sent
from
from
the
the
request
request
here
here
on
on
the
the
left
left
got
got
a
a
different
different
happiness
happiness
prediction
prediction
from
from
the
the
response
response
here
here
on
on
the
the
right.
right
In
in
other
other
words,
words
we
we
started
started
with
with
basically
basically
nothing,
nothing
And
and
now
now
we
we
have
have
an
an
endpoint
endpoint
in
in
the
the
Cloud
cloud
for
for
our
our
custom
custom
machine
machine
learning
learning
model.
model
I'm
i'm
gonna
gonna
go
go
ahead
ahead
and
and
wrap
wrap
things
things
up
up
there.
there
If
if
this
this
video
video
helped
helped
you,
you
please
please
like
like
and
and
subscribe
subscribe
and
and
you
you
can
can
find
find
the
the
full
full
source
source
code
code
on
on
angularfirebase.com.
angularfirebase.com
If
if
you're
you're
serious
serious
about
about
building
building
a
a
project
project
with
with
Firebase
firebase
or
or
Google
google
Cloud,
cloud
consider
consider
upgrading
upgrading
to
to
a
a
Pro
pro
membership
membership
at
at
angularfirebase.com.
angularfirebase.com
You'll
you'll
get
get
access
access
to
to
a
a
whole
whole
bunch
bunch
of
of
exclusive
exclusive
content
content
designed
designed
to
to
help
help
you
you
build
build
and
and
ship
ship
your
your
app
app
faster.
faster
Thanks
thanks
for
for
watching
watching
and
and
I'll
i'll
talk
talk
to
to
you
you
soon.
soon
success
The speakers discuss how to use machine learning models and the process of creating a Python Python Brain in the Cloud. They also introduce a new tool called scarches learn, which helps clean data sets and remove uncertainties. The process involves creating a label encoder, running an algorithm, and creating a new variable called x, training an algorithm using a random forest, and using the Cloud function for predictions. They also discuss how to create a model and use it for building a product, including setting up a model and using a free desktop app.
ML Engine - Machine Learning in the Cloud
Today, we'll start from 0 and finish the video with our own custom machine learning model deployed to the cloud as an API. We will pull some actual user data that we have saved in the Firestore database, then we'll export it to Datalab so we can train a Python based machine learning model with virtually unlimited compute resources. Once we have a solid model in place we'll deploy it to ML engine so we can version it and analyze it in production, and finally we will Expose it to the universe by making it available in a Firebase cloud function. But before you do that, make sure to like and subscribe, and I'll be giving away one of a kind Python Brain t shirt. All you have to do is leave a comment below, then we'll pick a random winner via livestream next week. This video was made in collaboration with Roddy Cholakov, who is, get this, only in the 7th grade and already doing crazy stuff with TensorFlow, Firebase, and a bunch of other tech. In addition to this video, you'll also want to check out his GitHub repo that shows you how to train a TensorFlow model directly in a Cloud Function. Now we're ready to Started and we'll do something that we've never done on this channel before and that's write Python code. Python is the most commonly used language for machine learning problems and it's pretty easy to learn if you already know script. If you wanna get inspired by Python, head over to Kaggle and check out some of the amazing work done by real data scientists. In today's video, you'll learn not only how to and train a machine learning model, but also how to deploy it to the Cloud as a real product that people can use. The first step in any machine learning problem is the underlying data. A practical use case for a Firebase developer would be to save some information about a user in the Firestore database, Then train that data to build a predictive model that adds some value to the user experience. If we look in our database, you can see here we have a collection of developer records and each one will tell us what Tools this developer uses and their overall happiness as a developer. This is just synthetic data that I generated for this demo, but you can think of it as being information that you might collect from a user survey or something along those lines. Machine learning problems generally require a huge amount of data and also a huge amount of compute resources to run operations on that data. We're going to use a service on GCP called Datalab. It allows us to stream and process our data in the Cloud, which means you can spin up a GPU powered supercomputer from your top and then just shut it off when you're done. To get going with Datalab, you'll need to have a Firebase or GCP project, and then we have a couple of things to run here from the command line. Basically, you're just installing Datalab and then calling Datalab create with the name that you want to give it. What this does is give you an interactive TIFF Python or Jupyter notebook in the Cloud that you can use to run Python code. And you can allocate as much CPU or memory that you need and also attach GPUs to it. The nice thing about this notebook is that it's tightly integrated with all of your other GCP resources such as storage buckets, BigQuery, and everything else. It should give you a link to localhost that will show a notebook that looks like this. The first thing I'm going to do is install Firebase Admin by running exclamation pip install Firebase Admin. Now if you're not familiar with Jupyter Notebooks, they're basically just a nice UI for executing Python code. We can put each code block in its own cell and then Execute those 1 by 1 by clicking the play button. You'll notice that there are a ton of examples in the Datalab environment, so definitely check those out, especially if you're not super comfortable with Python. The notebook we're building today does 4 main things. 1st, it brings in our data from Firestore. Then we do an exploratory data analysis. Then we split the data up and train an algorithm. And then finally, export it to a Cloud storage bucket. 1st, we'll need access to Firestore. So we'll go into the Firebase project settings and generate a new private key. That'll give you a JSON file that you can save in the root of the Datalab. There are 2 dependencies that you'll see commonly in machine learning problems. 1 is Pandas, which allows us to put data in a DataFrame. And the second one is NumPy, which helps us make scientific calculations on arrays. What we'll be doing in just a second here is taking our Firestore data and putting it into a Pandas data frame. And NumPy will help us generate some random synthetic data that we can use for this demo. From there, we have a library called pyplot that will help us generate some charts, then we'll import Firebase Admin and initialize it with our credentials. The process is basically identical to Node. Js or any other language for that matter. This next part shows how I generated the synthetic data and I'm not going to go through it line by line, but just Explain what's going on at a high level. Almost all machine learning problems boil down to finding a signal that's hidden within some noise. So the data I've generated below is slightly random, but also has some structure to it as well. The value that we're trying to predict is developer happiness. So I started by giving every user a random happiness between negative 2525. Then I have a dictionary with 6 different programming languages in it, each one that will give the developer a different level of happiness. And I also gave each developer a value of Firebase, which when true makes the developer much happier than when false. But the general idea here is that the data is partially random and partially fixed. Our goal is to create an algorithm that can detect which features are important and which aren't. In this example, Firebase should be the most important followed by programming language and then age shouldn't be important at all. If you have your own data in Firestore, you can just skip this step. And then in the next cell, I will actually retrieve that data from the database and then save it to a CSV file, we don't have to reread it each time and get charged for additional reads in Firestore. Conveniently, Pandas has a 2 CSV method that we can use to Just quickly save our data to a CSV file. So now if we come back to our notebook, we can just run pandas read CSV. Then the first thing I'll do is just sample 10 items from that data frame to see what the data looks like. Machine learning is not magic. And the first thing you should do once you have your data is run an exploratory analysis. The idea is to uncover some signals here that you can use to nudge your algorithm in the right direction. And the reason we put our data in a Pandas DataFrame is because it gives us a whole bunch of methods to help analyze and also clean the data before it goes into an algorithm. So here we're running DataFrame sample to just get 10 random rows out of the dataset. And a DataFrame is treated very much like an array. You can see here that I'm doing DataFrame with the happiness key on there. And then I'm gonna run plot histogram to give us a visual idea of what the actual label looks like. So you can see here we have a pretty normal distribution of happiness values. And this is what we want our algorithm to be able to predict based on the other features in the data. Another thing you can do is run DataFrame described to get common statistics on the dataset such as the mean, standard deviation, and things like that. So now that we've explored our data, it's time to introduce a new library called scikit learn. You've probably heard a lot of hype surrounding TensorFlow and deep neural networks and those Tools are great, but they're often overkill for a lot of problems. Scikit learn contains a huge collection of tools for tackling machine learning problems. The first thing we'll do is use it to clean our data to make sure that all the values are numeric, which is required for most machine learning algorithms. Currently, you'll notice that our programming language value is a string value. So what we need to do is get all of the unique elements in that array and convert them to a number, Then assign each value to the corresponding number. We can do this easily with scikit learn by creating a label encoder and then having it fit transform that column to the numeric classes. If we sample the data frame again, you'll see that our language column is now numeric and no longer a string. The next thing scikit learn will help us with is slicing the data into different chunks that we can use to train an algorithm. We'll define a new variable named x which is all of our features in the data minus the one we're trying to predict which is happiness. Then we'll define y as the variable that we are trying to predict. Now we need to separate these into sets for training and then another one for testing or validation. So I'm using this TrainTest Split method, passing it our x y values and then making the test size be 1 third of the total dataset size, which means we'll be training on 66% of the data. The reason you don't train all the data is because then you have no way to validate whether or not the model is any good, and you want a model that can kind of generalize the overall dataset. So now we're going to train an algorithm that's usually very good out of the box, which is a random forest. In our case, we have a value between negative 100 and positive 100. And the predicted value can be anything in between there. So we have an infinite number of possible predictions, which means we're dealing with a regression problem. This differs from a classification problem where you might be predicting different labels like whether or not a picture is a dog or a cat. The code in this last cell is very simple. We just have the Random Forest Regressor, which we fit to our x and y data. And then we predict on the features in the test set of that data. Yeah. There are a bunch of different options you can pass to the regressor to try to fine tune it. That's called hyperparameter tuning. And then you need to have a metric to analyze its performance. In this case, we're going to use Mean Absolute Error which means that our Random Forest Model is about 18 points wrong on average. So how good is that exactly? Well, the best place to start is just by comparing it to a random set of predictions. In the next cell, we can see that completely random predictions produce A mean absolute error of about 56 which means that our algorithm is relatively effective. Now the last thing we wanna do is figure out which features in the data were most important the decision making process. As you can see here, the age was not important at all because it was completely random. And then Firebase and programming languages were the most important. So at this point, we have a predictive model, but we have no way for anybody to use it. If you're building a product, you most likely want to serve that up as an API so it can be consumed by some kind of front end application like Angular or iOS or whatever. And that's where ML Engine comes in. The last cell in the notebook that you're looking Here is taking the model that we just trained, dumping it as a job lib file, and then saving it in our Firebase storage bucket. ML Engine will be able to read the model file in the bucket and then serve out predictions in a highly performant way with predictions being served over the network in just a matter of a few 100 milliseconds. So before we head over to ML Engine, go into your Firebase Storage Bucket and make sure that you have your job load file saved in its own individual folder. From there, you'll wanna go back to the GCP console and then go to enable APIs. You'll need to enable the Cloud Build API because your models will should be versioned. And, of course, we'll need the ML Engine API as well. From there, you can navigate over to the Cloud ML Engine dashboard. And the first thing I wanna do is create a model and give that model a name. You'll need this name later when we get to the Cloud Function. The great thing about this system is that Your models can have multiple versions, so you can see how their performance changes over time. It's important that you fill out things correctly here like our Python version as 2.7 in the environment And we're using the scikit learn framework. And the version is 19 and the runtime version is 1.9. From there, you'll need to select the folder that has this model in it which is that job load file that we saved earlier. It'll take a few minutes to save. But once that's done, you'll be able to either make predictions from the command line or from other Google APIs such as a Cloud function as we'll see here next. Once the model goes into production, we'll be able to get some actual performance metrics about how it's handling its workload. So now we're on to the final step and that's writing a Cloud Function that can consume this model. For that, we'll switch over to Versus Code. And first, we'll go into the command line, run Firebase init functions, Then we'll c d into the functions directory and install Google APIs. The Google APIs package for Node will allow us to authenticate the Cloud Function and then make requests to the ML API without the need for an HTTP library. I'm using TypeScript here, but you could just as easily use vanilla JS. We'll import Google from Google APIs, then reference ML version 1. The function itself is called predict happiness, and it's just a regular HTTPS function. But you could use any kind of Cloud Function here you want. A good alternative would be to run a prediction every time a new record is added to the Firestore database. We're going to use this single Cloud function to make requests to potentially multiple models. So we'll have the request body define which model should make the prediction. And then the instances will be the actual data that we're trying to predict, which is typically an array with the same shape in which you train your actual Machine Learning model. So now that we have the request body, we can format our request to the Google API. This request requires authentication, which we can get by calling Google auth get application default. And that will connect our Firebase credentials with the Google API. From there, we'll set up a variable for the model name, which needs to have your project ID in it as well as the name of the actual model that you gave it in the ML Engine console. And now everything is in place to make a prediction call to the ML Engine API. We call projects predict, pass in our auth credentials, the name of the model, and then also the instances that we're trying to predict. This can actually be an array of arrays if we want to make multiple predictions from a single Cloud Function call. And then TypeScript gets a little confused right here, so we need to add as any after the argument to predict. So that Should resolve with the predictions from our model, and then we can send that back out as the response from this HTTP function. At this point, you could deploy this Cloud function and you essentially an API endpoint with your machine learning model, but we're just going to run it locally. And then we'll test it with an application called Insomnia. Insomnia is a free desktop app that you can install and it allows you to make HTTP calls in this nice user interface. So first in the JSON body, we'll declare the model name, which is developer happiness. For the instances, we pass an array of arrays where each array is going to be a prediction that we're trying to make. So these values match the features in our original dataset. For example, column 1 would be the user's age, column 2, whether or not they use Firebase, and column 3, their encoded programming language. Now if we send this request to our function, we get a response back in about 400 milliseconds that gives us the prediction from those instances. So each set of features sent from the request here on the left got a different happiness prediction from the response here on the right. In other words, we started with basically nothing, And now we have an endpoint in the Cloud for our custom machine learning model. I'm gonna go ahead and wrap things up there. If this video helped you, please like and subscribe and you can find the full source code on angularfirebase.com. If you're serious about building a project with Firebase or Google Cloud, consider upgrading to a Pro membership at angularfirebase.com. You'll get access to a whole bunch of exclusive content designed to help you build and ship your app faster. Thanks for watching and I'll talk to you soon.
The speakers discuss how to use machine learning models and the process of creating a Python Python Brain in the Cloud. They also introduce a new tool called scarches learn, which helps clean data sets and remove uncertainties. The process involves creating a label encoder, running an algorithm, and creating a new variable called x, training an algorithm using a random forest, and using the Cloud function for predictions. They also discuss how to create a model and use it for building a product, including setting up a model and using a free desktop app.