PROJECT_ID = "[your-project-id]" #@param {type:"string"}
! gcloud config set project $PROJECT_ID

"""
Explanation: By deploying or using this software you agree to comply with the AI Hub Terms of Service and the Google APIs Terms of Service. To the extent of a direct conflict of terms, the AI Hub Terms of Service will control.

Overview
This notebook provides an example workflow of using the Distributed XGBoost ML container for training a classification ML model.
Dataset
The notebook uses the Iris dataset. It consists of 3 different types of irises (Setosa, Versicolour, and Virginica) petal and sepal length, stored in a 150x5 table.
Objective
The goal of this notebook is to go through a common training workflow:
- Create a dataset
- Train an ML model using the AI Platform Training service
- Monitor the training job with TensorBoard
- Identify if the model was trained successfully by looking at the generated "Run Report"
- Deploy the model for serving using the AI Platform Prediction service
- Use the endpoint for online predictions
- Interactively inspect the deployed ML model with the What-If Tool 
Costs
This tutorial uses billable components of Google Cloud Platform (GCP):

Cloud AI Platform
Cloud Storage

Learn about Cloud AI Platform
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
Set up your local development environment
If you are using Colab or AI Platform Notebooks, your environment already meets
all the requirements to run this notebook. You can skip this step.
Otherwise, make sure your environment meets this notebook's requirements.
You need the following:

The Google Cloud SDK
Git
Python 3
virtualenv
Jupyter notebook running in a virtual environment with Python 3

The Google Cloud guide to Setting up a Python development
environment and the Jupyter
installation guide provide detailed instructions
for meeting these requirements. The following steps provide a condensed set of
instructions:


Install and initialize the Cloud SDK.


Install Python 3.


Install
   virtualenv
   and create a virtual environment that uses Python 3.


Activate that environment and run pip install jupyter in a shell to install
   Jupyter.


Run jupyter notebook in a shell to launch Jupyter.


Open this notebook in the Jupyter Notebook Dashboard.


Set up your GCP project
The following steps are required, regardless of your notebook environment.


Select or create a GCP project.. When you first create an account, you get a $300 free credit towards your compute/storage costs.


Make sure that billing is enabled for your project.


Enable the AI Platform APIs and Compute Engine APIs.


Enter your project ID in the cell below. Then run the  cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.


Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.
End of explanation
"""


import sys

# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

if 'google.colab' in sys.modules:
  from google.colab import auth as google_auth
  google_auth.authenticate_user()

# If you are running this notebook locally, replace the string below with the
# path to your service account key and run this cell to authenticate your GCP
# account.
else:
  %env GOOGLE_APPLICATION_CREDENTIALS ''

"""
Explanation: Authenticate your GCP account
If you are using AI Platform Notebooks, your environment is already
authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions
when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:


In the GCP Console, go to the Create service account key
   page.


From the Service account drop-down list, select New service account.


In the Service account name field, enter a name.


From the Role drop-down list, select
   Machine Learning Engine > AI Platform Admin and
   Storage > Storage Object Admin.


Click Create. A JSON file that contains your key downloads to your
local environment.


Enter the path to your service account key as the
GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.
End of explanation
"""


BUCKET_NAME = "[your-bucket-name]" #@param {type:"string"}
REGION = 'us-central1' #@param {type:"string"}

"""
Explanation: Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
You need to have a "workspace" bucket that will hold the dataset and the output from the ML Container. Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets. 
You may also change the REGION variable, which is used for operations
throughout the rest of this notebook. Make sure to choose a region where Cloud AI Platform services are available. You may not use a Multi-Regional Storage bucket for training with AI Platform.
End of explanation
"""


! gsutil mb -l $REGION gs://$BUCKET_NAME

"""
Explanation: Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.
End of explanation
"""


! gsutil ls -al gs://$BUCKET_NAME

"""
Explanation: Finally, validate access to your Cloud Storage bucket by examining its contents:
End of explanation
"""


! pip install witwidget

"""
Explanation: PIP Install Packages and dependencies
End of explanation
"""


from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os
import time
import tensorflow as tf
import numpy as np
import pandas as pd
from sklearn import datasets
from sklearn.model_selection import train_test_split
from IPython.core.display import HTML
from googleapiclient import discovery

"""
Explanation: Import libraries and define constants
End of explanation
"""


# load Iris dataset
iris = datasets.load_iris()
names = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
data = pd.DataFrame(iris.data, columns=names)

# add target
data['target'] = iris.target

# split
training, validation = train_test_split(data, test_size=50, stratify=data['target'])

# standardization
training_targets = training.pop('target')
validation_targets = validation.pop('target')

data_mean = training.mean(axis=0)
data_std = training.std(axis=0)
training = (training - data_mean) / data_std
training['target'] = training_targets

validation = (validation - data_mean) / data_std
validation['target'] = validation_targets

print('Training data head')
display(training.head())

training_data = os.path.join('gs://', BUCKET_NAME, 'data/train.csv')
validation_data = os.path.join('gs://', BUCKET_NAME, 'data/valid.csv')

print('Copy the data in bucket ...')
with tf.io.gfile.GFile(training_data, 'w') as f:
  training.to_csv(f, index=False)
with tf.io.gfile.GFile(validation_data, 'w') as f:
  validation.to_csv(f, index=False)

"""
Explanation: Create a dataset
End of explanation
"""


output_location = os.path.join('gs://', BUCKET_NAME, 'output')

job_name = "xgboost_classification_{}".format(time.strftime("%Y%m%d%H%M%S"))
!gcloud ai-platform jobs submit training $job_name \
    --master-image-uri gcr.io/aihub-c2t-containers/kfp-components/trainer/dist_xgboost:latest \
    --region $REGION \
    --scale-tier CUSTOM \
    --master-machine-type standard \
    -- \
    --output-location {output_location} \
    --training-data {training_data} \
    --validation-data {validation_data} \
    --target-column target \
    --data-type csv \
    --number-of-classes 3 \
    --fresh-start True \
    --objective multi:softprob \

"""
Explanation: Cloud training
Accelerator and distribution support
| GPU | Multi-GPU Node | TPU | Workers | Parameter Server |
|---|---|---|---|---|
| Yes | No | No | Yes | No |
To have distribution and/or accelerators to your AI Platform training call, use parameters similar to the examples as shown below.
bash
    --master-machine-type standard_gpu
    --worker-machine-type standard_gpu \
    --worker-count 2 \
AI Platform training

Distributed XGBoost ML container documentation.
AI Platform training documentation.
End of explanation
"""


if not tf.io.gfile.exists(os.path.join(output_location, 'report.html')):
  raise RuntimeError('The file report.html was not found. Did the training job finish?')

with tf.io.gfile.GFile(os.path.join(output_location, 'report.html')) as f:
  display(HTML(f.read()))

"""
Explanation: Local training snippet
Note that the training can also be done locally with Docker
bash
docker run \
    -v /tmp:/tmp \
    -it gcr.io/aihub-c2t-containers/kfp-components/trainer/dist_xgboost:latest \
    --output-location /tmp/fm_classification \
    --training-data /tmp/iris_train.csv \
    --validation-data /tmp/iris_valid.csv \
    --target-column target \
    --data-type csv \
    --number-of-classes 3 \
    --objective multi:softprob
Inspect the Run Report
The "Run Report" will help you identify if the model was successfully trained.
End of explanation
"""


#@markdown ---
model = 'xgboost_iris' #@param {type:"string"}
version = 'v2' #@param {type:"string"}
#@markdown ---

"""
Explanation: Deployment parameters
End of explanation
"""


# the exact location of the model is in model_uri.txt
with tf.io.gfile.GFile(os.path.join(output_location, 'model_uri.txt')) as f:
  model_uri = f.read().replace('/model.bst', '')

# create a model
! gcloud ai-platform models create $model --region $REGION

# create a version
! gcloud ai-platform versions create $version --region $REGION \
  --model $model \
  --runtime-version 1.15 \
  --origin $model_uri \
  --framework XGBOOST \
  --project $PROJECT_ID

"""
Explanation: Deploy the model for serving

https://cloud.google.com/ai-platform/prediction/docs/deploying-models
End of explanation
"""


# format the data for serving
instances = validation.drop(columns='target').values.tolist()
validation_targets = validation['target']
display(instances[:2])

service = discovery.build('ml', 'v1')
name = 'projects/{project}/models/{model}/versions/{version}'.format(project=PROJECT_ID,
                                                                     model=model,
                                                                     version=version)
body = {'instances': instances}

response = service.projects().predict(name=name, body=body).execute()
if 'error' in response:
    raise RuntimeError(response['error'])

class_probabilties = [row for row in response['predictions']]
predicted_classes = np.array(class_probabilties).argmax(axis=1)
accuracy = (predicted_classes == validation_targets).mean()
print('Accuracy of the predictions: {}'.format(accuracy))

"""
Explanation: Use the endpoint for online predictions
End of explanation
"""


import witwidget
from witwidget.notebook.visualization import WitWidget, WitConfigBuilder

config_builder = WitConfigBuilder(examples=validation.values.tolist(),
                                  feature_names=validation.columns.tolist())
config_builder.set_ai_platform_model(project=PROJECT_ID,
                                     model=model,
                                     version=version)
config_builder.set_model_type('classification')
config_builder.set_target_feature('target')
WitWidget(config_builder)

"""
Explanation: Inspect the ML model

What if tool home page
Installation
End of explanation
"""


# Delete model version resource
! gcloud ai-platform versions delete $version --quiet --model $model 

# Delete model resource
! gcloud ai-platform models delete $model --quiet

# If training job is still running, cancel it
! gcloud ai-platform jobs cancel $job_name --quiet

# Delete Cloud Storage objects that were created
! gsutil -m rm -r $BUCKET_NAME

"""
Explanation: Cleaning up
To clean up all GCP resources used in this project, you can delete the GCP
project you used for the tutorial.
End of explanation
"""


