import os
import sys

# Google Cloud Notebook
if os.path.exists("/opt/deeplearning/metadata/env_version"):
    USER_FLAG = "--user"
else:
    USER_FLAG = ""

! pip3 install -U google-cloud-aiplatform $USER_FLAG

"""
Explanation: Vertex client library: AutoML tabular classification model for export to cloud
<table align="left">
  <td>
    <a href="https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_export_cloud.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/colab-logo-32px.png" alt="Colab logo"> Run in Colab
    </a>
  </td>
  <td>
    <a href="https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_export_cloud.ipynb">
      <img src="https://cloud.google.com/ml-engine/images/github-logo-32px.png" alt="GitHub logo">
      View on GitHub
    </a>
  </td>
</table>
<br/><br/><br/>
Overview
This tutorial demonstrates how to use the Vertex client library for Python to create tabular classification models to export as a cloud model using Google Cloud's AutoML.
Dataset
The dataset used for this tutorial is the Iris dataset from TensorFlow Datasets. This dataset does not require any feature engineering. The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of Iris flower species from a class of three species: setosa, virginica, or versicolor.
Objective
In this tutorial, you create a AutoML tabular classification model from a Python script using the Vertex client library, and then export the model in a TensorFlow SavedModel format. You can alternatively create models with AutoML using the the gcloud command-line tool or online using the Google Cloud Console.
The steps performed include:

Create a Vertex Dataset resource.
Train the model.
Export the model from the Model resource to Cloud Storage.
Download the model locally.
Make a local prediction.

Costs
This tutorial uses billable components of Google Cloud (GCP):

Vertex AI
Cloud Storage

Learn about Vertex AI
pricing and Cloud Storage
pricing, and use the Pricing
Calculator
to generate a cost estimate based on your projected usage.
Installation
Install the latest version of Vertex client library.
End of explanation
"""


! pip3 install -U google-cloud-storage $USER_FLAG

"""
Explanation: Install the latest GA version of google-cloud-storage library as well.
End of explanation
"""


if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

"""
Explanation: Restart the kernel
Once you've installed the Vertex client library and Google cloud-storage, you need to restart the notebook kernel so it can find the packages.
End of explanation
"""


PROJECT_ID = "[your-project-id]"  # @param {type:"string"}

if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)

! gcloud config set project $PROJECT_ID

"""
Explanation: Before you begin
GPU runtime
Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select Runtime > Change Runtime Type > GPU
Set up your Google Cloud project
The following steps are required, regardless of your notebook environment.


Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.


Make sure that billing is enabled for your project.


Enable the Vertex APIs and Compute Engine APIs.


The Google Cloud SDK is already installed in Google Cloud Notebook.


Enter your project ID in the cell below. Then run the  cell to make sure the
Cloud SDK uses the right project for all the commands in this notebook.


Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.
End of explanation
"""


REGION = "us-central1"  # @param {type: "string"}

"""
Explanation: Region
You can also change the REGION variable, which is used for operations
throughout the rest of this notebook.  Below are regions supported for Vertex. We recommend that you choose the region closest to you.

Americas: us-central1
Europe: europe-west4
Asia Pacific: asia-east1

You may not use a multi-regional bucket for training with Vertex. Not all regions provide support for all Vertex services. For the latest support per region, see the Vertex locations documentation
End of explanation
"""


from datetime import datetime

TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")

"""
Explanation: Timestamp
If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append onto the name of resources which will be created in this tutorial.
End of explanation
"""


# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

# If on Google Cloud Notebook, then don't execute this code
if not os.path.exists("/opt/deeplearning/metadata/env_version"):
    if "google.colab" in sys.modules:
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''

"""
Explanation: Authenticate your Google Cloud account
If you are using Google Cloud Notebook, your environment is already authenticated. Skip this step.
If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.
Otherwise, follow these steps:
In the Cloud Console, go to the Create service account key page.
Click Create service account.
In the Service account name field, enter a name, and click Create.
In the Grant this service account access to project section, click the Role drop-down list. Type "Vertex" into the filter box, and select Vertex Administrator. Type "Storage Object Admin" into the filter box, and select Storage Object Admin.
Click Create. A JSON file that contains your key downloads to your local environment.
Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.
End of explanation
"""


BUCKET_NAME = "gs://[your-bucket-name]"  # @param {type:"string"}

if BUCKET_NAME == "" or BUCKET_NAME is None or BUCKET_NAME == "gs://[your-bucket-name]":
    BUCKET_NAME = "gs://" + PROJECT_ID + "aip-" + TIMESTAMP

"""
Explanation: Create a Cloud Storage bucket
The following steps are required, regardless of your notebook environment.
This tutorial is designed to use training data that is in a public Cloud Storage bucket and a local Cloud Storage bucket for exporting the trained model. You may alternatively use your own training data that you have stored in a local Cloud Storage bucket.
Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
End of explanation
"""


! gsutil mb -l $REGION $BUCKET_NAME

"""
Explanation: Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.
End of explanation
"""


! gsutil ls -al $BUCKET_NAME

"""
Explanation: Finally, validate access to your Cloud Storage bucket by examining its contents:
End of explanation
"""


import time

from google.cloud.aiplatform import gapic as aip
from google.protobuf import json_format
from google.protobuf.json_format import MessageToJson, ParseDict
from google.protobuf.struct_pb2 import Struct, Value

"""
Explanation: Import Vertex client library
Import the Vertex client library into our Python environment.
End of explanation
"""


# API service endpoint
API_ENDPOINT = "{}-aiplatform.googleapis.com".format(REGION)

# Vertex location root path for your dataset, model and endpoint resources
PARENT = "projects/" + PROJECT_ID + "/locations/" + REGION

"""
Explanation: Vertex constants
Setup up the following constants for Vertex:

API_ENDPOINT: The Vertex API service endpoint for dataset, model, job, pipeline and endpoint services.
PARENT: The Vertex location root path for dataset, model, job, pipeline and endpoint resources.
End of explanation
"""


# Tabular Dataset type
DATA_SCHEMA = "gs://google-cloud-aiplatform/schema/dataset/metadata/tables_1.0.0.yaml"
# Tabular Labeling type
LABEL_SCHEMA = (
    "gs://google-cloud-aiplatform/schema/dataset/ioformat/table_io_format_1.0.0.yaml"
)
# Tabular Training task
TRAINING_SCHEMA = "gs://google-cloud-aiplatform/schema/trainingjob/definition/automl_tables_1.0.0.yaml"

"""
Explanation: AutoML constants
Set constants unique to AutoML datasets and training:

Dataset Schemas: Tells the Dataset resource service which type of dataset it is.
Data Labeling (Annotations) Schemas: Tells the Dataset resource service how the data is labeled (annotated).
Dataset Training Schemas: Tells the Pipeline resource service the task (e.g., classification) to train the model for.
End of explanation
"""


if "google.colab" in sys.modules:
    ! sudo apt update
    ! sudo apt install apt-transport-https ca-certificates curl software-properties-common
    ! curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    ! sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
    ! sudo apt update
    ! sudo apt install docker-ce

"""
Explanation: Install Docker (Colab or Local)
If you are using Google Cloud Notebook, Docker is already installed. Skip these steps.
By default, Docker is not installed on Colab. If you're running colab, then you need to do the following to install Docker.
End of explanation
"""


if "google.colab" in sys.modules:
    ! sudo service docker start

"""
Explanation: Start Docker service
Start the docker service.
End of explanation
"""


# client options same for all services
client_options = {"api_endpoint": API_ENDPOINT}


def create_dataset_client():
    client = aip.DatasetServiceClient(client_options=client_options)
    return client


def create_model_client():
    client = aip.ModelServiceClient(client_options=client_options)
    return client


def create_pipeline_client():
    client = aip.PipelineServiceClient(client_options=client_options)
    return client


clients = {}
clients["dataset"] = create_dataset_client()
clients["model"] = create_model_client()
clients["pipeline"] = create_pipeline_client()

for client in clients.items():
    print(client)

"""
Explanation: Tutorial
Now you are ready to start creating your own AutoML tabular classification model.
Set up clients
The Vertex client library works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the Vertex server.
You will use different clients in this tutorial for different steps in the workflow. So set them all up upfront.

Dataset Service for Dataset resources.
Model Service for Model resources.
Pipeline Service for training.
End of explanation
"""


IMPORT_FILE = "gs://cloud-samples-data/tables/iris_1000.csv"

"""
Explanation: Dataset
Now that your clients are ready, your first step is to create a Dataset resource instance. This step differs from Vision, Video and Language. For those products, after the Dataset resource is created, one then separately imports the data, using the import_data method.
For tabular, importing of the data is deferred until the training pipeline starts training the model. What do we do different? Well, first you won't be calling the import_data method. Instead, when you create the dataset instance you specify the Cloud Storage location of the CSV file or BigQuery location of the data table, which contains your tabular data as part of the Dataset resource's metadata.
Cloud Storage
metadata = {"input_config": {"gcs_source": {"uri": [gcs_uri]}}}
The format for a Cloud Storage path is:
gs://[bucket_name]/[folder(s)/[file]

BigQuery
metadata = {"input_config": {"bigquery_source": {"uri": [gcs_uri]}}}
The format for a BigQuery path is:
bq://[collection].[dataset].[table]

Note that the uri field is a list, whereby you can input multiple CSV files or BigQuery tables when your data is split across files.
Data preparation
The Vertex Dataset resource for tabular has a couple of requirements for your tabular data.

Must be in a CSV file or a BigQuery query.

CSV
For tabular classification, the CSV file has a few requirements:

The first row must be the heading -- note how this is different from Vision, Video and Language where the requirement is no heading.
All but one column are features.
One column is the label, which you will specify when you subsequently create the training pipeline.

Location of Cloud Storage training data.
Now set the variable IMPORT_FILE to the location of the CSV index file in Cloud Storage.
End of explanation
"""


count = ! gsutil cat $IMPORT_FILE | wc -l
print("Number of Examples", int(count[0]))

print("First 10 rows")
! gsutil cat $IMPORT_FILE | head

heading = ! gsutil cat $IMPORT_FILE | head -n1
label_column = str(heading).split(",")[-1].split("'")[0]
print("Label Column Name", label_column)
if label_column is None:
    raise Exception("label column missing")

"""
Explanation: Quick peek at your data
You will use a version of the Iris dataset that is stored in a public Cloud Storage bucket, using a CSV index file.
Start by doing a quick peek at the data. You count the number of examples by counting the number of rows in the CSV index file  (wc -l) and then peek at the first few rows.
You also need for training to know the heading name of the label column, which is save as label_column. For this dataset, it is the last column in the CSV file.
End of explanation
"""


TIMEOUT = 90


def create_dataset(name, schema, src_uri=None, labels=None, timeout=TIMEOUT):
    start_time = time.time()
    try:
        if src_uri.startswith("gs://"):
            metadata = {"input_config": {"gcs_source": {"uri": [src_uri]}}}
        elif src_uri.startswith("bq://"):
            metadata = {"input_config": {"bigquery_source": {"uri": [src_uri]}}}
        dataset = aip.Dataset(
            display_name=name,
            metadata_schema_uri=schema,
            labels=labels,
            metadata=json_format.ParseDict(metadata, Value()),
        )

        operation = clients["dataset"].create_dataset(parent=PARENT, dataset=dataset)
        print("Long running operation:", operation.operation.name)
        result = operation.result(timeout=TIMEOUT)
        print("time:", time.time() - start_time)
        print("response")
        print(" name:", result.name)
        print(" display_name:", result.display_name)
        print(" metadata_schema_uri:", result.metadata_schema_uri)
        print(" metadata:", dict(result.metadata))
        print(" create_time:", result.create_time)
        print(" update_time:", result.update_time)
        print(" etag:", result.etag)
        print(" labels:", dict(result.labels))
        return result
    except Exception as e:
        print("exception:", e)
        return None


result = create_dataset("iris-" + TIMESTAMP, DATA_SCHEMA, src_uri=IMPORT_FILE)

"""
Explanation: Dataset
Now that your clients are ready, your first step in training a model is to create a managed dataset instance, and then upload your labeled data to it.
Create Dataset resource instance
Use the helper function create_dataset to create the instance of a Dataset resource. This function does the following:

Uses the dataset client service.
Creates an Vertex Dataset resource (aip.Dataset), with the following parameters:
display_name: The human-readable name you choose to give it.
metadata_schema_uri: The schema for the dataset type.
metadata: The Cloud Storage or BigQuery location of the tabular data.
Calls the client dataset service method create_dataset, with the following parameters:
parent: The Vertex location root path for your Database, Model and Endpoint resources.
dataset: The Vertex dataset object instance you created.
The method returns an operation object.

An operation object is how Vertex handles asynchronous calls for long running operations. While this step usually goes fast, when you first use it in your project, there is a longer delay due to provisioning.
You can use the operation object to get status on the operation (e.g., create Dataset resource) or to cancel the operation, by invoking an operation method:
| Method      | Description |
| ----------- | ----------- |
| result()    | Waits for the operation to complete and returns a result object in JSON format.      |
| running()   | Returns True/False on whether the operation is still running.        |
| done()      | Returns True/False on whether the operation is completed. |
| canceled()  | Returns True/False on whether the operation was canceled. |
| cancel()    | Cancels the operation (this may take up to 30 seconds). |
End of explanation
"""


# The full unique ID for the dataset
dataset_id = result.name
# The short numeric ID for the dataset
dataset_short_id = dataset_id.split("/")[-1]

print(dataset_id)

"""
Explanation: Now save the unique dataset identifier for the Dataset resource instance you created.
End of explanation
"""


def create_pipeline(pipeline_name, model_name, dataset, schema, task):

    dataset_id = dataset.split("/")[-1]

    input_config = {
        "dataset_id": dataset_id,
        "fraction_split": {
            "training_fraction": 0.8,
            "validation_fraction": 0.1,
            "test_fraction": 0.1,
        },
    }

    training_pipeline = {
        "display_name": pipeline_name,
        "training_task_definition": schema,
        "training_task_inputs": task,
        "input_data_config": input_config,
        "model_to_upload": {"display_name": model_name},
    }

    try:
        pipeline = clients["pipeline"].create_training_pipeline(
            parent=PARENT, training_pipeline=training_pipeline
        )
        print(pipeline)
    except Exception as e:
        print("exception:", e)
        return None
    return pipeline

"""
Explanation: Train the model
Now train an AutoML tabular classification model using your Vertex Dataset resource. To train the model, do the following steps:

Create an Vertex training pipeline for the Dataset resource.
Execute the pipeline to start the training.

Create a training pipeline
You may ask, what do we use a pipeline for? You typically use pipelines when the job (such as training) has multiple steps, generally in sequential order: do step A, do step B, etc. By putting the steps into a pipeline, we gain the benefits of:

Being reusable for subsequent training jobs.
Can be containerized and ran as a batch job.
Can be distributed.
All the steps are associated with the same pipeline job for tracking progress.

Use this helper function create_pipeline, which takes the following parameters:

pipeline_name: A human readable name for the pipeline job.
model_name: A human readable name for the model.
dataset: The Vertex fully qualified dataset identifier.
schema: The dataset labeling (annotation) training schema.
task: A dictionary describing the requirements for the training job.

The helper function calls the Pipeline client service'smethod create_pipeline, which takes the following parameters:

parent: The Vertex location root path for your Dataset, Model and Endpoint resources.
training_pipeline: the full specification for the pipeline training job.

Let's look now deeper into the minimal requirements for constructing a training_pipeline specification:

display_name: A human readable name for the pipeline job.
training_task_definition: The dataset labeling (annotation) training schema.
training_task_inputs: A dictionary describing the requirements for the training job.
model_to_upload: A human readable name for the model.
input_data_config: The dataset specification.
dataset_id: The Vertex dataset identifier only (non-fully qualified) -- this is the last part of the fully-qualified identifier.
fraction_split: If specified, the percentages of the dataset to use for training, test and validation. Otherwise, the percentages are automatically selected by AutoML.
End of explanation
"""


TRANSFORMATIONS = [
    {"auto": {"column_name": "sepal_width"}},
    {"auto": {"column_name": "sepal_length"}},
    {"auto": {"column_name": "petal_length"}},
    {"auto": {"column_name": "petal_width"}},
]

PIPE_NAME = "iris_pipe-" + TIMESTAMP
MODEL_NAME = "iris_model-" + TIMESTAMP

task = Value(
    struct_value=Struct(
        fields={
            "target_column": Value(string_value=label_column),
            "prediction_type": Value(string_value="classification"),
            "train_budget_milli_node_hours": Value(number_value=1000),
            "disable_early_stopping": Value(bool_value=False),
            "transformations": json_format.ParseDict(TRANSFORMATIONS, Value()),
        }
    )
)

response = create_pipeline(PIPE_NAME, MODEL_NAME, dataset_id, TRAINING_SCHEMA, task)

"""
Explanation: Construct the task requirements
Next, construct the task requirements. Unlike other parameters which take a Python (JSON-like) dictionary, the task field takes a Google protobuf Struct, which is very similar to a Python dictionary. Use the json_format.ParseDict method for the conversion.
The minimal fields you need to specify are:

prediction_type: Whether we are doing "classification" or "regression".
target_column: The CSV heading column name for the column we want to predict (i.e., the label).
train_budget_milli_node_hours: The maximum time to budget (billed) for training the model, where 1000 = 1 hour.
disable_early_stopping: Whether True/False to let AutoML use its judgement to stop training early or train for the entire budget.
transformations: Specifies the feature engineering for each feature column.

For transformations, the list must have an entry for each column. The outer key field indicates the type of feature engineering for the corresponding column. In this tutorial, you set it to "auto" to tell AutoML to automatically determine it.
Finally, create the pipeline by calling the helper function create_pipeline, which returns an instance of a training pipeline object.
End of explanation
"""


# The full unique ID for the pipeline
pipeline_id = response.name
# The short numeric ID for the pipeline
pipeline_short_id = pipeline_id.split("/")[-1]

print(pipeline_id)

"""
Explanation: Now save the unique identifier of the training pipeline you created.
End of explanation
"""


def get_training_pipeline(name, silent=False):
    response = clients["pipeline"].get_training_pipeline(name=name)
    if silent:
        return response

    print("pipeline")
    print(" name:", response.name)
    print(" display_name:", response.display_name)
    print(" state:", response.state)
    print(" training_task_definition:", response.training_task_definition)
    print(" training_task_inputs:", dict(response.training_task_inputs))
    print(" create_time:", response.create_time)
    print(" start_time:", response.start_time)
    print(" end_time:", response.end_time)
    print(" update_time:", response.update_time)
    print(" labels:", dict(response.labels))
    return response


response = get_training_pipeline(pipeline_id)

"""
Explanation: Get information on a training pipeline
Now get pipeline information for just this training pipeline instance. The helper function  gets the job information for just this job by calling the the job client service's get_training_pipeline method, with the following parameter:

name: The Vertex fully qualified pipeline identifier.

When the model is done training, the pipeline state will be PIPELINE_STATE_SUCCEEDED.
End of explanation
"""


while True:
    response = get_training_pipeline(pipeline_id, True)
    if response.state != aip.PipelineState.PIPELINE_STATE_SUCCEEDED:
        print("Training job has not completed:", response.state)
        model_to_deploy_id = None
        if response.state == aip.PipelineState.PIPELINE_STATE_FAILED:
            raise Exception("Training Job Failed")
    else:
        model_to_deploy = response.model_to_upload
        model_to_deploy_id = model_to_deploy.name
        print("Training Time:", response.end_time - response.start_time)
        break
    time.sleep(60)

print("model to deploy:", model_to_deploy_id)

"""
Explanation: Deployment
Training the above model may take upwards of 30 minutes time.
Once your model is done training, you can calculate the actual time it took to train the model by subtracting end_time from start_time. For your model, you will need to know the fully qualified Vertex Model resource identifier, which the pipeline service assigned to it. You can get this from the returned pipeline instance as the field model_to_deploy.name.
End of explanation
"""


def list_model_evaluations(name):
    response = clients["model"].list_model_evaluations(parent=name)
    for evaluation in response:
        print("model_evaluation")
        print(" name:", evaluation.name)
        print(" metrics_schema_uri:", evaluation.metrics_schema_uri)
        metrics = json_format.MessageToDict(evaluation._pb.metrics)
        for metric in metrics.keys():
            print(metric)
        print("logloss", metrics["logLoss"])
        print("auPrc", metrics["auPrc"])

    return evaluation.name


last_evaluation = list_model_evaluations(model_to_deploy_id)

"""
Explanation: Model information
Now that your model is trained, you can get some information on your model.
Evaluate the Model resource
Now find out how good the model service believes your model is. As part of training, some portion of the dataset was set aside as the test (holdout) data, which is used by the pipeline service to evaluate the model.
List evaluations for all slices
Use this helper function list_model_evaluations, which takes the following parameter:

name: The Vertex fully qualified model identifier for the Model resource.

This helper function uses the model client service's list_model_evaluations method, which takes the same parameter. The response object from the call is a list, where each element is an evaluation metric.
For each evaluation (you probably only have one) we then print all the key names for each metric in the evaluation, and for a small set (logLoss and auPrc) you will print the result.
End of explanation
"""


MODEL_DIR = BUCKET_NAME + "/" + "iris"


def export_model(name, format, gcs_dest):
    output_config = {
        "artifact_destination": {"output_uri_prefix": gcs_dest},
        "export_format_id": format,
    }
    response = clients["model"].export_model(name=name, output_config=output_config)
    print("Long running operation:", response.operation.name)
    result = response.result(timeout=1800)
    metadata = response.operation.metadata
    artifact_uri = str(metadata.value).split("\\")[-1][4:-1]
    print("Artifact Uri", artifact_uri)
    return artifact_uri


model_package = export_model(model_to_deploy_id, "tf-saved-model", MODEL_DIR)

"""
Explanation: Export as cloud model
You can export an AutoML tabular classification model as a TensorFlow SavedFormat model which you can then custom deploy to Google Cloud or download locally. Use this helper function export_model to export the model to Google Cloud, which takes the following parameters:

name: The Vertex fully qualified identifier for the Model resource.
gcs_dest: The Cloud Storage location to store the SavedFormat model artifacts to.

This function calls the Model client service's method export_model, with the following parameters:

name: The Vertex fully qualified identifier for the Model resource.
output_config: The destination information for the exported model.
artifact_destination.output_uri_prefix: The Cloud Storage location to store the SavedFormat model artifacts to.
export_format_id: The format to save the model format as. For AutoML tabular classification there is just one option:
tf-saved-model: TensorFlow SavedFormat

The method returns a long running operation response. We will wait sychronously for the operation to complete by calling the response.result(), which will block until the model is exported.
End of explanation
"""


print("Model Package:", model_package)
print("Contents:")
! gsutil ls $model_package

print("\nTF Saved Model")
path = model_package + "/predict"
files = ! gsutil ls $path
saved_dir = files[1]
print(saved_dir)
! gsutil ls $saved_dir

"""
Explanation: Running the exported model package
Your model is now stored in a TensorFlow SavedModel format, along with a model server binary in a Cloud Storage bucket. Load the model locally from the Cloud Storage bucket, and then you can do some things like deploy the model and serving binary locally using Docker with a prebuilt Google Cloud container designed for running the model server.
Model package layout
Let's start with looking at what is in the exported package. The exported model package will be stored in your Cloud Storage bucket as:
&lt;your-gcs-bucket&gt;/&lt;model_name&gt;/model-&lt;model id&gt;/tf-saved-model/&lt;timedstamp&gt;

Next, take a look at the contents of the model package.
End of explanation
"""


! gsutil cp -r $model_package .

"""
Explanation: Downloading the export model package.
Your first step is to download the exported model package locally.
End of explanation
"""


subdir = model_package.split("/")[-1]
print(subdir)
! mv $subdir tbl_exported

"""
Explanation: Rename the model package directory name
In this tutorial, you will launch the model package using Docker. The timestamp in the last component of the path name for the package contains a ':', which is invalid for Docker. You will need to rename. In the next step, we rename the last component to tbl_exported.
End of explanation
"""


MODEL_SERVER = "gcr.io/cloud-aiplatform/automl_tables/prediction_server"
PORT = 8081

docker_id = ! docker run -d -v `pwd`/tbl_exported:/models/default -p 8081:8080 -it $MODEL_SERVER

"""
Explanation: Run the exported model package
This model package has dependencies on a prebuilt Google Cloud container that is specific for serving an export AutoML tabular classification model:
"gcr.io/cloud-aiplatform/automl_tables/prediction_server"

This container is currently not supported by Vertex endpoint. You can though launch the model server with the container, either locally or on another compute instance you manage.
In this tutorial, you will launch it with docker. The Docker container will run in the background of this notebook, using detached mode by specifying the -d option to docker.
End of explanation
"""


import time

time.sleep(10)
! curl -X GET http://localhost:8081/health

"""
Explanation: Check health status
We need to let a bit of time to go by for the model to be loaded. Let's pause 10 seconds and then do a health check. If the health check comes back empty, it is ready to go.
End of explanation
"""


! docker container ls --latest

"""
Explanation: Verify loading of docker image
You can also verify the loading of the Docker image with docker container ls. We will add --latest to get information on the latest (this image) that was loaded.
End of explanation
"""


INSTANCES = {
    "instances": [
        {
            "petal_length": "1.4",
            "petal_width": "1.3",
            "sepal_length": "5.1",
            "sepal_width": "2.8",
        }
    ]
}

import json

with open("test.json", "w") as f:
    data = json.dumps(INSTANCES)
    f.write(data)

"""
Explanation: Make a prediction request
Now do a prediction with your exported model.
Prediction request format
The format for the prediction request is a JSON object of the form:
{ "instances": [ { "column_name_1": value, "column_name_2": value, … } , … ] }

Place your prediction request in a text file, such as:
test.json
End of explanation
"""


! curl -X POST --data @/home/jupyter/test.json http://localhost:8081/predict

"""
Explanation: Send the prediction request
You can the send the prediction request using CURL:
curl -X POST --data @test.json http://localhost:8081/predict
End of explanation
"""


print(docker_id)
! docker kill $docker_id

"""
Explanation: Shutdown the Docker container
End of explanation
"""


delete_dataset = True
delete_pipeline = True
delete_model = True
delete_endpoint = True
delete_batchjob = True
delete_customjob = True
delete_hptjob = True
delete_bucket = True

# Delete the dataset using the Vertex fully qualified identifier for the dataset
try:
    if delete_dataset and "dataset_id" in globals():
        clients["dataset"].delete_dataset(name=dataset_id)
except Exception as e:
    print(e)

# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline
try:
    if delete_pipeline and "pipeline_id" in globals():
        clients["pipeline"].delete_training_pipeline(name=pipeline_id)
except Exception as e:
    print(e)

# Delete the model using the Vertex fully qualified identifier for the model
try:
    if delete_model and "model_to_deploy_id" in globals():
        clients["model"].delete_model(name=model_to_deploy_id)
except Exception as e:
    print(e)

# Delete the endpoint using the Vertex fully qualified identifier for the endpoint
try:
    if delete_endpoint and "endpoint_id" in globals():
        clients["endpoint"].delete_endpoint(name=endpoint_id)
except Exception as e:
    print(e)

# Delete the batch job using the Vertex fully qualified identifier for the batch job
try:
    if delete_batchjob and "batch_job_id" in globals():
        clients["job"].delete_batch_prediction_job(name=batch_job_id)
except Exception as e:
    print(e)

# Delete the custom job using the Vertex fully qualified identifier for the custom job
try:
    if delete_customjob and "job_id" in globals():
        clients["job"].delete_custom_job(name=job_id)
except Exception as e:
    print(e)

# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job
try:
    if delete_hptjob and "hpt_job_id" in globals():
        clients["job"].delete_hyperparameter_tuning_job(name=hpt_job_id)
except Exception as e:
    print(e)

if delete_bucket and "BUCKET_NAME" in globals():
    ! gsutil rm -r $BUCKET_NAME

"""
Explanation: Cleaning up
To clean up all GCP resources used in this project, you can delete the GCP
project you used for the tutorial.
Otherwise, you can delete the individual resources you created in this tutorial:

Dataset
Pipeline
Model
Endpoint
Batch Job
Custom Job
Hyperparameter Tuning Job
Cloud Storage Bucket
End of explanation
"""


