#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Explanation: Copyright 2020 The TensorFlow Authors.
End of explanation
"""


!pip install tensorflow_decision_forests

"""
Explanation: Build, train and evaluate models with TensorFlow Decision Forests
<table class="tfo-notebook-buttons" align="left">
  <td>
    <a target="_blank" href="https://www.tensorflow.org/decision_forests/tutorials/beginner_colab"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
  </td>
  <td>
    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/decision-forests/blob/main/documentation/tutorials/beginner_colab.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
  </td>
  <td>
    <a target="_blank" href="https://github.com/tensorflow/decision-forests/blob/main/documentation/tutorials/beginner_colab.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View on GitHub</a>
  </td>
  <td>
    <a href="https://storage.googleapis.com/tensorflow_docs/decision-forests/documentation/tutorials/beginner_colab.ipynb"><img src="https://www.tensorflow.org/images/download_logo_32px.png" />Download notebook</a>
  </td>
</table>

Introduction
Decision Forests (DF) are a large family of Machine Learning algorithms for
supervised classification, regression and ranking. As the name suggests, DFs use
decision trees as a building block. Today, the two most popular DF training
algorithms are Random Forests and
Gradient Boosted Decision Trees. Both algorithms are ensemble techniques that use multiple decision trees, but differ on how they do it.
TensorFlow Decision Forests (TF-DF) is a library for the training,
evaluation, interpretation and inference of Decision Forest models.
In this tutorial, you will learn how to:

Train a binary classification Random Forest on a dataset containing numerical, categorical and missing features.
Evaluate the model on a test dataset.
Prepare the model for
    TensorFlow Serving.
Examine the overall structure of the model and the importance of each feature.
Re-train the model with a different learning algorithm (Gradient Boosted Decision Trees).
Use a different set of input features.
Change the hyperparameters of the model.
Preprocess the features.
Train a model for regression.
Train a model for ranking.

Detailed documentation is available in the user manual.
The example directory contains other end-to-end examples.
Installing TensorFlow Decision Forests
Install TF-DF by running the following cell.
End of explanation
"""


!pip install wurlitzer

"""
Explanation: Wurlitzer is needed to display the detailed training logs in Colabs (when using verbose=2 in the model constructor).
End of explanation
"""


import tensorflow_decision_forests as tfdf

import os
import numpy as np
import pandas as pd
import tensorflow as tf
import math

"""
Explanation: Importing libraries
End of explanation
"""


#@title

from IPython.core.magic import register_line_magic
from IPython.display import Javascript
from IPython.display import display as ipy_display

# Some of the model training logs can cover the full
# screen if not compressed to a smaller viewport.
# This magic allows setting a max height for a cell.
@register_line_magic
def set_cell_height(size):
  ipy_display(
      Javascript("google.colab.output.setIframeHeight(0, true, {maxHeight: " +
                 str(size) + "})"))

# Check the version of TensorFlow Decision Forests
print("Found TensorFlow Decision Forests v" + tfdf.__version__)

"""
Explanation: The hidden code cell limits the output height in colab.
End of explanation
"""


# Download the dataset
!wget -q https://storage.googleapis.com/download.tensorflow.org/data/palmer_penguins/penguins.csv -O /tmp/penguins.csv

# Load a dataset into a Pandas Dataframe.
dataset_df = pd.read_csv("/tmp/penguins.csv")

# Display the first 3 examples.
dataset_df.head(3)

"""
Explanation: Training a Random Forest model
In this section, we train, evaluate, analyse and export a binary classification Random Forest trained on the Palmer's Penguins dataset.
<center>
<img src="https://allisonhorst.github.io/palmerpenguins/man/figures/palmerpenguins.png" width="150"/></center>
Note: The dataset was exported to a csv file without pre-processing: library(palmerpenguins); write.csv(penguins, file="penguins.csv", quote=F, row.names=F). 
Load the dataset and convert it in a tf.Dataset
This dataset is very small (300 examples) and stored as a .csv-like file. Therefore, use Pandas to load it.
Note: Pandas is practical as you don't have to type in name of the input features to load them. For larger datasets (>1M examples), using the
TensorFlow Dataset to read the files may be better suited.
Let's assemble the dataset into a csv file (i.e. add the header), and load it:
End of explanation
"""


# Encode the categorical label into an integer.
#
# Details:
# This stage is necessary if your classification label is represented as a
# string. Note: Keras expected classification labels to be integers.

# Name of the label column.
label = "species"

classes = dataset_df[label].unique().tolist()
print(f"Label classes: {classes}")

dataset_df[label] = dataset_df[label].map(classes.index)

"""
Explanation: The dataset contains a mix of numerical (e.g. bill_depth_mm), categorical
(e.g. island) and missing features. TF-DF supports all these feature types natively (differently than NN based models), therefore there is no need for preprocessing in the form of one-hot encoding, normalization or extra is_present feature.
Labels are a bit different: Keras metrics expect integers. The label (species) is stored as a string, so let's convert it into an integer.
End of explanation
"""


# Split the dataset into a training and a testing dataset.

def split_dataset(dataset, test_ratio=0.30):
  """Splits a panda dataframe in two."""
  test_indices = np.random.rand(len(dataset)) < test_ratio
  return dataset[~test_indices], dataset[test_indices]


train_ds_pd, test_ds_pd = split_dataset(dataset_df)
print("{} examples in training, {} examples for testing.".format(
    len(train_ds_pd), len(test_ds_pd)))

"""
Explanation: Next split the dataset into training and testing:
End of explanation
"""


train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label)
test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(test_ds_pd, label=label)

"""
Explanation: And finally, convert the pandas dataframe (pd.Dataframe) into tensorflow datasets (tf.data.Dataset):
End of explanation
"""


%set_cell_height 300

# Specify the model.
model_1 = tfdf.keras.RandomForestModel()

# Train the model.
model_1.fit(x=train_ds)

"""
Explanation: Notes: pd_dataframe_to_tf_dataset could have converted the label to integer for you.
And, if you wanted to create the tf.data.Dataset yourself, there is a couple of things to remember:

The learning algorithms work with a one-epoch dataset and without shuffling.
The batch size does not impact the training algorithm, but a small value might slow down reading the dataset.

Train the model
End of explanation
"""


model_1.compile(metrics=["accuracy"])
evaluation = model_1.evaluate(test_ds, return_dict=True)
print()

for name, value in evaluation.items():
  print(f"{name}: {value:.4f}")

"""
Explanation: Remarks

No input features are specified. Therefore, all the columns will be used as
    input features except for the label. The feature used by the model are shown
    in the training logs and in the model.summary().
DFs consume natively numerical, categorical, categorical-set features and
    missing-values. Numerical features do not need to be normalized. Categorical
    string values do not need to be encoded in a dictionary.
No training hyper-parameters are specified. Therefore the default
    hyper-parameters will be used. Default hyper-parameters provide
    reasonable results in most situations.
Calling compile on the model before the fit is optional. Compile can be
    used to provide extra evaluation metrics.
Training algorithms do not need validation datasets. If a validation dataset
    is provided, it will only be used to show metrics.
Add a verbose argument to RandomForestModel to control the amount of
    displayed training logs. Set verbose=0 to hide most of the logs. Set 
    verbose=2 to show all the logs.

Note: A Categorical-Set feature is composed of a set of categorical values (while a Categorical is only one value). More details and examples are given later.
Evaluate the model
Let's evaluate our model on the test dataset.
End of explanation
"""


model_1.save("/tmp/my_saved_model")

"""
Explanation: Remark: The test accuracy (0.86514) is close to the Out-of-bag accuracy
(0.8672) shown in the training logs.
See the Model Self Evaluation section below for more evaluation methods.
Prepare this model for TensorFlow Serving.
Export the model to the SavedModel format for later re-use e.g.
TensorFlow Serving.
End of explanation
"""


tfdf.model_plotter.plot_model_in_colab(model_1, tree_idx=0, max_depth=3)

"""
Explanation: Plot the model
Plotting a decision tree and following the first branches helps learning about decision forests. In some cases, plotting a model can even be used for debugging.
Because of the difference in the way they are trained, some models are more interesting to plan than others. Because of the noise injected during training and the depth of the trees, plotting Random Forest is less informative than plotting a CART or the first tree of a Gradient Boosted Tree.
Never the less, let's plot the first tree of our Random Forest model:
End of explanation
"""


%set_cell_height 300
model_1.summary()

"""
Explanation: The root node on the left contains the first condition (bill_depth_mm &gt;= 16.55), number of examples (240) and label distribution (the red-blue-green bar).
Examples that evaluates true to bill_depth_mm &gt;= 16.55 are branched to the green path. The other ones are branched to the red path.
The deeper the node, the more pure they become i.e. the label distribution is biased toward a subset of classes. 
Note: Over the mouse on top of the plot for details.
Model structure and feature importance
The overall structure of the model is show with .summary(). You will see:

Type: The learning algorithm used to train the model (Random Forest in
    our case).
Task: The problem solved by the model (Classification in our case).
Input Features: The input features of the model.
Variable Importance: Different measures of the importance of each
    feature for the model.
Out-of-bag evaluation: The out-of-bag evaluation of the model. This is a
    cheap and efficient alternative to cross-validation.
Number of {trees, nodes} and other metrics: Statistics about the
    structure of the decisions forests.

Remark: The summary's content depends on the learning algorithm (e.g.
Out-of-bag is only available for Random Forest) and the hyper-parameters (e.g.
the mean-decrease-in-accuracy variable importance can be disabled in the
hyper-parameters).
End of explanation
"""


# The input features
model_1.make_inspector().features()

# The feature importances
model_1.make_inspector().variable_importances()

"""
Explanation: The information in summary are all available programatically using the model inspector:
End of explanation
"""


model_1.make_inspector().evaluation()

"""
Explanation: The content of the summary and the inspector depends on the learning algorithm (tfdf.keras.RandomForestModel in this case) and its hyper-parameters (e.g. compute_oob_variable_importances=True will trigger the computation of Out-of-bag variable importances for the Random Forest learner).
Model Self Evaluation
During training TFDF models can self evaluate even if no validation dataset is provided to the fit() method. The exact logic depends on the model. For example, Random Forest will use Out-of-bag evaluation while Gradient Boosted Trees will use internal train-validation.
Note: While this evaluation is  computed during training, it is NOT computed on the training dataset and can be used as a low quality evaluation.
The model self evaluation is available with the inspector's evaluation():
End of explanation
"""


%set_cell_height 150
model_1.make_inspector().training_logs()

"""
Explanation: Plotting the training logs
The training logs show the quality of the model (e.g. accuracy evaluated on the out-of-bag or validation dataset) according to the number of trees in the model. These logs are helpful to study the balance between model size and model quality.
The logs are available in multiple ways:

Displayed in during training if fit() is wrapped in with sys_pipes(): (see example above).
At the end of the model summary i.e. model.summary() (see example above).
Programmatically, using the model inspector i.e. model.make_inspector().training_logs().
Using TensorBoard

Let's try the options 2 and 3:
End of explanation
"""


import matplotlib.pyplot as plt

logs = model_1.make_inspector().training_logs()

plt.figure(figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.plot([log.num_trees for log in logs], [log.evaluation.accuracy for log in logs])
plt.xlabel("Number of trees")
plt.ylabel("Accuracy (out-of-bag)")

plt.subplot(1, 2, 2)
plt.plot([log.num_trees for log in logs], [log.evaluation.loss for log in logs])
plt.xlabel("Number of trees")
plt.ylabel("Logloss (out-of-bag)")

plt.show()

"""
Explanation: Let's plot it:
End of explanation
"""


# This cell start TensorBoard that can be slow.
# Load the TensorBoard notebook extension
%load_ext tensorboard
# Google internal version
# %load_ext google3.learning.brain.tensorboard.notebook.extension

# Clear existing results (if any)
!rm -fr "/tmp/tensorboard_logs"

# Export the meta-data to tensorboard.
model_1.make_inspector().export_to_tensorboard("/tmp/tensorboard_logs")

# docs_infra: no_execute
# Start a tensorboard instance.
%tensorboard --logdir "/tmp/tensorboard_logs"

"""
Explanation: This dataset is small. You can see the model converging almost immediately.
Let's use TensorBoard:
End of explanation
"""


tfdf.keras.get_all_models()

"""
Explanation: <!-- <img class="tfo-display-only-on-site" src="images/beginner_tensorboard.png"/> -->

Re-train the model with a different learning algorithm
The learning algorithm is defined by the model class. For
example, tfdf.keras.RandomForestModel() trains a Random Forest, while
tfdf.keras.GradientBoostedTreesModel() trains a Gradient Boosted Decision
Trees.
The learning algorithms are listed by calling tfdf.keras.get_all_models() or in the
learner list.
End of explanation
"""


# help works anywhere.
help(tfdf.keras.RandomForestModel)

# ? only works in ipython or notebooks, it usually opens on a separate panel.
tfdf.keras.RandomForestModel?

"""
Explanation: The description of the learning algorithms and their hyper-parameters are also available in the API reference and builtin help:
End of explanation
"""


feature_1 = tfdf.keras.FeatureUsage(name="bill_length_mm")
feature_2 = tfdf.keras.FeatureUsage(name="island")

all_features = [feature_1, feature_2]

# Note: This model is only trained with two features. It will not be as good as
# the one trained on all features.

model_2 = tfdf.keras.GradientBoostedTreesModel(
    features=all_features, exclude_non_specified_features=True)

model_2.compile(metrics=["accuracy"])
model_2.fit(x=train_ds, validation_data=test_ds)

print(model_2.evaluate(test_ds, return_dict=True))

"""
Explanation: Using a subset of features
The previous example did not specify the features, so all the columns were used
as input feature (except for the label). The following example shows how to
specify input features.
End of explanation
"""


%set_cell_height 300

feature_1 = tfdf.keras.FeatureUsage(name="year", semantic=tfdf.keras.FeatureSemantic.CATEGORICAL)
feature_2 = tfdf.keras.FeatureUsage(name="bill_length_mm")
feature_3 = tfdf.keras.FeatureUsage(name="sex")
all_features = [feature_1, feature_2, feature_3]

model_3 = tfdf.keras.GradientBoostedTreesModel(features=all_features, exclude_non_specified_features=True)
model_3.compile( metrics=["accuracy"])

model_3.fit(x=train_ds, validation_data=test_ds)

"""
Explanation: Note: As expected, the accuracy is lower than previously.
TF-DF attaches a semantics to each feature. This semantics controls how
the feature is used by the model. The following semantics are currently supported:

Numerical: Generally for quantities or counts with full ordering. For
    example, the age of a person, or the number of items in a bag. Can be a
    float or an integer. Missing values are represented with float(Nan) or with
    an empty sparse tensor.
Categorical: Generally for a type/class in finite set of possible values
    without ordering. For example, the color RED in the set {RED, BLUE, GREEN}.
    Can be a string or an integer. Missing values are represented as "" (empty
    sting), value -2 or with an empty sparse tensor.
Categorical-Set: A set of categorical values. Great to represent
    tokenized text. Can be a string or an integer in a sparse tensor or a
    ragged tensor (recommended). The order/index of each item doesn't matter.

If not specified, the semantics is inferred from the representation type and shown in the training logs:

int, float (dense or sparse) → Numerical semantics.
str (dense or sparse) → Categorical semantics
int, str (ragged) → Categorical-Set semantics

In some cases, the inferred semantics is incorrect. For example: An Enum stored as an integer is semantically categorical, but it will be detected as numerical. In this case, you should specify the semantic argument in the input. The education_num field of the Adult dataset is classical example.
This dataset doesn't contain such a feature. However, for the demonstration, we will make the model treat the year as a categorical feature:
End of explanation
"""


# A classical but slighly more complex model.
model_6 = tfdf.keras.GradientBoostedTreesModel(
    num_trees=500, growing_strategy="BEST_FIRST_GLOBAL", max_depth=8)
model_6.fit(x=train_ds)

# A more complex, but possibly, more accurate model.
model_7 = tfdf.keras.GradientBoostedTreesModel(
    num_trees=500,
    growing_strategy="BEST_FIRST_GLOBAL",
    max_depth=8,
    split_axis="SPARSE_OBLIQUE",
    categorical_algorithm="RANDOM",
    )
model_7.fit(x=train_ds)

"""
Explanation: Note that year is in the list of CATEGORICAL features (unlike the first run).
Hyper-parameters
Hyper-parameters are parameters of the training algorithm that impact
the quality of the final model. They are specified in the model class
constructor. The list of hyper-parameters is visible with the question mark colab command (e.g. ?tfdf.keras.GradientBoostedTreesModel).
Alternatively, you can find them on the TensorFlow Decision Forest Github or the Yggdrasil Decision Forest documentation.
The default hyper-parameters of each algorithm matches approximatively the initial publication paper. To ensure consistancy, new features and their matching hyper-parameters are always disable by default. That's why it is a good idea to tune your hyper-parameters.
End of explanation
"""


# A good template of hyper-parameters.
model_8 = tfdf.keras.GradientBoostedTreesModel(hyperparameter_template="benchmark_rank1")
model_8.fit(x=train_ds)

"""
Explanation: As new training methods are published and implemented, combinaisons of hyper-parameters can emerge as good or almost-always-better than the default parameters. To avoid changing the default hyper-parameter values these good combinaisons are indexed and available as hyper-parameter templates.
For example, the benchmark_rank1 template is the best combinaison on our internal benchmarks. Those templates are versioned to allow training configuration stability e.g. benchmark_rank1@v1.
End of explanation
"""


# The hyper-parameter templates of the Gradient Boosted Tree model.
print(tfdf.keras.GradientBoostedTreesModel.predefined_hyperparameters())

"""
Explanation: The available tempaltes are available with predefined_hyperparameters. Note that different learning algorithms have different templates, even if the name is similar.
End of explanation
"""


%set_cell_height 300

body_mass_g = tf.keras.layers.Input(shape=(1,), name="body_mass_g")
body_mass_kg = body_mass_g / 1000.0

bill_length_mm = tf.keras.layers.Input(shape=(1,), name="bill_length_mm")

raw_inputs = {"body_mass_g": body_mass_g, "bill_length_mm": bill_length_mm}
processed_inputs = {"body_mass_kg": body_mass_kg, "bill_length_mm": bill_length_mm}

# "preprocessor" contains the preprocessing logic.
preprocessor = tf.keras.Model(inputs=raw_inputs, outputs=processed_inputs)

# "model_4" contains both the pre-processing logic and the decision forest.
model_4 = tfdf.keras.RandomForestModel(preprocessing=preprocessor)
model_4.fit(x=train_ds)

model_4.summary()

"""
Explanation: Feature Preprocessing
Pre-processing features is sometimes necessary to consume signals with complex
structures, to regularize the model or to apply transfer learning.
Pre-processing can be done in one of three ways:


Preprocessing on the Pandas dataframe. This solution is easy to implement
    and generally suitable for experimentation. However, the
    pre-processing logic will not be exported in the model by model.save().


Keras Preprocessing: While
    more complex than the previous solution, Keras Preprocessing is packaged in
    the model.


TensorFlow Feature Columns:
    This API is part of the TF Estimator library (!= Keras) and planned for
    deprecation. This solution is interesting when using existing preprocessing
    code.


Note: Using TensorFlow Hub
pre-trained embedding is often, a great way to consume text and image with
TF-DF. For example, hub.KerasLayer("https://tfhub.dev/google/nnlm-en-dim128/2"). See the Intermediate tutorial for more details.
In the next example, pre-process the body_mass_g feature into body_mass_kg = body_mass_g / 1000. The bill_length_mm is consumed without pre-processing. Note that such
monotonic transformations have generally no impact on decision forest models.
End of explanation
"""


def g_to_kg(x):
  return x / 1000

feature_columns = [
    tf.feature_column.numeric_column("body_mass_g", normalizer_fn=g_to_kg),
    tf.feature_column.numeric_column("bill_length_mm"),
]

preprocessing = tf.keras.layers.DenseFeatures(feature_columns)

model_5 = tfdf.keras.RandomForestModel(preprocessing=preprocessing)
model_5.fit(x=train_ds)

"""
Explanation: The following example re-implements the same logic using TensorFlow Feature
Columns.
End of explanation
"""


# Download the dataset.
!wget -q https://storage.googleapis.com/download.tensorflow.org/data/abalone_raw.csv -O /tmp/abalone.csv

dataset_df = pd.read_csv("/tmp/abalone.csv")
print(dataset_df.head(3))

# Split the dataset into a training and testing dataset.
train_ds_pd, test_ds_pd = split_dataset(dataset_df)
print("{} examples in training, {} examples for testing.".format(
    len(train_ds_pd), len(test_ds_pd)))

# Name of the label column.
label = "Rings"

train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label, task=tfdf.keras.Task.REGRESSION)
test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=label, task=tfdf.keras.Task.REGRESSION)

%set_cell_height 300

# Configure the model.
model_7 = tfdf.keras.RandomForestModel(task = tfdf.keras.Task.REGRESSION)

# Train the model.
model_7.fit(x=train_ds)

# Evaluate the model on the test dataset.
model_7.compile(metrics=["mse"])
evaluation = model_7.evaluate(test_ds, return_dict=True)

print(evaluation)
print()
print(f"MSE: {evaluation['mse']}")
print(f"RMSE: {math.sqrt(evaluation['mse'])}")

"""
Explanation: Training a regression model
The previous example trains a classification model (TF-DF does not differentiate
between binary classification and multi-class classification). In the next
example, train a regression model on the
Abalone dataset. The
objective of this dataset is to predict the number of shell's rings of an
abalone.
Note: The csv file is assembled by appending UCI's header and data files. No preprocessing was applied.
<center>
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/3/33/LivingAbalone.JPG/800px-LivingAbalone.JPG" width="200"/></center>
End of explanation
"""


%set_cell_height 200

archive_path = tf.keras.utils.get_file("letor.zip",
  "https://download.microsoft.com/download/E/7/E/E7EABEF1-4C7B-4E31-ACE5-73927950ED5E/Letor.zip",
  extract=True)

# Path to the train and test dataset using libsvm format.
raw_dataset_path = os.path.join(os.path.dirname(archive_path),"OHSUMED/Data/All/OHSUMED.txt")

"""
Explanation: Training a ranking model
Finaly, after having trained a classification and a regression models, train a ranking model.
The goal of a ranking is to order items by importance. The "value" of
relevance does not matter directly. Ranking a set of documents with regard to
a user query is an example of ranking problem: It is only important to get the right order, where the top documents matter more.
TF-DF expects for ranking datasets to be presented in a "flat" format. A
document+query dataset might look like that:
query | document_id | feature_1 | feature_2 | relevance/label
----- | ----------- | --------- | --------- | ---------------
cat   | 1           | 0.1       | blue      | 4
cat   | 2           | 0.5       | green     | 1
cat   | 3           | 0.2       | red       | 2
dog   | 4           | NA        | red       | 0
dog   | 5           | 0.2       | red       | 1
dog   | 6           | 0.6       | green     | 1
The relevance/label is a floating point numerical value between 0 and 5
(generally between 0 and 4) where 0 means "completely unrelated", 4 means "very
relevant" and 5 means "the same as the query".
Interestingly, decision forests are often good rankers, and many
state-of-the-art ranking models are decision forests.
In this example, use a sample of the
LETOR3
dataset. More precisely, we want to download the OHSUMED.zip from the LETOR3 repo. This dataset is stored in the
libsvm format, so we will need to convert it to csv.
End of explanation
"""


def convert_libsvm_to_csv(src_path, dst_path):
  """Converts a libsvm ranking dataset into a flat csv file.
  
  Note: This code is specific to the LETOR3 dataset.
  """
  dst_handle = open(dst_path, "w")
  first_line = True
  for src_line in open(src_path,"r"):
    # Note: The last 3 items are comments.
    items = src_line.split(" ")[:-3]
    relevance = items[0]
    group = items[1].split(":")[1]
    features = [ item.split(":") for item in items[2:]]

    if first_line:
      # Csv header
      dst_handle.write("relevance,group," + ",".join(["f_" + feature[0] for feature in features]) + "\n")
      first_line = False
    dst_handle.write(relevance + ",g_" + group + "," + (",".join([feature[1] for feature in features])) + "\n")
  dst_handle.close()

# Convert the dataset.
csv_dataset_path="/tmp/ohsumed.csv"
convert_libsvm_to_csv(raw_dataset_path, csv_dataset_path)

# Load a dataset into a Pandas Dataframe.
dataset_df = pd.read_csv(csv_dataset_path)

# Display the first 3 examples.
dataset_df.head(3)

train_ds_pd, test_ds_pd = split_dataset(dataset_df)
print("{} examples in training, {} examples for testing.".format(
    len(train_ds_pd), len(test_ds_pd)))

# Display the first 3 examples of the training dataset.
train_ds_pd.head(3)

"""
Explanation: The dataset is stored as a .txt file in a specific format, so first convert it into a csv file.
End of explanation
"""


# Name of the relevance and grouping columns.
relevance = "relevance"

ranking_train_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=relevance, task=tfdf.keras.Task.RANKING)
ranking_test_ds = tfdf.keras.pd_dataframe_to_tf_dataset(train_ds_pd, label=relevance, task=tfdf.keras.Task.RANKING)

%set_cell_height 400

model_8 = tfdf.keras.GradientBoostedTreesModel(
    task=tfdf.keras.Task.RANKING,
    ranking_group="group",
    num_trees=50)

model_8.fit(x=ranking_train_ds)

"""
Explanation: In this dataset, the relevance defines the ground-truth rank among rows of the same group.
End of explanation
"""


%set_cell_height 400

model_8.summary()

"""
Explanation: At this point, keras does not propose any ranking metrics. Instead, the training and validation (a GBDT uses a validation dataset) are shown in the training
logs. In this case the loss is LAMBDA_MART_NDCG5, and the final (i.e. at
the end of the training) NDCG (normalized discounted cumulative gain) is 0.510136 (see line Final model valid-loss: -0.510136).
Note that the NDCG is a value between 0 and 1. The larget the NDCG, the better
the model. For this reason, the loss to be -NDCG.
As before, the model can be analysed:
End of explanation
"""


