# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# +
# Copyright 2020 NVIDIA Corporation. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# -

# <img src="http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png" style="width: 90px; float: right;">
#
# # MovieLens: NVTabular for MultiHot Categories on MovieLens25M
#
# ## Overview
#
# NVTabular is a feature engineering and preprocessing library for tabular data designed to quickly and easily manipulate terabyte scale datasets used to train deep learning based recommender systems.  It provides a high level abstraction to simplify code and accelerates computation on the GPU using the RAPIDS cuDF library.<br><br>
# One challenge in recommender systems (tabular data) are multi-hot categorical features. A product can have multiple categories assigned, but the number of categories per product varies. For example, a movie can have one or multiple genres:<br>
# Father of the Bride Part II: \[Comedy\]<br>
# Toy Story: \[Adventure, Animation, Children, Comedy, Fantasy\]<br>
# Jumanji: \[Adventure, Children, Fantasy\]<br><br>
# One strategy is often to use only the first category or the most frequent ones. However, a better strategy is to use all provided categories per datapoint. [RAPID cuDF](https://github.com/rapidsai/cudf) added list support in its [latest release v0.16](https://medium.com/rapids-ai/two-years-in-a-snap-rapids-0-16-ae797795a5c4). NVTabular implemented support for multi-hot categorical features. In this notebook, we take a look on how to add multi-hot categorical features to our deep learning model in TensorFlow.
#
# ### Learning objectives
#
# This notebook explains, how to use multi-hot categorical features in a deep learning model.
# 1. Preprocess multi-hot categorical features with **NVTabular**. Deep learning models require continuous integers for embedding layers. Preprocessing multi-hot categorical features are more complicated than single-hot ones.
# 2. Add multi-hot categorical features to your deep learning model in TensorFlow 
#
# ### MovieLens25M
#
# The [MovieLens25M](https://grouplens.org/datasets/movielens/25m/) is a popular dataset for recommender systems and is used in academic publications. The dataset contains 25M movie ratings for 62,000 movies given by 162,000 users. Many projects use only the user/item/rating information of MovieLens, but the original dataset provides metadata for the movies, as well. For example, which genres a movie has. Although we may not improve state-of-the-art results with our neural network architecture, the purpose of this notebook is to explain how to integrate multi-hot categorical features into a neural network.

# ## Getting Started

# +
# External dependencies
import os
import cudf                 # cuDF is an implementation of Pandas-like Dataframe on GPU
import time
import gc

import pandas as pd
import nvtabular as nvt

from os import path
from sklearn.model_selection import train_test_split
# -

cudf.__version__

# We define our base directory, containing the data.

BASE_DIR = '/raid/data/ml/'

# If the data is not available in the base directory, we will download and unzip the data.

if not path.exists(BASE_DIR + 'ml-25m'):
    if not path.exists(BASE_DIR + 'ml-25m.zip'):
        os.system("wget http://files.grouplens.org/datasets/movielens/ml-25m.zip")
        os.system("mv ml-25m.zip " + BASE_DIR)
    os.system("unzip " + BASE_DIR + "ml-25m.zip -d " + BASE_DIR)

# ## Preparing the dataset with NVTabular

# First, we take a look on the movie metadata. 

movies = pd.read_csv(BASE_DIR + 'ml-25m/movies.csv')
movies.head()

# We can see, that genres are a multi-hot categorical features with different number of genres per movie. Currently, genres is a String and we want split the String into a list of Strings. In addition, we drop the title.

movies['genres'] = movies['genres'].str.split('|')
movies = movies.drop('title', axis=1)
movies.head()

# We used Pandas for this transformation. cuDF does not support `.str.split` with the parameter `expand=False`, yet, but the feature will be added in soon. We create a cudf.Dataframe from the Pandas.DataFrame.

movies = cudf.from_pandas(movies)

# We load the movie ratings.

ratings = cudf.read_csv(os.path.join(BASE_DIR, "ml-25m", "ratings.csv"))
ratings.head()

# We drop the timestamp column and split the ratings into training and test dataset. We use a simple random split, as this is a demonstration of multi-hot encoding.

ratings = ratings.drop('timestamp', axis=1)
train, valid = train_test_split(ratings, test_size=0.2, random_state=42)

# First, we need to define our NVTabular processing pipelines. NVTabular has already implemented multiple calculations, called `ops`. An `op` can be applied to a `ColumnGroup` from an overloaded `>>` operator, which in turn returns a new `ColumnGroup`. A `ColumnGroup` is a list of column names as text.<br><br>
# **Example:**<br>
# features = [*\<column name\>*, ...] >> *\<op1\>* >> *\<op2\>* >> ...

# Currently, our dataset consists of two separate dataframes. First, we use the `JoinExternal` operator to `left-join` the metadata (genres) to our rating dataset.

joined = (
    ['userId', 'movieId'] >> 
    nvt.ops.JoinExternal(movies, on=['movieId'])
)

# Embedding Layers of neural networks require, that categorial features are continuous, incremental Integers: 0, 1, 2, ... , |C|-1. We need to ensure that our categorical features fullfil the requirement.<br>
#
# Currently, our genres are a list of Strings. In addition, we should transform the single-hot categorical features userId and movieId, as well.<br>
# NVTabular provides the operator `Categorify`, which provides this functionality with a high-level API out of the box. In NVTabular release v0.3, list support was added for multi-hot categorical features. Both works in the same way with no need for changes.
#
#
# Next, we will add `Categorify`  for our categorical features (single hot: userId, movieId and multi-hot: genres).

cat_features = joined >> nvt.ops.Categorify()

# The ratings are on a scale between 1-5. We want to predict a binary target with 1 are all ratings `>=4` and 0 are all ratings `<=3`. We use the [LambdaOp](https://nvidia.github.io/NVTabular/main/api/ops/lambdaop.html) for it.

ratings = nvt.ColumnGroup(['rating']) >> (lambda col: (col>3).astype('int8'))

# We can visualize our calculation graph.

output = cat_features+ratings
(output).graph

# We initialize our NVTabular `workflow`.

proc = nvt.Workflow(output)

# We initialize NVTabular Datasets. First, we save it to a parquet file, that we can use the `part_size` parameter in `nvt.Dataset`. When the dataset contains a multi-hot feature, the workflow will write one file per partition. We want to have multiple output files and therefore, we need to use multiple partition by loading the dataset from disk. 

# +
train.to_parquet(BASE_DIR + 'train.parquet')
valid.to_parquet(BASE_DIR + 'valid.parquet')

del train
del valid
gc.collect()

train_iter = nvt.Dataset(BASE_DIR + 'train.parquet', part_size='100MB')
valid_iter = nvt.Dataset(BASE_DIR + 'valid.parquet', part_size='100MB')
# -

# First, we collect the training dataset statistics.

# +
# %%time

proc.fit(train_iter)
# -

# We clear our output directories.

# !rm -r $BASE_DIR/train
# !rm -r $BASE_DIR/valid

# We transform our workflow with `.transform`.

# +
# %time

proc.transform(train_iter).to_parquet(output_path=BASE_DIR + 'train/')
proc.transform(valid_iter).to_parquet(output_path=BASE_DIR + 'valid/')
# -

# We can take a look in the output dir.

# !ls $BASE_DIR/train

# ## TensorFlow: Training Neural Network with Multi-Hot categorical

# ### Reviewing data

# We can take a look on the data.

# +
import glob

TRAIN_PATHS = sorted(glob.glob(BASE_DIR + 'train/*.parquet'))
VALID_PATHS = sorted(glob.glob(BASE_DIR + 'valid/*.parquet'))
TRAIN_PATHS, VALID_PATHS
# -

# We can see, that genres are a list of Integers

df = cudf.read_parquet(TRAIN_PATHS[0])
df.head()

del df
gc.collect()

# ### Defining Hyperparameters

# First, we define the data schema and differentiate between single-hot and multi-hot categorical features. Note, that we do not have any numerical input features. 

BATCH_SIZE = 1024*32                            # Batch Size
CATEGORICAL_COLUMNS = ['movieId', 'userId']     # Single-hot
CATEGORICAL_MH_COLUMNS = ['genres']             # Multi-hot
NUMERIC_COLUMNS = []

# We get the embedding input and output dimensions.

EMBEDDING_TABLE_SHAPES = nvt.ops.get_embedding_sizes(proc)
EMBEDDING_TABLE_SHAPES

# ### Initializing NVTabular Data Loader for Tensorflow

# We import TensorFlow and some external dependencies, such as custom TensorFlow layers supporting multi-hot and the NVTabular TensorFlow data loader.

# +
import os, time
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"

import tensorflow as tf

from tensorflow.python.feature_column import feature_column_v2 as fc

# we can control how much memory to give tensorflow with this environment variable
# IMPORTANT: make sure you do this before you initialize TF's runtime, otherwise
# TF will have claimed all free GPU memory
os.environ['TF_MEMORY_ALLOCATION'] = "0.8" # fraction of free memory
from nvtabular.loader.tensorflow import KerasSequenceLoader, KerasSequenceValidater
from nvtabular.framework_utils.tensorflow import layers
from tensorflow.python.feature_column import feature_column_v2 as fc
# -

# First, we take a look on our data loader and how the data is represented as tensors. The NVTabular data loader are initialized as usually and we specify both single-hot and multi-hot categorical features as cat_names. The data loader will automatically recognize the single/multi-hot columns and represent them accordingly.

# +
train_dataset_tf = KerasSequenceLoader(
    TRAIN_PATHS, # you could also use a glob pattern
    batch_size=BATCH_SIZE,
    label_names=['rating'],
    cat_names=CATEGORICAL_COLUMNS+CATEGORICAL_MH_COLUMNS,
    cont_names=NUMERIC_COLUMNS,
    engine='parquet',
    shuffle=True,
    buffer_size=0.06, # how many batches to load at once
    parts_per_chunk=1
)

valid_dataset_tf = KerasSequenceLoader(
    VALID_PATHS, # you could also use a glob pattern
    batch_size=BATCH_SIZE,
    label_names=['rating'],
    cat_names = CATEGORICAL_COLUMNS+CATEGORICAL_MH_COLUMNS,
    cont_names=NUMERIC_COLUMNS,
    engine='parquet',
    shuffle=False,
    buffer_size=0.06,
    parts_per_chunk=1
)
# -

# Let's generate a batch and take a look on the input features.<br><br>
# We can see, that the single-hot categorical features (`userId` and `movieId`) have a shape of `(32768, 1)`, which is the batchsize (as usually).<br><br>
# For the multi-hot categorical feature `genres`, we receive two Tensors `genres__values` and `genres__nnzs`.<br><br>
# `genres__values` are the actual data, containing the genre IDs. Note that the Tensor has more values than the batch_size. The reason is, that one datapoint in the batch can contain more than one genre (multi-hot).<br>
# `genres__nnzs` are a supporting Tensor, describing how many genres are associated with each datapoint in the batch.<br><br>
# For example,
# - if the first value in `genres__nnzs` is `5`, then the first 5 values in `genres__values` are associated with the first datapoint in the batch (movieId/userId).<br>
# - if the second value in `genres__nnzs` is `2`, then the 6th and the 7th values in `genres__values` are associated with the second datapoint in the batch (continuing after the previous value stopped).<br> 
# - if the third value in `genres_nnzs` is `1`, then the 8th value in `genres__values` are associated with the third datapoint in the batch. 
# - and so on

batch = next(iter(train_dataset_tf))
batch[0]

# We can see that the sum of `genres__nnzs` is equal to the shape of `genres__values`.

tf.reduce_sum(batch[0]['genres__nnzs'])

# As each datapoint can have a different number of genres, it is more efficient to represent the genres as two flat tensors: One with the actual values (`genres__values`) and one with the length for each datapoint (`genres__nnzs`).

del batch
gc.collect()

# ### Defining Neural Network Architecture

# We will define a common neural network architecture for tabular data.
# * Single-hot categorical features are fed into an Embedding Layer
# * Each value of a multi-hot categorical features is fed into an Embedding Layer and the multiple Embedding outputs are combined via averaging
# * The output of the Embedding Layers are concatenated
# * The concatenated layers are fed through multiple feed-forward layers (Dense Layers with ReLU activations)
# * The final output is a single number with sigmoid activation function

# First, we will define some dictonary/lists for our network architecture.

inputs = {}    # tf.keras.Input placeholders for each feature to be used
emb_layers = []# output of all embedding layers, which will be concatenated

# We create `tf.keras.Input` tensors for all 4 input features.

for col in CATEGORICAL_COLUMNS:
    inputs[col] =  tf.keras.Input(
        name=col,
        dtype=tf.int32,
        shape=(1,)
    )
# Note that we need two input tensors for multi-hot categorical features
for col in CATEGORICAL_MH_COLUMNS:
    inputs[col+'__values'] = tf.keras.Input(
        name=f"{col}__values", 
        dtype=tf.int64, 
        shape=(1,)
    )
    inputs[col+'__nnzs'] = tf.keras.Input(
        name=f"{col}__nnzs", 
        dtype=tf.int64, 
        shape=(1,)
    )

# Next, we initialize Embedding Layers with `tf.feature_column.embedding_column`.

for col in CATEGORICAL_COLUMNS+CATEGORICAL_MH_COLUMNS:
    emb_layers.append(
        tf.feature_column.embedding_column(
            tf.feature_column.categorical_column_with_identity(
                col, 
                EMBEDDING_TABLE_SHAPES[col][0]                    # Input dimension (vocab size)
            ), EMBEDDING_TABLE_SHAPES[col][1]                     # Embedding output dimension
        )
    )
emb_layers

# NVTabular implemented a custom TensorFlow layer `layers.DenseFeatures`, which takes as an input the different `tf.Keras.Input` and pre-initialized `tf.feature_column` and automatically concatenate them into a flat tensor. In the case of multi-hot categorical features, `DenseFeatures` organizes the inputs `__values` and `__nnzs` to define a `RaggedTensor` and combine them. `DenseFeatures` can handle numeric inputs, as well, but MovieLens does not provide numerical input features.

emb_layer = layers.DenseFeatures(emb_layers)
x_emb_output = emb_layer(inputs)
x_emb_output

# We can see that the output shape of the concatenated layer is equal to the sum of the individual Embedding output dimensions (41 = 9+16+16).
#

EMBEDDING_TABLE_SHAPES

# We add multiple Dense Layers. Finally, we initialize the `tf.keras.Model` and add the optimizer.

# +
x = tf.keras.layers.Dense(128, activation='relu')(x_emb_output)
x = tf.keras.layers.Dense(128, activation='relu')(x)
x = tf.keras.layers.Dense(128, activation='relu')(x)
x = tf.keras.layers.Dense(1, activation='sigmoid')(x)

model = tf.keras.Model(inputs=inputs, outputs=x)
model.compile('sgd', 'binary_crossentropy')
# -

# ### Training the deep learning model

# We can train our model with `model.fit`.

# +
validation_callback = KerasSequenceValidater(valid_dataset_tf)

history = model.fit(train_dataset_tf, callbacks=[validation_callback], epochs=5)

StarcoderdataJupyterScriptsDedupFiltered