# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="fOaCES6J_FKK"
# # Hugging Face transformers 
#
# Hugging Face is an organization that is on a path to solve and democratize AI through natural language. Their open-source library 'transformers'  is very popular among the NLP community. It is very useful and powerful for several NLP and NLU tasks. It includes thousands of pre-trained models in about 100+ languages. One of the many advantages of the transformer library is that it is compatible with both PyTorch and TensorFlow.
#
# We can install transformers directly using pip as shown in the following: 
#

# + id="o7PQZSHt_FKT"
# %%capture
# !pip install transformers==3.5.1

# + [markdown] id="5ehF5uSR_FKZ"
#
# As we can see, in this book, we use transformers version 3.5.1. Now that we installed transformers, let's get started. 

# + [markdown] id="wi1Nq51R_FKc"
# # Generating BERT embedding 
# In this section, we will learn how to extract embeddings from the pre-trained BERT. Consider the sentence 'I love Paris'. Let's see how to obtain the contextualized word embedding of all the words in the sentence using the pre-trained BERT model with Hugging Face's transformer library. 
#
# First, let's import the necessary modules:

# + id="qmWTWzRX_FKe"
from transformers import BertModel, BertTokenizer
import torch

# + [markdown] id="zFPzX86E_FKh"
# Next, we download the pre-trained BERT model. We can check all the available pre-trained BERT models here - https://huggingface.co/transformers/pre-trained_models.html.We use the 'bert-base-uncased' model. As the name suggests, it is the BERT-base model with 12 encoders and it is trained with uncased tokens. Since we are using the BERT-base, the representation size will be 768. 
#
# Download and load the pre-trained bert-base-uncasedmodel:

# + colab={"base_uri": "https://localhost:8080/", "height": 115, "referenced_widgets": ["1758400be5b14d1383a14be51f129388", "66169204df0344a78bb29b5c66235f38", "96a5d14686674cebaab3fe3457579f15", "71872c1d14f04434a502db8328803169", "a54875acc17044a0b6683d9cc94ed95c", "e900b312c23641549d729f1e9f98849a", "6e66fccb4d4e4518acdefab5a51c9708", "<KEY>", "e79a470be88c4c2e9c5a7cc7e3eca6b1", "2320668b356245449a2ba6991e21dc71", "8c195883eb9f453781baf994903a07c4", "420c835be5304faeb77864c5f924b768", "ab9cede857e142f69b227bd15e7ac825", "08fc204525d24f0ca56c42c2890737ca", "f7a38dcc24f14a298d540d8827554757", "084587c872cb464f8afc0f88e6b33452"]} id="swqqjz08_FKj" outputId="e1ca7551-d012-41d0-e38e-d46c3169d9b8"
model = BertModel.from_pretrained('bert-base-uncased')

# + [markdown] id="KODXve4P_FKl"
# Next, we download and load the tokenizer which is used for pretraining the bert-base-uncased model: 

# + colab={"base_uri": "https://localhost:8080/", "height": 66, "referenced_widgets": ["a1fc82d07a6c4a46bef6b105b3f53bac", "b36a5b3e28394ee39058c537572ecd72", "67196c8f26d3422daf6c27589757b480", "7d1ef8f4cbfe46e1a8f9d6f6c944e93a", "6835f5dcea2e4832a1dc76dc28e24dd0", "7de7bcd4a0d841298de7f5ac7e5cc81c", "<KEY>", "8c7da9ea9fec4e7fbdc6ad56ee63c33f"]} id="lPHa9dI5_FKn" outputId="06f9ae89-87b2-49d4-c898-60d13a34c20d"
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# + [markdown] id="qIHYHG5A_FKr"
#
# Now, let's see how to preprocess the input before feeding it to the BERT. 
#

# + [markdown] id="j8El2dcG_FKs"
# ## Preprocessing the input 
# Define the sentence:

# + id="8Hkc5o3B_FKu"
sentence = 'I love Paris'

# + [markdown] id="FVn3-2XM_FKv"
# Tokenize the sentence and obtain the tokens:

# + id="4Ui-3oWr_FKw"
tokens = tokenizer.tokenize(sentence)

# + [markdown] id="RSYRLcvV_FKy"
# Let's print the tokens:

# + colab={"base_uri": "https://localhost:8080/"} id="CGHtPj5H_FKz" outputId="96afb8ca-8565-4b69-dad2-fc9471dd7227"
print(tokens)

# + [markdown] id="rL8VgpcM_FK0"
# Now, we will add the [CLS] token at the beginning and [SEP] token at the end of the tokens list: 

# + id="3CpARvDc_FK0"
tokens = ['[CLS]'] + tokens + ['[SEP]']

# + [markdown] id="jbFEyrk7_FK2"
# Let's look at our updated tokens list:

# + colab={"base_uri": "https://localhost:8080/"} id="Uaf_JScw_FK3" outputId="82f09500-1324-4022-9b5a-e165716c2443"
print(tokens)

# + [markdown] id="wVD29L7A_FK3"
# As we can observe, we have [CLS] token at the beginning and sep token at the end of our tokens list. We can also observe that length of our tokens is 5.
#
# Say, we need to keep the length of our tokens list to 7, then, in that case, we will add two [PAD] tokens at the end as shown in the following:
#
#

# + id="cqMI--rA_FK4"
tokens = tokens + ['[PAD]'] + ['[PAD]']

# + [markdown] id="XEEHs7CV_FK6"
# Let's print our updated tokens list:

# + colab={"base_uri": "https://localhost:8080/"} id="2tKmh_hk_FK6" outputId="31926f5d-4329-4c21-fc57-b3318b8f0c30"
print(tokens)

# + [markdown] id="aPd9TPWE_FK7"
#
#
# As we can observe, now we have the tokens list consists of [PAD] tokens and the length of our tokens list is 7. 
#
# Next, we create the attention mask. We set the attention mask value to 1 if the token is not a [PAD] token else we will set the attention mask to 0 as shown below:
#
#

# + id="3SbiaBws_FK8"
attention_mask = [1 if i!= '[PAD]' else 0 for i in tokens]

# + [markdown] id="SELKgxsG_FK9"
# Let's print the attention_mask:

# + colab={"base_uri": "https://localhost:8080/"} id="NmtZKFFq_FK-" outputId="83e13594-2f24-427d-cd99-bfba8930adaf"
print(attention_mask)

# + [markdown] id="qvtvC8HW_FK-"
#
# As we can observe, we have attention mask values 0 at the position where have [PAD] token and 1 at other positions. 
#
# Next, we convert all the tokens to their token_ids as shown below: 
#

# + id="KGbsHfxL_FK_"
token_ids = tokenizer.convert_tokens_to_ids(tokens)

# + [markdown] id="zaQWwz_y_FK_"
#
# Let's have a look at the token_ids:

# + colab={"base_uri": "https://localhost:8080/"} id="iih_wWeT_FLA" outputId="f83bd6c5-300a-4c1a-fac9-dc54db1e157b"
print(token_ids)

# + [markdown] id="S9RL1cBs_FLA"
#
# From the above output, we can observe that each token is mapped to a unique token id.
#
# Now, we convert the token_ids and attention_mask to tensors as shown below:

# + id="KQzsDUez_FLB"
token_ids = torch.tensor(token_ids).unsqueeze(0)
attention_mask = torch.tensor(attention_mask).unsqueeze(0)

# + [markdown] id="pQn5de_a_FLC"
#
# That's it. Next, we feed the token_ids and attention_mask to the pre-trained BERT model and get the embedding. 

# + [markdown] id="qYIDWAO-_FLD"
# ## Getting the embedding 
#
# As shown in the following code, we feed the token_ids, and attention_mask to the model and get the embeddings. Note that the model returns the output as a tuple with two values. The first value indicates the hidden state representation, hidden_rep and it consists of the representation of all the tokens obtained from the final encoder (encoder 12), and the second value, cls_head consists of the representation of the [CLS] token: 

# + id="Rh-Ohh71_FLE"
hidden_rep, cls_head = model(token_ids, attention_mask = attention_mask)

# + [markdown] id="F3Kfvd3F_FLF"
# In the preceding code, hidden_rep contains the embedding(representation) of all the tokens in our input. Let's print the shape of hidden_rep tensor:

# + colab={"base_uri": "https://localhost:8080/"} id="Gx1RfsaU_FLF" outputId="8c50bb57-c886-4d3a-d029-e2c87e43adda"
print(hidden_rep.shape)


# + [markdown] id="zWxFowIS_FLG"
# The size [1,7,768] indicates the[batch_size, sequence_length, hidden_size].
#
# Our batch size is 1, the sequence length is the token length, since we have 7 tokens, the sequence length is 7, and the hidden size is the representation (embedding) size and it is 768 for the BERT-base model. 
#
# We can obtain the representation of each token as: 
#
# - hidden_rep[0][0] gives the representation of the first token which is [CLS]
# - hidden_rep[0][1] gives the representation of the second token which is 'I' 
# - hidden_repo[0][2] gives the representation of the third token which is 'love' 
#
# In this way, we can obtain the contextual representation of all the tokens. This is basically the contextualized word embeddings of all the words in the given sentence. 
#
# Now, let's take a look at the cls_head. It contains the representation of the [CLS] token. Let's print the shape of cls_head :

# + colab={"base_uri": "https://localhost:8080/"} id="zAL6KEXw_FLI" outputId="fb3a0ca9-9a1a-44e2-f292-d0b2c3f9bd68"
print(cls_head.shape)

# + [markdown] id="UL8eMwZY_FLJ"
# The above code will print:
#
# torch.Size([1, 768])
#
# The size [1,768] indicates the[batch_size, hidden_size].
#
# We learned that cls_head holds the aggregate representation of the sentence, so we can use the cls_head as the representation of the given sentence 'I love Paris'. 
#
# We learned how to extract embeddings from the pre-trained BERT. But these are the embeddings obtained only from the topmost encoder layer of BERT which is encoder 12. Can we also extract the embeddings from all the encoder layers of BERT? Yes! We will find out how to do that in the next section. 

StarcoderdataJupyterScriptsDedupFiltered