SCBert/SCBert.py
def tokenize (self, data, MAX_LEN = 256) :
            """
            This function call the tokenizer corresponding to the BERT model specified in the constructor. Then it generates
            a vector of id corresponding to the words in the vocabulary. Also an attention vector which has the same size as the vector of id with ones 
            for real words and zeros corresponding to padding id. 
            
            Parameters
            ----------
            data : `Numpy array` or `Pandas DataFrame`
                Corresponds to your datas, must be a list of your texts texts. 
                
            MAX_LEN : int, optional
                Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and 
                MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence. 
                Default: The default is 256.
                

            Returns
            -------
            tokenized_texts : List of list of strings
                Corresponds of the list of your tokenized texts. Each text has been transformed into a vector of word according the tokenizer of the BERT model stated into the constructor.
            input_ids_tensor : List of List of int
                Same as tokenized_texts but with ids corresponding to the tokens, converted into torch tensor. 
            masks_tensor : List of list of float
                Corresponds to the attention torch tensor
            """
            tokenized_texts = np.array([self.tokenizer.tokenize(text) for text in data])
            input_ids = np.array([self.tokenizer.encode(text, max_length=MAX_LEN, pad_to_max_length=True,  add_special_tokens= True ) for text in data])
            # Create attention masks
            attention_masks = []
          
            # Create a mask of 1s for each token followed by 0s for padding
            for seq in input_ids:
                seq_mask = [float(i != self.pad_id) for i in seq]
                attention_masks.append(seq_mask)
          
            # Convert all of our data into torch tensors, the required datatype for our model
            input_ids_tensor = torch.tensor(input_ids)
            masks_tensor = torch.tensor(attention_masks)
            
            return tokenized_texts, input_ids_tensor, masks_tensor

            This function call the tokenizer corresponding to the BERT model specified in the constructor. Then it generates
            a vector of id corresponding to the words in the vocabulary. Also an attention vector which has the same size as the vector of id with ones 
            for real words and zeros corresponding to padding id. 
            
            Parameters
            ----------
            data : `Numpy array` or `Pandas DataFrame`
                Corresponds to your datas, must be a list of your texts texts. 
                
            MAX_LEN : int, optional
                Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and 
                MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence. 
                Default: The default is 256.
                

            Returns
            -------
            tokenized_texts : List of list of strings
                Corresponds of the list of your tokenized texts. Each text has been transformed into a vector of word according the tokenizer of the BERT model stated into the constructor.
            input_ids_tensor : List of List of int
                Same as tokenized_texts but with ids corresponding to the tokens, converted into torch tensor. 
            masks_tensor : List of list of float
                Corresponds to the attention torch tensor
            
This function call the tokenizer corresponding to the BERT model specified in the constructor. Then it generates
a vector of id corresponding to the words in the vocabulary. Also an attention vector which has the same size as the vector of id with ones
for real words and zeros corresponding to padding id.

Parameters

data : `Numpy array` or `Pandas DataFrame`
Corresponds to your datas, must be a list of your texts texts.

MAX_LEN : int, optional
Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and
MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence.
Default: The default is 256.

Returns

tokenized_texts : List of list of strings
Corresponds of the list of your tokenized texts. Each text has been transformed into a vector of word according the tokenizer of the BERT model stated into the constructor.
input_ids_tensor : List of List of int
Same as tokenized_texts but with ids corresponding to the tokens, converted into torch tensor.
masks_tensor : List of list of float
Corresponds to the attention torch tensor
This function call the tokenizer corresponding to the BERT model specified in the constructor . Then it generates a vector of id corresponding to the words in the vocabulary . Also an attention vector which has the same size as the vector of id with ones for real words and zeros corresponding to padding id . Parameters data : ` Numpy array ` or ` Pandas DataFrame ` Corresponds to your datas must be a list of your texts texts . MAX_LEN : int optional Corresponds to the max number of word to take into account during tokenizing . If a text is 350 words long and MAX_LEN is 256 the text will be truncated after the 256th word starting at the beginning of the sentence . Default : The default is 256 . Returns tokenized_texts : List of list of strings Corresponds of the list of your tokenized texts . Each text has been transformed into a vector of word according the tokenizer of the BERT model stated into the constructor . input_ids_tensor : List of List of int Same as tokenized_texts but with ids corresponding to the tokens converted into torch tensor . masks_tensor : List of list of float Corresponds to the attention torch tensor
def tokenize (self, data, MAX_LEN = 256) :
            tokenized_texts = np.array([self.tokenizer.tokenize(text) for text in data])
            input_ids = np.array([self.tokenizer.encode(text, max_length=MAX_LEN, pad_to_max_length=True,  add_special_tokens= True ) for text in data])
            attention_masks = []
            for seq in input_ids:
                seq_mask = [float(i != self.pad_id) for i in seq]
                attention_masks.append(seq_mask)
            input_ids_tensor = torch.tensor(input_ids)
            masks_tensor = torch.tensor(attention_masks)
            return tokenized_texts, input_ids_tensor, masks_tensor
def tokenize ( self , data , MAX_LEN = 256 ) : tokenized_texts = np . array ( [ self . tokenizer . tokenize ( text ) for text in data ] ) input_ids = np . array ( [ self . tokenizer . encode ( text , max_length = MAX_LEN , pad_to_max_length = True , add_special_tokens = True ) for text in data ] ) attention_masks = [ ] for seq in input_ids : seq_mask = [ float ( i != self . pad_id ) for i in seq ] attention_masks . append ( seq_mask ) input_ids_tensor = torch . tensor ( input_ids ) masks_tensor = torch . tensor ( attention_masks ) return tokenized_texts , input_ids_tensor , masks_tensor
This function call the tokenizer corresponding to the BERT model specified in the constructor.
This function call the tokenizer corresponding to the BERT model specified in the constructor .
"""
            This function call the tokenizer corresponding to the BERT model specified in the constructor. Then it generates
            a vector of id corresponding to the words in the vocabulary. Also an attention vector which has the same size as the vector of id with ones 
            for real words and zeros corresponding to padding id. 
            
            Parameters
            ----------
            data : `Numpy array` or `Pandas DataFrame`
                Corresponds to your datas, must be a list of your texts texts. 
                
            MAX_LEN : int, optional
                Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and 
                MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence. 
                Default: The default is 256.
                

            Returns
            -------
            tokenized_texts : List of list of strings
                Corresponds of the list of your tokenized texts. Each text has been transformed into a vector of word according the tokenizer of the BERT model stated into the constructor.
            input_ids_tensor : List of List of int
                Same as tokenized_texts but with ids corresponding to the tokens, converted into torch tensor. 
            masks_tensor : List of list of float
                Corresponds to the attention torch tensor
            """ # Create attention masks # Create a mask of 1s for each token followed by 0s for padding # Convert all of our data into torch tensors, the required datatype for our model
self
data
MAX_LEN