SCBert/SCBert.py
def vectorize (self, data, MAX_LEN = 256,sentence_pooling_method="average", word_pooling_method="average", layers = 11, batch_size=50, path_to_save=None) :
            """
            Transform the input raw data into tensors according to the selected models and the pooling methods. 
            
            Parameters
            ----------
            data : `Numpy array` or `Pandas DataFrame`
                Corresponds to your datas, must be a list of your texts texts. 
                
            MAX_LEN : int, optional
                Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and 
                MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence. 
                Default: The default is 256.
            sentence_pooling_method : str, optional
                Corresponds to the method of pooling to create a unique vector for each text
                The default is "average".
            word_pooling_method : str, optional
                Corresponds to the method of word pooling method in the case multiple layers have been stated. In other words it is the way to compute a single word vector from multiple layers.
                The default is "average".
            layers : int or list, optional
                Corresponds to the BERT layers to use to create the embeddings of words.
                The default is 11.
            batch_size : int, optional
                Size of batch when executing the forward pass into the BERT model, should get lower as your computational power gets lower. 
                The default is 50.
            path_to_save : str or None, optional
                Is the path to save the vector of texts if you want to avoid doing this computation again as it may be long. 
                if None, nothing will be saved. 
                The default is None.

            Returns
            -------
            texts_vectors : list
                A list of tensors, each tensor corresponding to an input text.

            """
            tokenized_texts, input_ids_tensor, masks_tensor = self.tokenize(data,MAX_LEN)
            texts_vectors = self.forward_and_pool(input_ids_tensor,masks_tensor,sentence_pooling_method,word_pooling_method,layers,batch_size,path_to_save)

            return texts_vectors

            Transform the input raw data into tensors according to the selected models and the pooling methods. 
            
            Parameters
            ----------
            data : `Numpy array` or `Pandas DataFrame`
                Corresponds to your datas, must be a list of your texts texts. 
                
            MAX_LEN : int, optional
                Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and 
                MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence. 
                Default: The default is 256.
            sentence_pooling_method : str, optional
                Corresponds to the method of pooling to create a unique vector for each text
                The default is "average".
            word_pooling_method : str, optional
                Corresponds to the method of word pooling method in the case multiple layers have been stated. In other words it is the way to compute a single word vector from multiple layers.
                The default is "average".
            layers : int or list, optional
                Corresponds to the BERT layers to use to create the embeddings of words.
                The default is 11.
            batch_size : int, optional
                Size of batch when executing the forward pass into the BERT model, should get lower as your computational power gets lower. 
                The default is 50.
            path_to_save : str or None, optional
                Is the path to save the vector of texts if you want to avoid doing this computation again as it may be long. 
                if None, nothing will be saved. 
                The default is None.

            Returns
            -------
            texts_vectors : list
                A list of tensors, each tensor corresponding to an input text.

            
Transform the input raw data into tensors according to the selected models and the pooling methods.
Parameters

data : `Numpy array` or `Pandas DataFrame`
Corresponds to your datas, must be a list of your texts texts.

MAX_LEN : int, optional
Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and
MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence.
Default: The default is 256.
sentence_pooling_method : str, optional
Corresponds to the method of pooling to create a unique vector for each text
The default is "average".
word_pooling_method : str, optional
Corresponds to the method of word pooling method in the case multiple layers have been stated. In other words it is the way to compute a single word vector from multiple layers.
The default is "average".
layers : int or list, optional
Corresponds to the BERT layers to use to create the embeddings of words.
The default is 11.
batch_size : int, optional
Size of batch when executing the forward pass into the BERT model, should get lower as your computational power gets lower.
The default is 50.
path_to_save : str or None, optional
Is the path to save the vector of texts if you want to avoid doing this computation again as it may be long.
if None, nothing will be saved.
The default is None.

Returns

texts_vectors : list
A list of tensors, each tensor corresponding to an input text.
Transform the input raw data into tensors according to the selected models and the pooling methods . Parameters data : ` Numpy array ` or ` Pandas DataFrame ` Corresponds to your datas must be a list of your texts texts . MAX_LEN : int optional Corresponds to the max number of word to take into account during tokenizing . If a text is 350 words long and MAX_LEN is 256 the text will be truncated after the 256th word starting at the beginning of the sentence . Default : The default is 256 . sentence_pooling_method : str optional Corresponds to the method of pooling to create a unique vector for each text The default is " average " . word_pooling_method : str optional Corresponds to the method of word pooling method in the case multiple layers have been stated . In other words it is the way to compute a single word vector from multiple layers . The default is " average " . layers : int or list optional Corresponds to the BERT layers to use to create the embeddings of words . The default is 11 . batch_size : int optional Size of batch when executing the forward pass into the BERT model should get lower as your computational power gets lower . The default is 50 . path_to_save : str or None optional Is the path to save the vector of texts if you want to avoid doing this computation again as it may be long . if None nothing will be saved . The default is None . Returns texts_vectors : list A list of tensors each tensor corresponding to an input text .
def vectorize (self, data, MAX_LEN = 256,sentence_pooling_method="average", word_pooling_method="average", layers = 11, batch_size=50, path_to_save=None) :
            tokenized_texts, input_ids_tensor, masks_tensor = self.tokenize(data,MAX_LEN)
            texts_vectors = self.forward_and_pool(input_ids_tensor,masks_tensor,sentence_pooling_method,word_pooling_method,layers,batch_size,path_to_save)
            return texts_vectors
def vectorize ( self , data , MAX_LEN = 256 , sentence_pooling_method = "average" , word_pooling_method = "average" , layers = 11 , batch_size = 50 , path_to_save = None ) : tokenized_texts , input_ids_tensor , masks_tensor = self . tokenize ( data , MAX_LEN ) texts_vectors = self . forward_and_pool ( input_ids_tensor , masks_tensor , sentence_pooling_method , word_pooling_method , layers , batch_size , path_to_save ) return texts_vectors
Transform the input raw data into tensors according to the selected models and the pooling methods.
Transform the input raw data into tensors according to the selected models and the pooling methods .
"""
            Transform the input raw data into tensors according to the selected models and the pooling methods. 
            
            Parameters
            ----------
            data : `Numpy array` or `Pandas DataFrame`
                Corresponds to your datas, must be a list of your texts texts. 
                
            MAX_LEN : int, optional
                Corresponds to the max number of word to take into account during tokenizing. If a text is 350 words long and 
                MAX_LEN is 256, the text will be truncated after the 256th word, starting at the beginning of the sentence. 
                Default: The default is 256.
            sentence_pooling_method : str, optional
                Corresponds to the method of pooling to create a unique vector for each text
                The default is "average".
            word_pooling_method : str, optional
                Corresponds to the method of word pooling method in the case multiple layers have been stated. In other words it is the way to compute a single word vector from multiple layers.
                The default is "average".
            layers : int or list, optional
                Corresponds to the BERT layers to use to create the embeddings of words.
                The default is 11.
            batch_size : int, optional
                Size of batch when executing the forward pass into the BERT model, should get lower as your computational power gets lower. 
                The default is 50.
            path_to_save : str or None, optional
                Is the path to save the vector of texts if you want to avoid doing this computation again as it may be long. 
                if None, nothing will be saved. 
                The default is None.

            Returns
            -------
            texts_vectors : list
                A list of tensors, each tensor corresponding to an input text.

            """
self
data
MAX_LEN
sentence_pooling_method
word_pooling_method
layers
batch_size
path_to_save