def _encode(self, tokens: List[str], mean: bool) -> Union[List[np.ndarray], np.ndarray]:
if self.vectorizer:<EOL><INDENT>detokenized_sample = self.tokenizer([tokens])[<NUM_LIT:0>]  <EOL>vectorized_sample = self.vectorizer([detokenized_sample])  <EOL>weights = np.array([vectorized_sample[<NUM_LIT:0>, np.where(self.vocabulary == token)[<NUM_LIT:0>][<NUM_LIT:0>]]<EOL>if len(np.where(self.vocabulary == token)[<NUM_LIT:0>]) else <NUM_LIT:0.><EOL>for token in tokens])<EOL><DEDENT>else:<EOL><INDENT>weights = np.array([self.get_weight(max(self.counter_vocab.get(token, <NUM_LIT:0>), self.idf_base_count))<EOL>for token in tokens])<EOL><DEDENT>if sum(weights) == <NUM_LIT:0>:<EOL><INDENT>weights = np.ones(len(tokens))<EOL><DEDENT>embedded_tokens = np.array(self.embedder([tokens]))[<NUM_LIT:0>, :, :]<EOL>if mean is None:<EOL><INDENT>mean = self.mean<EOL><DEDENT>if mean:<EOL><INDENT>embedded_tokens = np.average(embedded_tokens, weights=weights, axis=<NUM_LIT:0>)<EOL><DEDENT>else:<EOL><INDENT>embedded_tokens = np.array([weights[i] * embedded_tokens[i] for i in range(len(tokens))])<EOL><DEDENT>return embedded_tokens<EOL>
Embed one text sample

Args:
    tokens: tokenized text sample
    mean: whether to return mean token embedding (does not depend on self.mean)

Returns:
    list of embedded tokens or array of mean values