docstr: Converts each text example in a column into encoded state. This can be either a vector embedding of the [CLS] token (represents the full text input) OR the logits prediction of the output.
  
  The transformer model is of form:
  transformer base + pre-classifier linear layer + classifier layer
  
  The embedding returned is of the [CLS] token after the pre-classifier layer; from internal testing, we found the latent space most highly separated across classes. 
  
  If the encoder represents the logits in classification, returns a soft-maxed output of the class vector.
  
  :param column_data: List of text data as strings
  :returns: Embedded vector N_rows x Nembed_dim OR logits vector N_rows x N_classes depending on if `embed_mode` is True or not.
docstr_tok: ["Converts","each","text","example","in","a","column","into","encoded","state",".","This","can","be","either","a","vector","embedding","of","the","[","CLS","]","token","(","represents","the","full","text","input",")","OR","the","logits","prediction","of","the","output",".","The","transformer","model","is","of","form",":","transformer","base","+","pre-classifier","linear","layer","+","classifier","layer","The","embedding","returned","is","of","the","[","CLS","]","token","after","the","pre-classifier","layer",";","from","internal","testing",",","we","found","the","latent","space","most","highly","separated","across","classes",".","If","the","encoder","represents","the","logits","in","classification",",","returns","a","soft-maxed","output","of","the","class","vector",".",":","param","column_data",":","List","of","text","data","as","strings",":","returns",":","Embedded","vector","N_rows","x","Nembed_dim","OR","logits","vector","N_rows","x","N_classes","depending","on","if","`","embed_mode","`","is","True","or","not","."]
code: def encode(self, column_data: Iterable[str]) -> torch.Tensor:
      """
      Converts each text example in a column into encoded state. This can be either a vector embedding of the [CLS] token (represents the full text input) OR the logits prediction of the output.
  
      The transformer model is of form:
      transformer base + pre-classifier linear layer + classifier layer
  
      The embedding returned is of the [CLS] token after the pre-classifier layer; from internal testing, we found the latent space most highly separated across classes.
  
      If the encoder represents the logits in classification, returns a soft-maxed output of the class vector.
  
      :param column_data: List of text data as strings
      :returns: Embedded vector N_rows x Nembed_dim OR logits vector N_rows x N_classes depending on if `embed_mode` is True or not.
      """  # noqa
      if self.is_prepared is False:
          raise Exception("You need to first prepare the encoder.")
  
      # Set model to testing/eval mode.
      self._model.eval()
  
      encoded_representation = []
  
      with torch.no_grad():
          # Set the weights; this is GPT-2
          for text in column_data:
              # Omit NaNs
              if is_none(text):
                  text = ""
  
              # Tokenize the text with the built-in tokenizer.
              inp = self._tokenizer.encode(
                  text, truncation=True, return_tensors="pt"
              ).to(self.device)
  
              if self.embed_mode:  # Embedding mode ON; return [CLS]
                  output = self._model.base_model(
                      inp
                  ).last_hidden_state[:, 0]
  
                  # If the model has a pre-classifier layer, use this embedding.
                  if hasattr(self._model, "pre_classifier"):
                      output = self._model.pre_classifier(output)
  
              else:  # Embedding mode off; return classes
                  output = self._model(inp).logits
  
              encoded_representation.append(output.detach())
  
      return torch.stack(encoded_representation).squeeze(1).to("cpu")
  
imports: import os
  import time
  from typing import Iterable
  from collections import deque
  import numpy
  import torch
  from torch.utils.data import DataLoader
  import pandas
  from type_infer.dtype import dtype
  from transformers import DistilBertModel, DistilBertForSequenceClassification, DistilBertTokenizerFast, AdamW, get_linear_schedule_with_warmup
  from sklearn.model_selection import train_test_split
  from lightwood.encoder.text.helpers.pretrained_helpers import TextEmbed
  from lightwood.helpers.device import get_device_from_name
  from lightwood.encoder.base import BaseEncoder
  from lightwood.helpers.log import log
  from lightwood.helpers.torch import LightwoodAutocast
  from lightwood.helpers.general import is_none
label_desc: Use image node_id 6 for calling the PretrainedLangEncoder obj's underlying member method code with example usage: obj.encode(column_data) and returns: torch