class_docstr: Each `Instance` in a `Scenario` looks like this:
  
      <input> -> <reference1>
                 <reference2>
                 <reference3> [correct]
                 <reference4>
  
  For language modeling, we don't use the references (even if they exist), just feed the input:
  
      <input>
docstr: This method is used for adapting instances for language modeling scenarios.
  For some tokenizers (e.g. AI21), decoding then encoding k tokens may result
  in > k tokens. This method trims the tokens and check with the tokenizer
  repeatedly until they fit in the context window.
  
  For models using the GPT-2 tokenizer, conditioning_tokens and pred_tokens
  are integers; for AI21 models, the tokens are TokenizationTokens.
docstr_tok: ["This","method","is","used","for","adapting","instances","for","language","modeling","scenarios",".","For","some","tokenizers","(","e.g",".","AI21",")",",","decoding","then","encoding","k","tokens","may","result","in",">","k","tokens",".","This","method","trims","the","tokens","and","check","with","the","tokenizer","repeatedly","until","they","fit","in","the","context","window",".","For","models","using","the","GPT-2","tokenizer",",","conditioning_tokens","and","pred_tokens","are","integers",";","for","AI21","models",",","the","tokens","are","TokenizationTokens","."]
code: def fits_tokens_within_context_window(
      self,
      conditioning_tokens: List[TokenizationToken],
      pred_tokens: List[TokenizationToken],
      max_req_len: int,
      text: Optional[str] = None,
  ) -> Tuple[str, List[TokenizationToken]]:
      """
      This method is used for adapting instances for language modeling scenarios.
      For some tokenizers (e.g. AI21), decoding then encoding k tokens may result
      in > k tokens. This method trims the tokens and check with the tokenizer
      repeatedly until they fit in the context window.
  
      For models using the GPT-2 tokenizer, conditioning_tokens and pred_tokens
      are integers; for AI21 models, the tokens are TokenizationTokens.
      """
      prompt: str = self.window_service.decode(
          conditioning_tokens + pred_tokens, text
      )
      prompt_length: int = len(
          self.window_service.encode(prompt).tokens
      )
  
      # If the prompt is too long, removes the overflowing tokens.
      # Since encoding might generate extra tokens, we need to repeat this until prompt_length <= max_req_len.
      # For AI21, for example, this happens especially frequently when a document contains different types of
      # whitespace characters because some whitespaces are tokenized to multiple tokens and the others
      # are tokenized to a single token. However, the AI21 tokenizer seems to normalize all types
      # of whitespaces to the same whitespace character.
      #
      # e.g. original text: ",  (", which is tokenized to:
      # [('▁', 0, 0), (',', 0, 1), ('▁▁', 1, 3), ('(', 3, 4)]
      # normalized text: ",  (", which is tokenized to:
      # [('▁', 0, 0), (',', 0, 1), ('▁', 1, 2), ('▁', 2, 3), ('(', 3, 4)]
      while prompt_length > max_req_len:
          # Trims the extra (prompt_length - max_req_len) tokens
          pred_tokens = pred_tokens[: -(prompt_length - max_req_len)]
          prompt = self.window_service.decode(
              conditioning_tokens + pred_tokens, text
          )
          prompt_length = len(self.window_service.encode(prompt).tokens)
  
          # When the input text contains languages the tokenizer cannot process, the input text
          # might be inflated so the truncation cannot work properly.
          # e.g.
          # With the OpenAI tokenizer:
          # >>> tokenizer.decode(tokenizer.encode("行星运转"))
          # '行星运转'
          # With the YaLM tokenizer:
          # >>> tokenizer.decode(tokenizer.tokenize("行星运转"))
          # '行<0xE6><0x98><0x9F><0xE8><0xBF><0x90><0xE8><0xBD><0xAC>'
          if len(pred_tokens) == 0:
              raise ValueError(
                  "Truncating pred_tokens to fit them in the context window, "
                  "got len(pred_tokens) == 0, which will lead to an infinite loop."
              )
  
      return prompt, pred_tokens
  
file_path: H2O/h2o_hf/helm/src/helm/benchmark/adaptation/adapters/language_modeling_adapter.py
imports: from typing import List, Tuple, Optional
  from helm.benchmark.adaptation.request_state import RequestState
  from helm.benchmark.adaptation.scenario_state import ScenarioState
  from helm.benchmark.scenarios.scenario import Instance, EVAL_SPLITS
  from helm.benchmark.window_services.window_service import EncodeResult
  from helm.common.general import flatten_list, parallel_map
  from helm.common.hierarchical_logger import hlog, htrack
  from helm.common.request import Request
  from helm.common.tokenization_request import TokenizationToken
  from .adapter import Adapter
label_desc: Use image node_id 4 for calling the LanguageModelingAdapter obj's underlying member method code with example usage: obj.fits_tokens_within_context_window(conditioning_tokens, pred_tokens, max_req_len, text) and returns: prompt,  pred_tokens