class_docstr: Reads a text file and converts it into a ``Dataset`` suitable for training a language model.
  
  Note that there's one issue that needs to be fixed before this is actually usable for language
  modeling - the way start and end tokens for sentences are handled is not correct; we need to
  add a sentence splitter before this will be done right.
  
  Parameters
  ----------
  tokens_per_instance : ``int``, optional (default=``None``)
      If this is ``None``, we will have each training instance be a single sentence.  If this is
      not ``None``, we will instead take all sentences, including their start and stop tokens,
      line them up, and split the tokens into groups of this number, for more efficient training
      of the language model.
  tokenizer : ``Tokenizer``, optional (default=``WordTokenizer()``)
      We use this ``Tokenizer`` for the text.  See :class:`Tokenizer`.
  token_indexers : ``Dict[str, TokenIndexer]``, optional (default=``{"tokens": SingleIdTokenIndexer()}``)
      We use this to define the input representation for the text.  See :class:`TokenIndexer`.
      Note that the `output` representation will always be single token IDs - if you've specified
      a ``SingleIdTokenIndexer`` here, we use the first one you specify.  Otherwise, we create
      one with default parameters.
class_docstr_tok: ["Reads","a","text","file","and","converts","it","into","a","``","Dataset","``","suitable","for","training","a","language","model",".","Note","that","there","'s","one","issue","that","needs","to","be","fixed","before","this","is","actually","usable","for","language","modeling","-","the","way","start","and","end","tokens","for","sentences","are","handled","is","not","correct",";","we","need","to","add","a","sentence","splitter","before","this","will","be","done","right",".","Parameters","--","--","--","--","--","tokens_per_instance",":","``","int","``",",","optional","(","default=","``","None","``",")","If","this","is","``","None","``",",","we","will","have","each","training","instance","be","a","single","sentence",".","If","this","is","not","``","None","``",",","we","will","instead","take","all","sentences",",","including","their","start","and","stop","tokens",",","line","them","up",",","and","split","the","tokens","into","groups","of","this","number",",","for","more","efficient","training","of","the","language","model",".","tokenizer",":","``","Tokenizer","``",",","optional","(","default=","``","WordTokenizer","(",")","``",")","We","use","this","``","Tokenizer","``","for","the","text",".","See",":","class",":","`","Tokenizer","`",".","token_indexers",":","``","Dict","[","str",",","TokenIndexer","]","``",",","optional","(","default=","``","{","``","tokens","''",":","SingleIdTokenIndexer","(",")","}","``",")","We","use","this","to","define","the","input","representation","for","the","text",".","See",":","class",":","`","TokenIndexer","`",".","Note","that","the","`","output","`","representation","will","always","be","single","token","IDs","-","if","you","'ve","specified","a","``","SingleIdTokenIndexer","``","here",",","we","use","the","first","one","you","specify",".","Otherwise",",","we","create","one","with","default","parameters","."]
code: def text_to_instance(self, sentence):  # type: ignore
      # pylint: disable=arguments-differ
      tokenized_string = self._tokenizer.tokenize(sentence)
      input_field = TextField(
          tokenized_string[:-1], self._token_indexers
      )
      output_field = TextField(
          tokenized_string[1:], self._output_indexer
      )
      return Instance(
          {"input_tokens": input_field, "output_tokens": output_field}
      )
  
file_path: magnitude/pymagnitude/third_party/allennlp/data/dataset_readers/language_modeling.py
imports: from __future__ import with_statement
  from __future__ import absolute_import
  import logging
  from allennlp.common.file_utils import cached_path
  from allennlp.common.tqdm import Tqdm
  from allennlp.data.instance import Instance
  from allennlp.data.tokenizers.tokenizer import Tokenizer
  from allennlp.data.tokenizers import WordTokenizer
  from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  from allennlp.data.token_indexers.token_indexer import TokenIndexer
  from allennlp.data.fields import TextField
  from allennlp.data.token_indexers import SingleIdTokenIndexer
  from io import open
label_desc: Use image node_id 3 for calling the LanguageModelingReader obj's underlying member method code with example usage: obj.text_to_instance(sentence) and returns: Instance