class_docstr: Reads constituency parses from the WSJ part of the Penn Tree Bank from the LDC.
  This ``DatasetReader`` is designed for use with a span labelling model, so
  it enumerates all possible spans in the sentence and returns them, along with gold
  labels for the relevant spans present in a gold tree, if provided.
  
  Parameters
  ----------
  token_indexers : ``Dict[str, TokenIndexer]``, optional (default=``{"tokens": SingleIdTokenIndexer()}``)
      We use this to define the input representation for the text.  See :class:`TokenIndexer`.
      Note that the `output` tags will always correspond to single token IDs based on how they
      are pre-tokenised in the data file.
  use_pos_tags : ``bool``, optional, (default = ``True``)
      Whether or not the instance should contain gold POS tags
      as a field.
  lazy : ``bool``, optional, (default = ``False``)
      Whether or not instances can be consumed lazily.
class_docstr_tok: ["Reads","constituency","parses","from","the","WSJ","part","of","the","Penn","Tree","Bank","from","the","LDC",".","This","``","DatasetReader","``","is","designed","for","use","with","a","span","labelling","model",",","so","it","enumerates","all","possible","spans","in","the","sentence","and","returns","them",",","along","with","gold","labels","for","the","relevant","spans","present","in","a","gold","tree",",","if","provided",".","Parameters","--","--","--","--","--","token_indexers",":","``","Dict","[","str",",","TokenIndexer","]","``",",","optional","(","default=","``","{","``","tokens","''",":","SingleIdTokenIndexer","(",")","}","``",")","We","use","this","to","define","the","input","representation","for","the","text",".","See",":","class",":","`","TokenIndexer","`",".","Note","that","the","`","output","`","tags","will","always","correspond","to","single","token","IDs","based","on","how","they","are","pre-tokenised","in","the","data","file",".","use_pos_tags",":","``","bool","``",",","optional",",","(","default","=","``","True","``",")","Whether","or","not","the","instance","should","contain","gold","POS","tags","as","a","field",".","lazy",":","``","bool","``",",","optional",",","(","default","=","``","False","``",")","Whether","or","not","instances","can","be","consumed","lazily","."]
docstr: We take `pre-tokenized` input here, because we don't have a tokenizer in this class.
  
  Parameters
  ----------
  tokens : ``List[str]``, required.
      The tokens in a given sentence.
  pos_tags ``List[str]``, optional, (default = None).
      The POS tags for the words in the sentence.
  gold_tree : ``Tree``, optional (default = None).
      The gold parse tree to create span labels from.
  
  Returns
  -------
  An ``Instance`` containing the following fields:
      tokens : ``TextField``
          The tokens in the sentence.
      pos_tags : ``SequenceLabelField``
          The POS tags of the words in the sentence.
          Only returned if ``use_pos_tags`` is ``True``
      spans : ``ListField[SpanField]``
          A ListField containing all possible subspans of the
          sentence.
      span_labels : ``SequenceLabelField``, optional.
          The constiutency tags for each of the possible spans, with
          respect to a gold parse tree. If a span is not contained
          within the tree, a span will have a ``NO-LABEL`` label.
      gold_tree : ``MetadataField(Tree)``
          The gold NLTK parse tree for use in evaluation.
docstr_tok: ["We","take","`","pre-tokenized","`","input","here",",","because","we","do","n't","have","a","tokenizer","in","this","class",".","Parameters","--","--","--","--","--","tokens",":","``","List","[","str","]","``",",","required",".","The","tokens","in","a","given","sentence",".","pos_tags","``","List","[","str","]","``",",","optional",",","(","default","=","None",")",".","The","POS","tags","for","the","words","in","the","sentence",".","gold_tree",":","``","Tree","``",",","optional","(","default","=","None",")",".","The","gold","parse","tree","to","create","span","labels","from",".","Returns","--","--","--","-","An","``","Instance","``","containing","the","following","fields",":","tokens",":","``","TextField","``","The","tokens","in","the","sentence",".","pos_tags",":","``","SequenceLabelField","``","The","POS","tags","of","the","words","in","the","sentence",".","Only","returned","if","``","use_pos_tags","``","is","``","True","``","spans",":","``","ListField","[","SpanField","]","``","A","ListField","containing","all","possible","subspans","of","the","sentence",".","span_labels",":","``","SequenceLabelField","``",",","optional",".","The","constiutency","tags","for","each","of","the","possible","spans",",","with","respect","to","a","gold","parse","tree",".","If","a","span","is","not","contained","within","the","tree",",","a","span","will","have","a","``","NO-LABEL","``","label",".","gold_tree",":","``","MetadataField","(","Tree",")","``","The","gold","NLTK","parse","tree","for","use","in","evaluation","."]
code: def text_to_instance(
      self, tokens, pos_tags=None, gold_tree=None  # type: ignore
  ):
      """
      We take `pre-tokenized` input here, because we don't have a tokenizer in this class.
  
      Parameters
      ----------
      tokens : ``List[str]``, required.
          The tokens in a given sentence.
      pos_tags ``List[str]``, optional, (default = None).
          The POS tags for the words in the sentence.
      gold_tree : ``Tree``, optional (default = None).
          The gold parse tree to create span labels from.
  
      Returns
      -------
      An ``Instance`` containing the following fields:
          tokens : ``TextField``
              The tokens in the sentence.
          pos_tags : ``SequenceLabelField``
              The POS tags of the words in the sentence.
              Only returned if ``use_pos_tags`` is ``True``
          spans : ``ListField[SpanField]``
              A ListField containing all possible subspans of the
              sentence.
          span_labels : ``SequenceLabelField``, optional.
              The constiutency tags for each of the possible spans, with
              respect to a gold parse tree. If a span is not contained
              within the tree, a span will have a ``NO-LABEL`` label.
          gold_tree : ``MetadataField(Tree)``
              The gold NLTK parse tree for use in evaluation.
      """
      # pylint: disable=arguments-differ
      text_field = TextField(
          [Token(x) for x in tokens],
          token_indexers=self._token_indexers,
      )
      fields = {"tokens": text_field}
  
      if self._use_pos_tags and pos_tags is not None:
          pos_tag_field = SequenceLabelField(
              pos_tags, text_field, label_namespace="pos"
          )
          fields["pos_tags"] = pos_tag_field
      elif self._use_pos_tags:
          raise ConfigurationError(
              "use_pos_tags was set to True but no gold pos"
              " tags were passed to the dataset reader."
          )
      spans = []
      gold_labels = []
  
      if gold_tree is not None:
          gold_spans = {}
          self._get_gold_spans(gold_tree, 0, gold_spans)
  
      else:
          gold_spans = None
      for start, end in enumerate_spans(tokens):
          spans.append(SpanField(start, end, text_field))
  
          if gold_spans is not None:
              if (start, end) in list(gold_spans.keys()):
                  gold_labels.append(gold_spans[(start, end)])
              else:
                  gold_labels.append("NO-LABEL")
  
      metadata = {"tokens": tokens}
      if gold_tree:
          metadata["gold_tree"] = gold_tree
      if self._use_pos_tags:
          metadata["pos_tags"] = pos_tags
  
      fields["metadata"] = MetadataField(metadata)
  
      span_list_field = ListField(spans)
      fields["spans"] = span_list_field
      if gold_tree is not None:
          fields["span_labels"] = SequenceLabelField(
              gold_labels, span_list_field
          )
      return Instance(fields)
  
file_path: magnitude/pymagnitude/third_party/allennlp/data/dataset_readers/penn_tree_bank.py
imports: from __future__ import absolute_import
  import logging
  import os
  from nltk.corpus.reader.bracket_parse import BracketParseCorpusReader
  from nltk.tree import Tree
  from allennlp.common.file_utils import cached_path
  from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  from allennlp.data.fields import TextField, SpanField, SequenceLabelField, ListField, MetadataField, Field
  from allennlp.data.instance import Instance
  from allennlp.data.token_indexers import TokenIndexer, SingleIdTokenIndexer
  from allennlp.data.tokenizers import Token
  from allennlp.data.dataset_readers.dataset_utils.span_utils import enumerate_spans
  from allennlp.common.checks import ConfigurationError
label_desc: Use image node_id 3 for calling the PennTreeBankConstituencySpanDatasetReader obj's underlying member method code with example usage: obj.text_to_instance(tokens, pos_tags, gold_tree) and returns: Instance