class_docstr: Reads constituency parses from the WSJ part of the Penn Tree Bank from the LDC.
  This ``DatasetReader`` is designed for use with a span labelling model, so
  it enumerates all possible spans in the sentence and returns them, along with gold
  labels for the relevant spans present in a gold tree, if provided.
  
  Parameters
  ----------
  token_indexers : ``Dict[str, TokenIndexer]``, optional (default=``{"tokens": SingleIdTokenIndexer()}``)
      We use this to define the input representation for the text.  See :class:`TokenIndexer`.
      Note that the `output` tags will always correspond to single token IDs based on how they
      are pre-tokenised in the data file.
  use_pos_tags : ``bool``, optional, (default = ``True``)
      Whether or not the instance should contain gold POS tags
      as a field.
  lazy : ``bool``, optional, (default = ``False``)
      Whether or not instances can be consumed lazily.
class_docstr_tok: ["Reads","constituency","parses","from","the","WSJ","part","of","the","Penn","Tree","Bank","from","the","LDC",".","This","``","DatasetReader","``","is","designed","for","use","with","a","span","labelling","model",",","so","it","enumerates","all","possible","spans","in","the","sentence","and","returns","them",",","along","with","gold","labels","for","the","relevant","spans","present","in","a","gold","tree",",","if","provided",".","Parameters","--","--","--","--","--","token_indexers",":","``","Dict","[","str",",","TokenIndexer","]","``",",","optional","(","default=","``","{","``","tokens","''",":","SingleIdTokenIndexer","(",")","}","``",")","We","use","this","to","define","the","input","representation","for","the","text",".","See",":","class",":","`","TokenIndexer","`",".","Note","that","the","`","output","`","tags","will","always","correspond","to","single","token","IDs","based","on","how","they","are","pre-tokenised","in","the","data","file",".","use_pos_tags",":","``","bool","``",",","optional",",","(","default","=","``","True","``",")","Whether","or","not","the","instance","should","contain","gold","POS","tags","as","a","field",".","lazy",":","``","bool","``",",","optional",",","(","default","=","``","False","``",")","Whether","or","not","instances","can","be","consumed","lazily","."]
docstr: Recursively construct the gold spans from an nltk ``Tree``.
  Labels are the constituents, and in the case of nested constituents
  with the same spans, labels are concatenated in parent-child order.
  For example, ``(S (NP (D the) (N man)))`` would have an ``S-NP`` label
  for the outer span, as it has both ``S`` and ``NP`` label.
  Spans are inclusive.
  
  TODO(Mark): If we encounter a gold nested labelling at test time
  which we haven't encountered, we won't be able to run the model
  at all.
  
  Parameters
  ----------
  tree : ``Tree``, required.
      An NLTK parse tree to extract spans from.
  index : ``int``, required.
      The index of the current span in the sentence being considered.
  typed_spans : ``Dict[Tuple[int, int], str]``, required.
      A dictionary mapping spans to span labels.
  
  Returns
  -------
  typed_spans : ``Dict[Tuple[int, int], str]``.
      A dictionary mapping all subtree spans in the parse tree
      to their constituency labels. POS tags are ignored.
docstr_tok: ["Recursively","construct","the","gold","spans","from","an","nltk","``","Tree","``",".","Labels","are","the","constituents",",","and","in","the","case","of","nested","constituents","with","the","same","spans",",","labels","are","concatenated","in","parent-child","order",".","For","example",",","``","(","S","(","NP","(","D","the",")","(","N","man",")",")",")","``","would","have","an","``","S-NP","``","label","for","the","outer","span",",","as","it","has","both","``","S","``","and","``","NP","``","label",".","Spans","are","inclusive",".","TODO","(","Mark",")",":","If","we","encounter","a","gold","nested","labelling","at","test","time","which","we","have","n't","encountered",",","we","wo","n't","be","able","to","run","the","model","at","all",".","Parameters","--","--","--","--","--","tree",":","``","Tree","``",",","required",".","An","NLTK","parse","tree","to","extract","spans","from",".","index",":","``","int","``",",","required",".","The","index","of","the","current","span","in","the","sentence","being","considered",".","typed_spans",":","``","Dict","[","Tuple","[","int",",","int","]",",","str","]","``",",","required",".","A","dictionary","mapping","spans","to","span","labels",".","Returns","--","--","--","-","typed_spans",":","``","Dict","[","Tuple","[","int",",","int","]",",","str","]","``",".","A","dictionary","mapping","all","subtree","spans","in","the","parse","tree","to","their","constituency","labels",".","POS","tags","are","ignored","."]
code: def _get_gold_spans(
      self, tree, index, typed_spans  # pylint: disable=arguments-differ
  ):
      """
      Recursively construct the gold spans from an nltk ``Tree``.
      Labels are the constituents, and in the case of nested constituents
      with the same spans, labels are concatenated in parent-child order.
      For example, ``(S (NP (D the) (N man)))`` would have an ``S-NP`` label
      for the outer span, as it has both ``S`` and ``NP`` label.
      Spans are inclusive.
  
      TODO(Mark): If we encounter a gold nested labelling at test time
      which we haven't encountered, we won't be able to run the model
      at all.
  
      Parameters
      ----------
      tree : ``Tree``, required.
          An NLTK parse tree to extract spans from.
      index : ``int``, required.
          The index of the current span in the sentence being considered.
      typed_spans : ``Dict[Tuple[int, int], str]``, required.
          A dictionary mapping spans to span labels.
  
      Returns
      -------
      typed_spans : ``Dict[Tuple[int, int], str]``.
          A dictionary mapping all subtree spans in the parse tree
          to their constituency labels. POS tags are ignored.
      """
      # NLTK leaves are strings.
      if isinstance(tree[0], unicode):
          # The "length" of a tree is defined by
          # NLTK as the number of children.
          # We don't actually want the spans for leaves, because
          # their labels are POS tags. Instead, we just add the length
          # of the word to the end index as we iterate through.
          end = index + len(tree)
      else:
          # otherwise, the tree has children.
          child_start = index
          for child in tree:
              # typed_spans is being updated inplace.
              end = self._get_gold_spans(
                  child, child_start, typed_spans
              )
              child_start = end
          # Set the end index of the current span to
          # the last appended index - 1, as the span is inclusive.
          span = (index, end - 1)
          current_span_label = typed_spans.get(span)
          if current_span_label is None:
              # This span doesn't have nested labels, just
              # use the current node's label.
              typed_spans[span] = tree.label()
          else:
              # This span has already been added, so prepend
              # this label (as we are traversing the tree from
              # the bottom up).
              typed_spans[span] = (
                  tree.label() + "-" + current_span_label
              )
  
      return end
  
file_path: magnitude/pymagnitude/third_party/allennlp/data/dataset_readers/penn_tree_bank.py
imports: from __future__ import absolute_import
  import logging
  import os
  from nltk.corpus.reader.bracket_parse import BracketParseCorpusReader
  from nltk.tree import Tree
  from allennlp.common.file_utils import cached_path
  from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  from allennlp.data.fields import TextField, SpanField, SequenceLabelField, ListField, MetadataField, Field
  from allennlp.data.instance import Instance
  from allennlp.data.token_indexers import TokenIndexer, SingleIdTokenIndexer
  from allennlp.data.tokenizers import Token
  from allennlp.data.dataset_readers.dataset_utils.span_utils import enumerate_spans
  from allennlp.common.checks import ConfigurationError
label_desc: Use image node_id 5 for calling the PennTreeBankConstituencySpanDatasetReader obj's underlying member method code with example usage: obj._get_gold_spans(tree, index, typed_spans) and returns: end