class_docstr: This DatasetReader is designed to read in the English OntoNotes v5.0 data
  in the format used by the CoNLL 2011/2012 shared tasks. In order to use this
  Reader, you must follow the instructions provided `here (v12 release):
  <http://cemantix.org/data/ontonotes.html>`_, which will allow you to download
  the CoNLL style annotations for the  OntoNotes v5.0 release -- LDC2013T19.tgz
  obtained from LDC.
  
  Once you have run the scripts on the extracted data, you will have a folder
  structured as follows:
  
  conll-formatted-ontonotes-5.0/
   ── data
     ├── development
         └── data
             └── english
                 └── annotations
                     ├── bc
                     ├── bn
                     ├── mz
                     ├── nw
                     ├── pt
                     ├── tc
                     └── wb
     ├── test
         └── data
             └── english
                 └── annotations
                     ├── bc
                     ├── bn
                     ├── mz
                     ├── nw
                     ├── pt
                     ├── tc
                     └── wb
     └── train
         └── data
             └── english
                 └── annotations
                     ├── bc
                     ├── bn
                     ├── mz
                     ├── nw
                     ├── pt
                     ├── tc
                     └── wb
  
  The file path provided to this class can then be any of the train, test or development
  directories(or the top level data directory, if you are not utilizing the splits).
  
  The data has the following format, ordered by column.
  
  1 Document ID : ``str``
      This is a variation on the document filename
  2 Part number : ``int``
      Some files are divided into multiple parts numbered as 000, 001, 002, ... etc.
  3 Word number : ``int``
      This is the word index of the word in that sentence.
  4 Word : ``str``
      This is the token as segmented/tokenized in the Treebank. Initially the ``*_skel`` file
      contain the placeholder [WORD] which gets replaced by the actual token from the
      Treebank which is part of the OntoNotes release.
  5 POS Tag : ``str``
      This is the Penn Treebank style part of speech. When parse information is missing,
      all part of speeches except the one for which there is some sense or proposition
      annotation are marked with a XX tag. The verb is marked with just a VERB tag.
  6 Parse bit: ``str``
      This is the bracketed structure broken before the first open parenthesis in the parse,
      and the word/part-of-speech leaf replaced with a ``*``. When the parse information is
      missing, the first word of a sentence is tagged as ``(TOP*`` and the last word is tagged
      as ``*)`` and all intermediate words are tagged with a ``*``.
  7 Predicate lemma: ``str``
      The predicate lemma is mentioned for the rows for which we have semantic role
      information or word sense information. All other rows are marked with a "-".
  8 Predicate Frameset ID: ``int``
      The PropBank frameset ID of the predicate in Column 7.
  9 Word sense: ``float``
      This is the word sense of the word in Column 3.
  10 Speaker/Author: ``str``
      This is the speaker or author name where available. Mostly in Broadcast Conversation
      and Web Log data. When not available the rows are marked with an "-".
  11 Named Entities: ``str``
      These columns identifies the spans representing various named entities. For documents
      which do not have named entity annotation, each line is represented with an ``*``.
  12+ Predicate Arguments: ``str``
      There is one column each of predicate argument structure information for the predicate
      mentioned in Column 7. If there are no predicates tagged in a sentence this is a
      single column with all rows marked with an ``*``.
  -1 Co-reference: ``str``
      Co-reference chain information encoded in a parenthesis structure. For documents that do
       not have co-reference annotations, each line is represented with a "-".
class_docstr_tok: ["This","DatasetReader","is","designed","to","read","in","the","English","OntoNotes","v5.0","data","in","the","format","used","by","the","CoNLL","2011\/2012","shared","tasks",".","In","order","to","use","this","Reader",",","you","must","follow","the","instructions","provided","`","here","(","v12","release",")",":","<","http",":","\/\/cemantix.org\/data\/ontonotes.html",">","`","_",",","which","will","allow","you","to","download","the","CoNLL","style","annotations","for","the","OntoNotes","v5.0","release","--","LDC2013T19.tgz","obtained","from","LDC",".","Once","you","have","run","the","scripts","on","the","extracted","data",",","you","will","have","a","folder","structured","as","follows",":","conll-formatted-ontonotes-5.0\/","\u2500\u2500","data","\u251c\u2500\u2500","development","\u2514\u2500\u2500","data","\u2514\u2500\u2500","english","\u2514\u2500\u2500","annotations","\u251c\u2500\u2500","bc","\u251c\u2500\u2500","bn","\u251c\u2500\u2500","mz","\u251c\u2500\u2500","nw","\u251c\u2500\u2500","pt","\u251c\u2500\u2500","tc","\u2514\u2500\u2500","wb","\u251c\u2500\u2500","test","\u2514\u2500\u2500","data","\u2514\u2500\u2500","english","\u2514\u2500\u2500","annotations","\u251c\u2500\u2500","bc","\u251c\u2500\u2500","bn","\u251c\u2500\u2500","mz","\u251c\u2500\u2500","nw","\u251c\u2500\u2500","pt","\u251c\u2500\u2500","tc","\u2514\u2500\u2500","wb","\u2514\u2500\u2500","train","\u2514\u2500\u2500","data","\u2514\u2500\u2500","english","\u2514\u2500\u2500","annotations","\u251c\u2500\u2500","bc","\u251c\u2500\u2500","bn","\u251c\u2500\u2500","mz","\u251c\u2500\u2500","nw","\u251c\u2500\u2500","pt","\u251c\u2500\u2500","tc","\u2514\u2500\u2500","wb","The","file","path","provided","to","this","class","can","then","be","any","of","the","train",",","test","or","development","directories","(","or","the","top","level","data","directory",",","if","you","are","not","utilizing","the","splits",")",".","The","data","has","the","following","format",",","ordered","by","column",".","1","Document","ID",":","``","str","``","This","is","a","variation","on","the","document","filename","2","Part","number",":","``","int","``","Some","files","are","divided","into","multiple","parts","numbered","as","000",",","001",",","002",",","...","etc",".","3","Word","number",":","``","int","``","This","is","the","word","index","of","the","word","in","that","sentence",".","4","Word",":","``","str","``","This","is","the","token","as","segmented\/tokenized","in","the","Treebank",".","Initially","the","``","*","_skel","``","file","contain","the","placeholder","[","WORD","]","which","gets","replaced","by","the","actual","token","from","the","Treebank","which","is","part","of","the","OntoNotes","release",".","5","POS","Tag",":","``","str","``","This","is","the","Penn","Treebank","style","part","of","speech",".","When","parse","information","is","missing",",","all","part","of","speeches","except","the","one","for","which","there","is","some","sense","or","proposition","annotation","are","marked","with","a","XX","tag",".","The","verb","is","marked","with","just","a","VERB","tag",".","6","Parse","bit",":","``","str","``","This","is","the","bracketed","structure","broken","before","the","first","open","parenthesis","in","the","parse",",","and","the","word\/part-of-speech","leaf","replaced","with","a","``","*","``",".","When","the","parse","information","is","missing",",","the","first","word","of","a","sentence","is","tagged","as","``","(","TOP","*","``","and","the","last","word","is","tagged","as","``","*",")","``","and","all","intermediate","words","are","tagged","with","a","``","*","``",".","7","Predicate","lemma",":","``","str","``","The","predicate","lemma","is","mentioned","for","the","rows","for","which","we","have","semantic","role","information","or","word","sense","information",".","All","other","rows","are","marked","with","a","``","-","''",".","8","Predicate","Frameset","ID",":","``","int","``","The","PropBank","frameset","ID","of","the","predicate","in","Column","7",".","9","Word","sense",":","``","float","``","This","is","the","word","sense","of","the","word","in","Column","3",".","10","Speaker\/Author",":","``","str","``","This","is","the","speaker","or","author","name","where","available",".","Mostly","in","Broadcast","Conversation","and","Web","Log","data",".","When","not","available","the","rows","are","marked","with","an","``","-","''",".","11","Named","Entities",":","``","str","``","These","columns","identifies","the","spans","representing","various","named","entities",".","For","documents","which","do","not","have","named","entity","annotation",",","each","line","is","represented","with","an","``","*","``",".","12+","Predicate","Arguments",":","``","str","``","There","is","one","column","each","of","predicate","argument","structure","information","for","the","predicate","mentioned","in","Column","7",".","If","there","are","no","predicates","tagged","in","a","sentence","this","is","a","single","column","with","all","rows","marked","with","an","``","*","``",".","-1","Co-reference",":","``","str","``","Co-reference","chain","information","encoded","in","a","parenthesis","structure",".","For","documents","that","do","not","have","co-reference","annotations",",","each","line","is","represented","with","a","``","-","''","."]
code: def _conll_rows_to_sentence(self, conll_rows):
      document_id = None
      sentence_id = None
      # The words in the sentence.
      sentence = []
      # The pos tags of the words in the sentence.
      pos_tags = []
      # the pieces of the parse tree.
      parse_pieces = []
      # The lemmatised form of the words in the sentence which
      # have SRL or word sense information.
      predicate_lemmas = []
      # The FrameNet ID of the predicate.
      predicate_framenet_ids = []
      # The sense of the word, if available.
      word_senses = []
      # The current speaker, if available.
      speakers = []
  
      verbal_predicates = []
      span_labels = []
      current_span_labels = []
  
      # Cluster id -> List of (start_index, end_index) spans.
      clusters = defaultdict(list)
      # Cluster id -> List of start_indices which are open for this id.
      coref_stacks = defaultdict(list)
  
      for index, row in enumerate(conll_rows):
          conll_components = row.split()
  
          document_id = conll_components[0]
          sentence_id = int(conll_components[1])
          word = conll_components[3]
          pos_tag = conll_components[4]
          parse_piece = conll_components[5]
  
          # Replace brackets in text and pos tags
          # with a different token for parse trees.
          if pos_tag != "XX" and word != "XX":
              if word == "(":
                  parse_word = "-LRB-"
              elif word == ")":
                  parse_word = "-RRB-"
              else:
                  parse_word = word
              if pos_tag == "(":
                  pos_tag = "-LRB-"
              if pos_tag == ")":
                  pos_tag = "-RRB-"
              (left_brackets, right_hand_side) = parse_piece.split("*")
              # only keep ')' if there are nested brackets with nothing in them.
              right_brackets = right_hand_side.count(")") * ")"
              parse_piece = "{left_brackets} ({pos_tag} {parse_word}) {right_brackets}"
          else:
              # There are some bad annotations in the CONLL data.
              # They contain no information, so to make this explicit,
              # we just set the parse piece to be None which will result
              # in the overall parse tree being None.
              parse_piece = None
  
          lemmatised_word = conll_components[6]
          framenet_id = conll_components[7]
          word_sense = conll_components[8]
          speaker = conll_components[9]
  
          if not span_labels:
              # If this is the first word in the sentence, create
              # empty lists to collect the NER and SRL BIO labels.
              # We can't do this upfront, because we don't know how many
              # components we are collecting, as a sentence can have
              # variable numbers of SRL frames.
              span_labels = [[] for _ in conll_components[10:-1]]
              # Create variables representing the current label for each label
              # sequence we are collecting.
              current_span_labels = [
                  None for _ in conll_components[10:-1]
              ]
  
          self._process_span_annotations_for_word(
              conll_components[10:-1], span_labels, current_span_labels
          )
  
          # If any annotation marks this word as a verb predicate,
          # we need to record its index. This also has the side effect
          # of ordering the verbal predicates by their location in the
          # sentence, automatically aligning them with the annotations.
          word_is_verbal_predicate = any(
              ["(V" in x for x in conll_components[11:-1]]
          )
          if word_is_verbal_predicate:
              verbal_predicates.append(word)
  
          self._process_coref_span_annotations_for_word(
              conll_components[-1], index, clusters, coref_stacks
          )
  
          sentence.append(word)
          pos_tags.append(pos_tag)
          parse_pieces.append(parse_piece)
          predicate_lemmas.append(
              lemmatised_word if lemmatised_word != "-" else None
          )
          predicate_framenet_ids.append(
              framenet_id if framenet_id != "-" else None
          )
          word_senses.append(
              float(word_sense) if word_sense != "-" else None
          )
          speakers.append(speaker if speaker != "-" else None)
  
      named_entities = span_labels[0]
      srl_frames = [
          (predicate, labels)
          for predicate, labels in izip(
              verbal_predicates, span_labels[1:]
          )
      ]
  
      if all(parse_pieces):
          parse_tree = Tree.fromstring("".join(parse_pieces))
      else:
          parse_tree = None
      coref_span_tuples = set(
          [
              (cluster_id, span)
              for cluster_id, span_list in list(clusters.items())
              for span in span_list
          ]
      )
      return OntonotesSentence(
          document_id,
          sentence_id,
          sentence,
          pos_tags,
          parse_tree,
          predicate_lemmas,
          predicate_framenet_ids,
          word_senses,
          speakers,
          named_entities,
          srl_frames,
          coref_span_tuples,
      )
  
file_path: magnitude/pymagnitude/third_party/allennlp/data/dataset_readers/dataset_utils/ontonotes.py
imports: from __future__ import with_statement
  from __future__ import absolute_import
  from collections import defaultdict
  import codecs
  import os
  import logging
  from nltk import Tree
label_desc: Use image node_id 5 for calling the Ontonotes obj's underlying member method code with example usage: obj._conll_rows_to_sentence(conll_rows) and returns: OntonotesSentence