class_docstr: This class uses the external EVALB software for computing a broad range of metrics
  on parse trees. Here, we use it to compute the Precision, Recall and F1 metrics.
  You can download the source for EVALB from here: <http://nlp.cs.nyu.edu/evalb/>.
  
  Note that this software is 20 years old. In order to compile it on modern hardware,
  you may need to remove an ``include <malloc.h>`` statement in ``evalb.c`` before it
  will compile.
  
  AllenNLP contains the EVALB software, but you will need to compile it yourself
  before using it because the binary it generates is system depenedent. To build it,
  run ``make`` inside the ``allennlp/tools/EVALB`` directory.
  
  Note that this metric reads and writes from disk quite a bit. You probably don't
  want to include it in your training loop; instead, you should calculate this on
  a validation set only.
  
  Parameters
  ----------
  evalb_directory_path : ``str``, required.
      The directory containing the EVALB executable.
  evalb_param_filename: ``str``, optional (default = "COLLINS.prm")
      The relative name of the EVALB configuration file used when scoring the trees.
      By default, this uses the COLLINS.prm configuration file which comes with EVALB.
      This configuration ignores POS tags and some punctuation labels.
class_docstr_tok: ["This","class","uses","the","external","EVALB","software","for","computing","a","broad","range","of","metrics","on","parse","trees",".","Here",",","we","use","it","to","compute","the","Precision",",","Recall","and","F1","metrics",".","You","can","download","the","source","for","EVALB","from","here",":","<","http",":","\/\/nlp.cs.nyu.edu\/evalb\/",">",".","Note","that","this","software","is","20","years","old",".","In","order","to","compile","it","on","modern","hardware",",","you","may","need","to","remove","an","``","include","<","malloc.h",">","``","statement","in","``","evalb.c","``","before","it","will","compile",".","AllenNLP","contains","the","EVALB","software",",","but","you","will","need","to","compile","it","yourself","before","using","it","because","the","binary","it","generates","is","system","depenedent",".","To","build","it",",","run","``","make","``","inside","the","``","allennlp\/tools\/EVALB","``","directory",".","Note","that","this","metric","reads","and","writes","from","disk","quite","a","bit",".","You","probably","don't","want","to","include","it","in","your","training","loop",";","instead",",","you","should","calculate","this","on","a","validation","set","only",".","Parameters","--","--","--","--","--","evalb_directory_path",":","``","str","``",",","required",".","The","directory","containing","the","EVALB","executable",".","evalb_param_filename",":","``","str","``",",","optional","(","default","=","``","COLLINS.prm","''",")","The","relative","name","of","the","EVALB","configuration","file","used","when","scoring","the","trees",".","By","default",",","this","uses","the","COLLINS.prm","configuration","file","which","comes","with","EVALB",".","This","configuration","ignores","POS","tags","and","some","punctuation","labels","."]
docstr: Parameters
  ----------
  predicted_trees : ``List[Tree]``
      A list of predicted NLTK Trees to compute score for.
  gold_trees : ``List[Tree]``
      A list of gold NLTK Trees to use as a reference.
docstr_tok: ["Parameters","--","--","--","--","--","predicted_trees",":","``","List","[","Tree","]","``","A","list","of","predicted","NLTK","Trees","to","compute","score","for",".","gold_trees",":","``","List","[","Tree","]","``","A","list","of","gold","NLTK","Trees","to","use","as","a","reference","."]
code: def __call__(self, predicted_trees, gold_trees):  # type: ignore
      """
      Parameters
      ----------
      predicted_trees : ``List[Tree]``
          A list of predicted NLTK Trees to compute score for.
      gold_trees : ``List[Tree]``
          A list of gold NLTK Trees to use as a reference.
      """
      if not os.path.exists(self._evalb_program_path):
          compile_command = (
              "python -c 'from allennlp.training.metrics import EvalbBracketingScorer; "
              "EvalbBracketingScorer.compile_evalb()'"
          )
          raise ConfigurationError(
              "You must compile the EVALB scorer before using it."
              " Run 'make' in the '{}' directory or run: {}".format(
                  self._evalb_program_path, compile_command
              )
          )
      tempdir = tempfile.mkdtemp()
      gold_path = os.path.join(tempdir, "gold.txt")
      predicted_path = os.path.join(tempdir, "predicted.txt")
      output_path = os.path.join(tempdir, "output.txt")
      with open(gold_path, "w") as gold_file:
          for tree in gold_trees:
              gold_file.write("{tree.pformat(margin=1000000)}\n")
  
      with open(predicted_path, "w") as predicted_file:
          for tree in predicted_trees:
              predicted_file.write("{tree.pformat(margin=1000000)}\n")
  
      command = (
          "{self._evalb_program_path} -p {self._evalb_param_path} "
          "{gold_path} {predicted_path} > {output_path}"
      )
      subprocess.run(command, shell=True, check=True)
  
      with open(output_path) as infile:
          for line in infile:
              stripped = line.strip().split()
              if len(stripped) == 12 and stripped != self._header_line:
                  # This line contains results for a single tree.
                  numeric_line = [float(x) for x in stripped]
                  self._correct_predicted_brackets += numeric_line[5]
                  self._gold_brackets += numeric_line[6]
                  self._predicted_brackets += numeric_line[7]
  
      shutil.rmtree(tempdir)
  
file_path: magnitude/pymagnitude/third_party/allennlp/training/metrics/evalb_bracketing_scorer.py
imports: from __future__ import division
  from __future__ import with_statement
  from __future__ import absolute_import
  import os
  import tempfile
  import subprocess
  import shutil
  from nltk import Tree
  from allennlp.common.checks import ConfigurationError
  from allennlp.training.metrics.metric import Metric
  from io import open
label_desc: Use image node_id 2 for calling the EvalbBracketingScorer obj's underlying member method code with example usage: obj.__call__(predicted_trees, gold_trees) without return types