class_docstr: Base class for various task test suites.
  
  This is a wrapper class around the CheckList toolkit introduced
  in the paper
  [Beyond Accuracy: Behavioral Testing of NLP models with CheckList (Ribeiro et al)]
  (https://api.semanticscholar.org/CorpusID:218551201).
  
  !!! Note
      To use the checklist integration you should install ``allennlp`` with the
      "checklist" extra (e.g. ``conda install allennlp-checklist``,
      ``pip install allennlp[checklist]`` or just install `checklist` after the fact).
  
  Task suites are intended to be used as a form of behavioral testing
  for NLP models to check for robustness across several general linguistic
  capabilities; eg. Vocabulary, SRL, Negation, etc.
  
  An example of the entire checklist process can be found at:
  [https://github.com/marcotcr/checklist/blob/master/notebooks/tutorials/]
  (https://github.com/marcotcr/checklist/blob/master/notebooks/tutorials/).
  
  A task suite should contain tests that check general capabilities, including
  but not limited to:
  
  * Vocabulary + POS : Important words/word types for the task
  * Taxonomy : Synonyms/antonyms, etc.
  * Robustness : To typos, irrelevant changes, etc.
  * NER : Appropriately understanding named entities.
  * Temporal : Understanding the order of events.
  * Negation
  * Coreference
  * Semantic Role Labeling : Understanding roles such as agents and objects.
  * Logic : Ability to handle symmetry, consistency, and conjunctions.
  * Fairness
  
  
  # Parameters
  
  suite: `checklist.test_suite.TestSuite`, optional (default = `None`)
      Pass in an existing test suite.
  
  add_default_tests: `bool` (default = `False`)
      Whether to add default checklist tests for the task.
  
  data: `List[Any]`, optional (default = `None`)
      If the data is provided, and `add_default_tests` is `True`,
      tests that perturb the data are also added.
  
      For instance, if the task is sentiment analysis, and the a
      list of sentences is passed, it will add tests that check
      a model's robustness to typos, etc.
docstr: Adds a fully specified checklist test to the suite.
  The tests can be of the following types:
  
  * MFT: A minimum functionality test. It checks if the predicted output
         matches the expected output.
         For example, for a sentiment analysis task, a simple MFT can check
         if the model always predicts a positive sentiment for very
         positive words.
         The test's data contains the input and the expected output.
  
  * INV: An invariance test. It checks if the predicted output is invariant
         to some change in the input.
         For example, for a sentiment analysis task, an INV test can check
         if the prediction stays consistent if simple typos are added.
         The test's data contains the pairs (input, modified input).
  
  * DIR: A directional expectation test. It checks if the predicted output
         changes in some specific way in response to the change in input.
         For example, for a sentiment analysis task, a DIR test can check if
         adding a reducer (eg. "good" -> "somewhat good") causes the
         prediction's positive confidence score to decrease (or at least not
         increase).
         The test's data contains single inputs or pairs (input, modified input).
  
  Please refer to [the paper](https://api.semanticscholar.org/CorpusID:218551201)
  for more details and examples.
  
  Note: `test` needs to be fully specified; with name, capability and description.
docstr_tok: ["Adds","a","fully","specified","checklist","test","to","the","suite",".","The","tests","can","be","of","the","following","types",":","*","MFT",":","A","minimum","functionality","test",".","It","checks","if","the","predicted","output","matches","the","expected","output",".","For","example",",","for","a","sentiment","analysis","task",",","a","simple","MFT","can","check","if","the","model","always","predicts","a","positive","sentiment","for","very","positive","words",".","The","test","'s","data","contains","the","input","and","the","expected","output",".","*","INV",":","An","invariance","test",".","It","checks","if","the","predicted","output","is","invariant","to","some","change","in","the","input",".","For","example",",","for","a","sentiment","analysis","task",",","an","INV","test","can","check","if","the","prediction","stays","consistent","if","simple","typos","are","added",".","The","test","'s","data","contains","the","pairs","(","input",",","modified","input",")",".","*","DIR",":","A","directional","expectation","test",".","It","checks","if","the","predicted","output","changes","in","some","specific","way","in","response","to","the","change","in","input",".","For","example",",","for","a","sentiment","analysis","task",",","a","DIR","test","can","check","if","adding","a","reducer","(","eg",".","``","good","''","-",">","``","somewhat","good","''",")","causes","the","prediction","'s","positive","confidence","score","to","decrease","(","or","at","least","not","increase",")",".","The","test","'s","data","contains","single","inputs","or","pairs","(","input",",","modified","input",")",".","Please","refer","to","[","the","paper","]","(","https",":","\/\/api.semanticscholar.org\/CorpusID:218551201",")","for","more","details","and","examples",".","Note",":","`","test","`","needs","to","be","fully","specified",";","with","name",",","capability","and","description","."]
code: def add_test(self, test: Union[MFT, INV, DIR]):
      """
      Adds a fully specified checklist test to the suite.
      The tests can be of the following types:
  
      * MFT: A minimum functionality test. It checks if the predicted output
             matches the expected output.
             For example, for a sentiment analysis task, a simple MFT can check
             if the model always predicts a positive sentiment for very
             positive words.
             The test's data contains the input and the expected output.
  
      * INV: An invariance test. It checks if the predicted output is invariant
             to some change in the input.
             For example, for a sentiment analysis task, an INV test can check
             if the prediction stays consistent if simple typos are added.
             The test's data contains the pairs (input, modified input).
  
      * DIR: A directional expectation test. It checks if the predicted output
             changes in some specific way in response to the change in input.
             For example, for a sentiment analysis task, a DIR test can check if
             adding a reducer (eg. "good" -> "somewhat good") causes the
             prediction's positive confidence score to decrease (or at least not
             increase).
             The test's data contains single inputs or pairs (input, modified input).
  
      Please refer to [the paper](https://api.semanticscholar.org/CorpusID:218551201)
      for more details and examples.
  
      Note: `test` needs to be fully specified; with name, capability and description.
      """
      if test.data:  # test data should contain at least one example.
          self.suite.add(test)
      else:
          logger.warning(
              "'{}' was not added, as it contains no examples.".format(
                  test.name
              )
          )
  
imports: import sys
  import logging
  from typing import Type, Optional, Dict, Any, Callable, List, Iterable, Union, TextIO, Tuple
  import numpy
  from checklist.test_suite import TestSuite
  from checklist.editor import Editor
  from checklist.test_types import MFT, INV, DIR
  from checklist.perturb import Perturb
  from allennlp.common.registrable import Registrable
  from allennlp.common.file_utils import cached_path
  from allennlp.predictors.predictor import Predictor
  from allennlp.confidence_checks.task_checklists import utils
label_desc: Use image node_id 18 for calling the TaskSuite obj's underlying member method code with example usage: obj.add_test(test) without return types