class_docstr: The BoolQ dataset is from the paper:
  https://arxiv.org/abs/1905.10044
  
  Original repository can be found at:
  https://github.com/google-research-datasets/boolean-questions
  
  BoolQ is a QA dataset containing 15942 (9427 training, 3270 dev, 3245 test) boolean (Yes/No) questions.
  Each sample contains a passage, a question and an answer that are generated in an unconstrained/unprompted setting.
  
  We prompt models using the following format
  
      <passage>
      Question: <question>?
      Answer:
  
      Target completion:
          <answer>
  
  Using an example from the training dataset, we have
  
      Context: Epsom railway station serves the town of Epsom in Surrey. It is located off Waterloo Road and is
      less than two minutes' walk from the High Street.
      It is not in the London Oyster card zone unlike Epsom Downs or Tattenham Corner stations.
      The station building was replaced in 2012/2013 with a new building with apartments above the station.
      Question: Can you use oyster card at epsom station?
      Answer:
  
      Target completion:
          Yes
  
  We also integrate contrast sets for this dataset from the paper:
  https://arxiv.org/abs/2004.02709
  
  Original repository can be found at:
  https://github.com/allenai/contrast-sets
  
  Each sample contains the original <passage, question, answer> triplet, and the human-perturbed version
  i.e. <passage, perturbed question, perturbed answer>.
  
  Contrast Sets for BoolQ contains 339 perturbed questions, forming 70 contrast sets in total.
  Perturbations to the original questions are generated by humans, with the intention of flipping the gold label.
  For more details, see the original paper, Appendix B.9.
  
  An example instance of a perturbation (from the original paper):
  
  ```
  The Fate of the Furious premiered in Berlin on April 4, 2017, and was theatrically released in the
  United States on April 14, 2017, playing in 3D, IMAX 3D and 4DX internationally. . . A spinoff film starring
  Johnson and Statham’s characters is scheduled for release in August 2019, while the ninth and tenth films are
  scheduled for releases on the years 2020 and 2021.
  question: is “Fate and the Furious” the last movie?
  answer: no
  
  perturbed question: is “Fate and the Furious” the first of multiple movies?
  perturbed answer: Yes
  perturbation strategy: adjective change.
  ```
class_docstr_tok: ["The","BoolQ","dataset","is","from","the","paper",":","https",":","\/\/arxiv.org\/abs\/1905.10044","Original","repository","can","be","found","at",":","https",":","\/\/github.com\/google-research-datasets\/boolean-questions","BoolQ","is","a","QA","dataset","containing","15942","(","9427","training",",","3270","dev",",","3245","test",")","boolean","(","Yes\/No",")","questions",".","Each","sample","contains","a","passage",",","a","question","and","an","answer","that","are","generated","in","an","unconstrained\/unprompted","setting",".","We","prompt","models","using","the","following","format","<","passage",">","Question",":","<","question",">","?","Answer",":","Target","completion",":","<","answer",">","Using","an","example","from","the","training","dataset",",","we","have","Context",":","Epsom","railway","station","serves","the","town","of","Epsom","in","Surrey",".","It","is","located","off","Waterloo","Road","and","is","less","than","two","minutes","'","walk","from","the","High","Street",".","It","is","not","in","the","London","Oyster","card","zone","unlike","Epsom","Downs","or","Tattenham","Corner","stations",".","The","station","building","was","replaced","in","2012\/2013","with","a","new","building","with","apartments","above","the","station",".","Question",":","Can","you","use","oyster","card","at","epsom","station","?","Answer",":","Target","completion",":","Yes","We","also","integrate","contrast","sets","for","this","dataset","from","the","paper",":","https",":","\/\/arxiv.org\/abs\/2004.02709","Original","repository","can","be","found","at",":","https",":","\/\/github.com\/allenai\/contrast-sets","Each","sample","contains","the","original","<","passage",",","question",",","answer",">","triplet",",","and","the","human-perturbed","version","i.e",".","<","passage",",","perturbed","question",",","perturbed","answer",">",".","Contrast","Sets","for","BoolQ","contains","339","perturbed","questions",",","forming","70","contrast","sets","in","total",".","Perturbations","to","the","original","questions","are","generated","by","humans",",","with","the","intention","of","flipping","the","gold","label",".","For","more","details",",","see","the","original","paper",",","Appendix","B.9",".","An","example","instance","of","a","perturbation","(","from","the","original","paper",")",":","``","`","The","Fate","of","the","Furious","premiered","in","Berlin","on","April","4",",","2017",",","and","was","theatrically","released","in","the","United","States","on","April","14",",","2017",",","playing","in","3D",",","IMAX","3D","and","4DX","internationally",".",".",".","A","spinoff","film","starring","Johnson","and","Statham","\u2019","s","characters","is","scheduled","for","release","in","August","2019",",","while","the","ninth","and","tenth","films","are","scheduled","for","releases","on","the","years","2020","and","2021.","question",":","is","\u201c","Fate","and","the","Furious","\u201d","the","last","movie","?","answer",":","no","perturbed","question",":","is","\u201c","Fate","and","the","Furious","\u201d","the","first","of","multiple","movies","?","perturbed","answer",":","Yes","perturbation","strategy",":","adjective","change",".","``","`"]
docstr: Args:
    only_contrast: Produce only inputs that have a contrast version.
docstr_tok: ["Args",":","only_contrast",":","Produce","only","inputs","that","have","a","contrast","version","."]
code: def __init__(self, only_contrast=False):
      """
      Args:
        only_contrast: Produce only inputs that have a contrast version.
      """
      super().__init__()
      self.only_contrast = only_contrast
  
file_path: H2O/h2o_hf/helm/src/helm/benchmark/scenarios/boolq_scenario.py
imports: import json
  import os
  from typing import List, Dict, Optional
  from helm.common.general import ensure_file_downloaded, ensure_directory_exists
  from .scenario import Scenario, Instance, Reference, TRAIN_SPLIT, VALID_SPLIT, CORRECT_TAG, PassageQuestionInput, Input, Output
label_desc: Use image node_id 2 to create a new BoolQScenario object from inherited base classes: Scenario with example: obj = BoolQScenario(only_contrast)