class_docstr: The NewsQA dataset is from the paper:
  https://arxiv.org/abs/1611.09830
  
  Original repository can be found at:
  https://github.com/Maluuba/newsqa
  
  Note: The training dataset cannot be directly shared due to copyright issues, and needs to be downloaded by
  following the instructions in the repo above. These instructions are duplicated here for
  convenience.
  
  1. Clone the repo (https://github.com/Maluuba/newsqa)
  2. Download the data from (https://msropendata.com/datasets/939b1042-6402-4697-9c15-7a28de7e1321).
  You need to create a login account to download this data.
  3. Download the CNN stories tar file from "https://cs.nyu.edu/~kcho/DMQA/"
  4. Create the conda environment using the command (conda create --name newsqa python=2.7 "pandas>=0.19.2")
  5. Install the requirements (conda activate newsqa && pip install --requirement requirements.txt)
  
  This should result in the creation of the file (combined-newsqa-data-v1.json) in the repo
  which is used in this scenario.
  
  NewsQA is a QA dataset containing 12,744 stories,
  and over 119,633 question-answer pairs. There are 92549 training qa pairs,
  5166 qas in the dev set, and 5126 in the test set.
  Particularly, given the a news article from CNN,
  the goal is answer questions with answers consisting of spans of text from the corresponding articles.
  All of the questions and answers are written by crowd sourced human annotators.
  For more details, see https://arxiv.org/abs/1611.09830.
  
  More concretely, we prompt models using the following format
  
      Passage: <news article>
      Question: <question>
      Answer:
  
  Note: Some of the questions do not have an answer in the context so the
  model needs to answer "No Answer". While this behavior might be tricky to
  learn in the few-shot setting, we still include these examples in the
  scenario.
  
  Using an example from the training dataset, we have:
  
  ```
  NEW DELHI, India (CNN) -- A high court in northern India on Friday acquitted a wealthy businessman
  facing the death sentence for the killing of a teen in a case dubbed 'the house of horrors.'
  Moninder Singh Pandher was sentenced to death by a lower court in February...
  Question: Who was sentenced to death in February?
  Answer:
  ```
  
  References
  
  ```
  ['Moninder Singh Pandher']
  ```
class_docstr_tok: ["The","NewsQA","dataset","is","from","the","paper",":","https",":","\/\/arxiv.org\/abs\/1611.09830","Original","repository","can","be","found","at",":","https",":","\/\/github.com\/Maluuba\/newsqa","Note",":","The","training","dataset","can","not","be","directly","shared","due","to","copyright","issues",",","and","needs","to","be","downloaded","by","following","the","instructions","in","the","repo","above",".","These","instructions","are","duplicated","here","for","convenience",".","1",".","Clone","the","repo","(","https",":","\/\/github.com\/Maluuba\/newsqa",")","2",".","Download","the","data","from","(","https",":","\/\/msropendata.com\/datasets\/939b1042-6402-4697-9c15-7a28de7e1321",")",".","You","need","to","create","a","login","account","to","download","this","data",".","3",".","Download","the","CNN","stories","tar","file","from","``","https",":","\/\/cs.nyu.edu\/~kcho\/DMQA\/","''","4",".","Create","the","conda","environment","using","the","command","(","conda","create","--","name","newsqa","python=2.7","``","pandas",">","=0.19.2","''",")","5",".","Install","the","requirements","(","conda","activate","newsqa","&","&","pip","install","--","requirement","requirements.txt",")","This","should","result","in","the","creation","of","the","file","(","combined-newsqa-data-v1.json",")","in","the","repo","which","is","used","in","this","scenario",".","NewsQA","is","a","QA","dataset","containing","12,744","stories",",","and","over","119,633","question-answer","pairs",".","There","are","92549","training","qa","pairs",",","5166","qas","in","the","dev","set",",","and","5126","in","the","test","set",".","Particularly",",","given","the","a","news","article","from","CNN",",","the","goal","is","answer","questions","with","answers","consisting","of","spans","of","text","from","the","corresponding","articles",".","All","of","the","questions","and","answers","are","written","by","crowd","sourced","human","annotators",".","For","more","details",",","see","https",":","\/\/arxiv.org\/abs\/1611.09830",".","More","concretely",",","we","prompt","models","using","the","following","format","Passage",":","<","news","article",">","Question",":","<","question",">","Answer",":","Note",":","Some","of","the","questions","do","not","have","an","answer","in","the","context","so","the","model","needs","to","answer","``","No","Answer","''",".","While","this","behavior","might","be","tricky","to","learn","in","the","few-shot","setting",",","we","still","include","these","examples","in","the","scenario",".","Using","an","example","from","the","training","dataset",",","we","have",":","``","`","NEW","DELHI",",","India","(","CNN",")","--","A","high","court","in","northern","India","on","Friday","acquitted","a","wealthy","businessman","facing","the","death","sentence","for","the","killing","of","a","teen","in","a","case","dubbed","'the","house","of","horrors",".","'","Moninder","Singh","Pandher","was","sentenced","to","death","by","a","lower","court","in","February","...","Question",":","Who","was","sentenced","to","death","in","February","?","Answer",":","``","`","References","``","`","[","'Moninder","Singh","Pandher","'","]","``","`"]
docstr: Helper for generating instances for a split.
  Args:
      target_file (str): Data file.
      splits (dict): Which splits to partition the data into.
  Returns:
      List[Instance]: Instances from the file for the specified split.
docstr_tok: ["Helper","for","generating","instances","for","a","split",".","Args",":","target_file","(","str",")",":","Data","file",".","splits","(","dict",")",":","Which","splits","to","partition","the","data","into",".","Returns",":","List","[","Instance","]",":","Instances","from","the","file","for","the","specified","split","."]
code: def get_file_instances(
      self, target_file: str, splits: Dict
  ) -> List[Instance]:
      """
      Helper for generating instances for a split.
      Args:
          target_file (str): Data file.
          splits (dict): Which splits to partition the data into.
      Returns:
          List[Instance]: Instances from the file for the specified split.
      """
      file_instances: List[Instance] = []
      with open(target_file, encoding="utf-8") as f:
          all_samples: List[Dict] = json.load(f)["data"]
  
      clean_samples: List[Dict] = self.cleaned_samples(all_samples)
      for sample in clean_samples:
          prompt, answers = self.process_example(sample)
          split = "train" if sample["type"] == "train" else "valid"
          instance = Instance(
              input=prompt,
              references=[
                  Reference(Output(text=answer), tags=[CORRECT_TAG])
                  for answer in answers
              ],
              split=splits[split],
          )
          file_instances.append(instance)
      return file_instances
  
file_path: H2O/h2o_hf/helm/src/helm/benchmark/scenarios/newsqa_scenario.py
imports: import os
  import json
  import random
  from typing import Dict, List, Tuple
  from .scenario import Scenario, Instance, Reference, TRAIN_SPLIT, VALID_SPLIT, CORRECT_TAG, PassageQuestionInput, Input, Output
label_desc: Use image node_id 3 for calling the NewsQAScenario obj's underlying member method code with example usage: obj.get_file_instances(target_file, splits) and returns: file_instances