class_docstr: The NaturalQA dataset is from the paper:
  https://ai.google/research/pubs/pub47761
  
  Original repository can be found at:
  https://github.com/google-research-datasets/natural-questions
  
  This scenario is adapted from https://huggingface.co/datasets/natural_questions
  
  NaturalQA is a dataset containing 307,373 training examples with one-way
  annotations, 7,830 development examples with 5-way annotations, and 7,842 5-way annotated
  test examples. Each example consists of a context (a wikipedia document), a question, and
  one or five manually annotated long and short answers. The short answer is either a set of
  entities in the long answer, yes/no or Null.
  
  In this scenario, we restrict our attention to short answers. For efficiency, we
  use only the dev set---splitting in into train/validation. Additionally, we omit
  all samples in the dev set for which none of the annotators provided a short
  answer (and exclude the separate yes/no field). We only provide a single (randomly chosen)
  answer during training, and the set of all possible answers during validation.
  
  We consider three modes of this scenario:
  
  1. closed book: No context provided
  2. open book w/ wiki document: The entire wiki document is used as context
  3. open book w/ long answer: Only the long answer marked by the annotators is
      provided as the context.
  
  The motivation to consider (3) is that the entire wiki document may not fit into
  the language model's context window.
  
  Concretely, we prompt models using the following format:
  
      (Optional) Title: <title_1>
      (Optional) Context: <context text_1>
      Question: <question_1>
      Answer: <answer_1>
      (Optional) Title: <title_2>
      (Optional) Context: <context text_2>
      Question: <question_2>
      Answer: <answer_2>
      ...
      Optional) Title: <title_k>
      (Optional) Context: <context text_k>
      Question: <question_k>
      Answer:
      Target completion:
          <answer>
  
  Example (mode:closed):
  
  ```
  Question: how many customers does edf have in the uk
  Answer: '5.7 million'
  
  Question: who is the largest supermarket chain in the uk
  ```
  
  Reference
  
  ```
  ['Tesco', 'Aldi']
  ```
  
  Example (mode:open_longans)
  
  ```
  Context: A dissenting opinion (or dissent) is an opinion in a legal case in certain legal
  systems written by one or more judges expressing disagreement with the majority opinion
  of the court which gives rise to its judgment. When not necessarily
  referring to a legal decision, this can also be referred to as a minority report.[1][2]
  
  Question: a justice of the supreme court may write a dissenting opinion to
  Answer: 'the majority opinion of the court'
  
  Context: Set and filmed in New York City and based on the 1997 book of the same name by
  Candace Bushnell, the show follows the lives of a group of four women—three in their
  mid-thirties and one in her forties—who, despite their different natures and
  ever-changing sex lives, remain inseparable and confide in each other. Starring Sarah
  Jessica Parker (as Carrie Bradshaw), Kim Cattrall (as Samantha Jones), Kristin Davis
  (as Charlotte York), and Cynthia Nixon (as Miranda Hobbes), the quirky series had multiple
  continuing storylines that tackled relevant and modern social issues such as sexuality,
  safe sex, promiscuity, and femininity, while exploring the difference between friendships
  and romantic relationships. The deliberate omission of the better part of the early
  lives of the four women was the writers' way of exploring social life – from sex to
  relationships – through each of their four very different, individual perspectives.
  
  Question: where does sex and the city take place
  ```
  
  Reference
  
  ```
  ['New York City']
  ```
  
  Example (mode:wiki)
  
  ```
  Title: Upstream (petroleum industry)
  
  Context: Upstream ( petroleum industry ) - wikipedia  Upstream ( petroleum industry )  Jump to :
  navigation, search For other uses, see Upstream (disambiguation).  The oil and gas industry
  is usually divided into three major sectors : upstream
  ( or exploration and production - E&P),...
  
  Question: what is upstream project in oil and gas
  Answer: 'searching for potential underground or underwater crude oil and natural gas fields,
  drilling exploratory wells, and subsequently drilling and operating the wells that recover and
  bring the crude oil or raw natural gas to the surface'
  
  Title: Collective Soul
  
  Context: Collective Soul - Wikipedia  Collective Soul  Jump to : navigation , search
  For other uses , see Collective Soul (disambiguation ) .      This article needs additional
  citations for verification .  Please help improve this article by adding citations to
  reliable sources . Unsourced material may be challenged and removed .( September 2009 )
  ( Learn how and when to remove this template message )       Collective Soul     Collective Soul
  performing at MMRBQ 2016 , Camden NJ May 21 , 2016 ...
  
  Question: who is the lead singer of collective soul
  ```
  
  Reference
  
  ```
  ['Ed Roland']
  ```
docstr: Given an example in dataset format, create the prompt and the list of
  correct references.
docstr_tok: ["Given","an","example","in","dataset","format",",","create","the","prompt","and","the","list","of","correct","references","."]
code: def create_raw_instance(self, sample: Dict) -> RawInstance:
      """
      Given an example in dataset format, create the prompt and the list of
      correct references.
      """
      # TODO: this detokenization is pretty naive, improve
      document = " ".join(
          [self._clean_token(t) for t in sample["document_tokens"]]
      )
      html_bytes = sample["document_html"].encode("utf-8")
      title = sample["document_title"]
      question = sample["question_text"]
  
      short_answers, long_answers = [], []
      for ans_json in sample["annotations"]:
          long_ans = ans_json["long_answer"]
          long_ans = self._clean_text(
              html_bytes[long_ans["start_byte"] : long_ans["end_byte"]]
          )
  
          for ans in ans_json["short_answers"]:
              short_ans = html_bytes[
                  ans["start_byte"] : ans["end_byte"]
              ]
              short_ans = self._clean_text(short_ans)
  
              short_answers.append(short_ans)
              long_answers.append(long_ans)
  
      return RawInstance(
          title=title,
          document=document,
          question=question,
          long_answers=long_answers,
          short_answers=short_answers,
      )
  
imports: from dataclasses import dataclass
  import dacite
  import gzip
  import json
  import os
  import re
  import html
  import random
  from helm.common.hierarchical_logger import htrack_block, hlog
  from typing import List, Dict
  from helm.common.general import ensure_file_downloaded, ensure_directory_exists, asdict_without_nones
  from .scenario import Scenario, Instance, Reference, TRAIN_SPLIT, VALID_SPLIT, CORRECT_TAG, PassageQuestionInput, Input, Output
label_desc: Use image node_id 4 for calling the NaturalQAScenario obj's underlying member method code with example usage: obj.create_raw_instance(sample) and returns: RawInstance