class_docstr: From Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  (https://arxiv.org/abs/2206.04615), the Beyond the Imitation Game Benchmark (BIG-bench) is a
  collaborative benchmark with more than 200 tasks intended to probe large language models and extrapolate
  their future capabilities.
  
  `BigBenchScenario` currently only supports JSON tasks and not programmatic tasks.
  See https://github.com/google/BIG-bench#creating-a-programmatic-task for more information.
  
  The following is a comprehensive list of the JSON tasks and programmatic tasks:
  https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/keywords_to_tasks.md#json.
  
  ```
  @misc{https://doi.org/10.48550/arxiv.2206.04615,
    doi = {10.48550/ARXIV.2206.04615},
    url = {https://arxiv.org/abs/2206.04615},
    author = {Srivastava et al.},
    title = {Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models},
    publisher = {arXiv},
    year = {2022},
    copyright = {arXiv.org perpetual, non-exclusive license}
  }
  ```
class_docstr_tok: ["From","Beyond","the","Imitation","Game",":","Quantifying","and","extrapolating","the","capabilities","of","language","models","(","https",":","\/\/arxiv.org\/abs\/2206.04615",")",",","the","Beyond","the","Imitation","Game","Benchmark","(","BIG-bench",")","is","a","collaborative","benchmark","with","more","than","200","tasks","intended","to","probe","large","language","models","and","extrapolate","their","future","capabilities",".","`","BigBenchScenario","`","currently","only","supports","JSON","tasks","and","not","programmatic","tasks",".","See","https",":","\/\/github.com\/google\/BIG-bench","#","creating-a-programmatic-task","for","more","information",".","The","following","is","a","comprehensive","list","of","the","JSON","tasks","and","programmatic","tasks",":","https",":","\/\/github.com\/google\/BIG-bench\/blob\/main\/bigbench\/benchmark_tasks\/keywords_to_tasks.md","#","json",".","``","`","@","misc","{","https",":","\/\/doi.org\/10.48550\/arxiv.2206.04615",",","doi","=","{","10.48550\/ARXIV.2206.04615","}",",","url","=","{","https",":","\/\/arxiv.org\/abs\/2206.04615","}",",","author","=","{","Srivastava","et","al",".","}",",","title","=","{","Beyond","the","Imitation","Game",":","Quantifying","and","extrapolating","the","capabilities","of","language","models","}",",","publisher","=","{","arXiv","}",",","year","=","{","2022","}",",","copyright","=","{","arXiv.org","perpetual",",","non-exclusive","license","}","}","``","`"]
docstr: Downloads the task JSON from https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks
  if it doesn't already exist. Then, loads the BIG-bench task definition from task.json.
docstr_tok: ["Downloads","the","task","JSON","from","https",":","\/\/github.com\/google\/BIG-bench\/tree\/main\/bigbench\/benchmark_tasks","if","it","does","n't","already","exist",".","Then",",","loads","the","BIG-bench","task","definition","from","task.json","."]
code: def download_and_get_task(
      output_path: str, task: str, subtask: str
  ) -> Dict:
      """
      Downloads the task JSON from https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks
      if it doesn't already exist. Then, loads the BIG-bench task definition from task.json.
      """
      ensure_directory_exists(output_path)
      task_path: str = os.path.join(output_path, task)
      ensure_directory_exists(task_path)
  
      base_url: str = f"https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/{task}/"
      if subtask:
          base_url = urljoin(base_url, f"{subtask}/")
          task_path = os.path.join(task_path, subtask)
          ensure_directory_exists(task_path)
  
      target_path: str = os.path.join(
          task_path, BIGBenchScenario.TASK_FILE_NAME
      )
      ensure_file_downloaded(
          source_url=urljoin(base_url, BIGBenchScenario.TASK_FILE_NAME),
          target_path=target_path,
      )
      with open(target_path, "r") as f:
          return json.load(f)
  
file_path: H2O/h2o_hf/helm/src/helm/benchmark/scenarios/big_bench_scenario.py
imports: import json
  import os
  import random
  from typing import List, Dict
  from urllib.parse import urljoin
  from helm.common.general import ensure_directory_exists, ensure_file_downloaded
  from .scenario import Scenario, Instance, Reference, Input, CORRECT_TAG, TRAIN_SPLIT, VALID_SPLIT, TEST_SPLIT, Output
label_desc: Use image node_id 1 for calling the BIGBenchScenario obj's underlying member method code with example usage: obj.download_and_get_task(output_path, task, subtask) and returns: json