class_docstr: Scenario implementing MS MARCO challenge tasks.
  
  I. Overview
  
      MS MARCO (Microsoft MAchine Reading COmprehension) is a collection of
      large search datasets, collected using BING search questions, first
      released in (Bajaj et. al., 2016) and expanded ever since. The official
      MS MARCO website details all the available datasets and the proposed
      tasks: https://microsoft.github.io/msmarco/.
  
  II. Task
  
      In this scenario, we are focusing on information retrieval tasks from
      the MS MARCO benchmark. We frame the information retrieval task as a
      binary classification problem, similar to
      (Nogueira and Jiang et. al., 2020). Specifically, given a context and a
      question, the model's job is to predict whether the context includes an
      answer to the question by producing either a correct answer or a wrong
      answer. The specific tokens used for the correct and wrong answers are
      specified in the adapter specification. Shared below is an example of
      how we would construct a prompt for a question, using 4 in-context
      training instances, where the correct and wrong output tokens are
      respectively specified as `Yes` and `No` in the adapter specification.
      Note that the last instance in the example, which is the instance we
      are evaluating, doesn't have an answer - since we want our model to
      answer the question.
  
          Passage: Its 25 drops per ml, you guys are all wrong. If it is water, the standard was changed 15 - 20 years ago to make 20 drops = 1mL. The viscosity of most things is temperature dependent, so this would be at room temperature. Hope this helps.
          Query: how many eye drops per ml
          Does the passage answer the query?
          Answer: Yes
  
          Passage: RE: How many eyedrops are there in a 10 ml bottle of Cosopt? My Kaiser pharmacy insists that 2 bottles should last me 100 days but I run out way before that time when I am using 4 drops per day.In the past other pharmacies have given me 3 10-ml bottles for 100 days.E: How many eyedrops are there in a 10 ml bottle of Cosopt? My Kaiser pharmacy insists that 2 bottles should last me 100 days but I run out way before that time when I am using 4 drops per day.
          Query: how many eye drops per ml
          Does the passage answer the query?
          Answer: No
  
          Passage: : You can transfer money to your checking account from other Wells Fargo. accounts through Wells Fargo Mobile Banking with the mobile app, online, at any. Wells Fargo ATM, or at a Wells Fargo branch. 1 Money in — deposits.
          Query: can you open a wells fargo account online
          Does the passage answer the query?
          Answer: No
  
          Passage: You can open a Wells Fargo banking account from your home or even online. It is really easy to do, provided you have all of the appropriate documentation. Wells Fargo has so many bank account options that you will be sure to find one that works for you. They offer free checking accounts with free online banking.
          Query: can you open a wells fargo account online
          Does the passage answer the query?
          Answer: Yes
  
          Passage: SIZE: There are few measurements available for this bear. Adult spectacled bears can weigh between 140 and 385 pounds (63-173 kg). However, the body length of adults is about 150 to 180 centimeters (60 to 72 inches) and males may be 30 to 40 percent larger than females.
          Query: how much does a spectacled bear weigh
          Does the passage answer the query?
          Answer:
  
      As a result of each request, the model would produce a token or a set
      of tokens. To determine the ranking of a list of contexts for a
      question, we create a separate request for each context, where we pair
      the question with the context and ask for model's answer.
  
      Then, in the corresponding metric for our scenario, the contexts are
      ranked using the answer token and its log probability. Specifically, the
      ordering looks like the list given below, from good contexts at the top
      to bad contexts at the bottom, where UNKNOWN_ANSWER would correspond
      to any token that is not one of correct or wrong answer tokens, using
      case insensitive match excluding whitespace.
  
          (1) CORRECT_ANSWER, highest log probability
              ...
              CORRECT_ANSWER, lowest log probability
              ...
              WRONG_ANSWER, lowest log probability
              ...
              WRONG_ANSWER, highest log probability
              ...
          (n) UNKNOWN_ANSWER(s)
  
      We then use standard information retrieval metrics, such as RR and
      nDCG, to score the model using the rankings obtained using the strategy
      described above.
  
  III. Datasets
  
      There are two ranking tasks in the MS MARCO benchmark: document ranking
      and passage ranking. Both of these tasks have several tracks, using
      different subsets for the evaluation of the models. This scenario
      currently supports the passage ranking task tracks.
  
      All the datasets used in this scenario are hosted and retrieved from one
      of the following repositories:
  
          Official MS MARCO Website      | https://microsoft.github.io/msmarco/
          Benchmarking CodaLab Worksheet | https://worksheets.codalab.org/worksheets/0xf451c0dec2a6414aae0b68e8e325426c  # noqa
          TREC Website                   | https://trec.nist.gov
  
      This scenario makes use of 4 different types of files, explanation for
      each is given below, followed by a table listing the details for each
      of the datasets used.
  
          document: The document files contain all the documents that could be
              ranked for a question, each specified with an document ID
              (docid). For example, for the passage track, the documents would
              be passages.
          query: The query files contain the questions for a given task,
              each specified with a query ID (qid). Each task has a query file
              including the training examples. The validation queries are
              determined by the selected track of the task. Depending on the
              task and split/track, the queries read from the queries file
              are filtered to ensure they have corresponding qrels and top-k
              information before instances for the query are created. Because
              of this filtering, the number of queries in the query file
              doesn't directly correspond the number of queries for which
              instances can be created.
          qrels: Each query file is accompanied by a qrels file, which
              specifies the relationship between a query with ID qid and an
              document with ID docid. The relevance values can have different
              meanings depending on the split and the track. Note that not
              all queries would have corresponding query relevances in the
              accompanied file. Also note that multiple documents may have the
              same relevance value with a qiven query.
          topk: Each query file is accompanied by a top-k file, which lists
              the IDs of the top k best documents for a query with their
              accompanied rank. The top documents for each query were selected
              using the BM25 algorithm. The notebook used to generate the
              top-k files used in this scenario can be found at the
              Benchmarking CodaLab Worksheet. Note that not all queries would
              have a corresponding top-k documents in the accompanied file.
  
          |      LOCAL FILE NAME        |  TRACK  |  TRACK  |              CONTENT              |        FORMAT         |              Host              | Notes |  # noqa
          | passage_document.tsv        | passage |    -    | 8,841,823 passages                | <docid> <text>        | Benchmarking CodaLab Worksheet | (1)   |  # noqa
          | passage_train_queries.tsv   | passage |    -    | 808,731   queries                 | <qid> <text>          | Official MS MARCO Website      |       |  # noqa
          | passage_train_qrels.tsv     | passage |    -    | 532,761   query relations         | <qid> 0 <docid> <rel> | Official MS MARCO Website      | (2)   |  # noqa
          | passage_train_topk.tsv      | passage |    -    | 20        top documents per query | <qid> <docid> <rank>  | Benchmarking CodaLab Worksheet | (3)   |  # noqa
          | passage_regular_queries.tsv | passage | regular | 6980      queries                 | <qid> <text>          | Official MS MARCO Website      | (4)   |  # noqa
          | passage_regular_qrels.tsv   | passage | regular | 7437      query relations         | <qid> 0 <docid> <rel> | Official MS MARCO Website      | (2)   |  # noqa
          | passage_regular_topk.tsv    | passage | regular | 1000      top documents per query | <qid> <docid> <rank>  | Benchmarking CodaLab Worksheet |       |  # noqa
          | passage_trec_queries.tsv    | passage | trec    | 200       queries                 | <qid> <text>          | Official MS MARCO Website      |       |  # noqa
          | passage_trec_qrels.tsv      | passage | trec    | 502,982   query relations         | <qid> 0 <docid> <rel> | Official MS MARCO Website      | (5)   |  # noqa
          | passage_trec_topk.tsv       | passage | trec    | 1000      top documents per query | <qid> <docid> <rank>  | Benchmarking CodaLab Worksheet |       |  # noqa
  
              Notes:
                  (1) We use a pre-processed version of the passage
                      collection, introduced in (MacAvaney, et. al. 2021),
                      which greatly improves the quality of the passages. The
                      cleaned collection is hosted on the Benchmarking CodaLab
                      Worksheet as there is no other reliable publicly hosted
                      copy.
                  (2) The only relevance values is 1, which indicates that the
                      document with the ID the docid is the gold match for the
                      query with the ID qid.
                  (3) The number of top documents ranked was limited to 20 for
                      the training set as we only generate 2 instances per
                      training query, one corresponding to a gold matching
                      instance, and the other one corresponding to a probable
                      non-matching instance.
                  (4) The labels (qrels) of the official test queries are not
                      publicly released. Since we need to have access to the
                      qrels file to evaluate the models, we instead use the
                      development set ("queries.dev.small.tsv"), which can be
                      found at
                      https://msmarco.blob.core.windows.net/msmarcoranking/collectionandqueries.tar.gz
                  (5) The relevance values for the TREC task of the passage
                      track can be any of [0, 1, 2, 3]. We consider [0, 1] to
                      be wrong matches and [2, 3] to be gold matches.
  
  IV. Baselines
  
      Currently, we use 4 baselines for the MS MARCO scenario, details for
      which are summarized in the table below.
  
          Baseline | The baseline name.
          #VR      | Number of validation requests. For each effective
                     validation query, multiple requests would be created,
                     governed by the provided parameters.
  
      | Baseline                |  #VR   | Parameters
      | regular_topk            | 10,000 | track=regular,use_topk_passages=True,valid_topk=50
      | regular_topk_with_qrels | 10,085 | track=regular,use_qrels_passages=True,use_topk_passages=True,valid_topk=50
      | trec_topk               | 4300   | track=trec,use_topk_passages=True,valid_topk=100
      | trec_qrels              | 9260   | track=trec,use_qrels_passages=True
  
      On average, the requests for the MS MARCO scenario have ~550 tokens
      using the GPT-2 tokenizer. Multiplying this number with the #VR column
      gives an estimate on the number of request tokens that would be required
      to run the given baseline on GPT models.
  
  References
  
       (Bajaj et. al., 2016)              | https://arxiv.org/abs/1611.09268
       (Nogueira and Jiang et. al., 2020) | https://arxiv.org/abd/2003.06713
       (MacAvaney, et. al. 2021)          | https://arxiv.org/abs/2103.02280
class_docstr_tok: ["Scenario","implementing","MS","MARCO","challenge","tasks",".","I.","Overview","MS","MARCO","(","Microsoft","MAchine","Reading","COmprehension",")","is","a","collection","of","large","search","datasets",",","collected","using","BING","search","questions",",","first","released","in","(","Bajaj","et",".","al.",",","2016",")","and","expanded","ever","since",".","The","official","MS","MARCO","website","details","all","the","available","datasets","and","the","proposed","tasks",":","https",":","\/\/microsoft.github.io\/msmarco\/",".","II",".","Task","In","this","scenario",",","we","are","focusing","on","information","retrieval","tasks","from","the","MS","MARCO","benchmark",".","We","frame","the","information","retrieval","task","as","a","binary","classification","problem",",","similar","to","(","Nogueira","and","Jiang","et",".","al.",",","2020",")",".","Specifically",",","given","a","context","and","a","question",",","the","model","'s","job","is","to","predict","whether","the","context","includes","an","answer","to","the","question","by","producing","either","a","correct","answer","or","a","wrong","answer",".","The","specific","tokens","used","for","the","correct","and","wrong","answers","are","specified","in","the","adapter","specification",".","Shared","below","is","an","example","of","how","we","would","construct","a","prompt","for","a","question",",","using","4","in-context","training","instances",",","where","the","correct","and","wrong","output","tokens","are","respectively","specified","as","`","Yes","`","and","`","No","`","in","the","adapter","specification",".","Note","that","the","last","instance","in","the","example",",","which","is","the","instance","we","are","evaluating",",","does","n't","have","an","answer","-","since","we","want","our","model","to","answer","the","question",".","Passage",":","Its","25","drops","per","ml",",","you","guys","are","all","wrong",".","If","it","is","water",",","the","standard","was","changed","15","-","20","years","ago","to","make","20","drops","=","1mL",".","The","viscosity","of","most","things","is","temperature","dependent",",","so","this","would","be","at","room","temperature",".","Hope","this","helps",".","Query",":","how","many","eye","drops","per","ml","Does","the","passage","answer","the","query","?","Answer",":","Yes","Passage",":","RE",":","How","many","eyedrops","are","there","in","a","10","ml","bottle","of","Cosopt","?","My","Kaiser","pharmacy","insists","that","2","bottles","should","last","me","100","days","but","I","run","out","way","before","that","time","when","I","am","using","4","drops","per","day.In","the","past","other","pharmacies","have","given","me","3","10-ml","bottles","for","100","days.E",":","How","many","eyedrops","are","there","in","a","10","ml","bottle","of","Cosopt","?","My","Kaiser","pharmacy","insists","that","2","bottles","should","last","me","100","days","but","I","run","out","way","before","that","time","when","I","am","using","4","drops","per","day",".","Query",":","how","many","eye","drops","per","ml","Does","the","passage","answer","the","query","?","Answer",":","No","Passage",":",":","You","can","transfer","money","to","your","checking","account","from","other","Wells","Fargo",".","accounts","through","Wells","Fargo","Mobile","Banking","with","the","mobile","app",",","online",",","at","any",".","Wells","Fargo","ATM",",","or","at","a","Wells","Fargo","branch",".","1","Money","in","\u2014","deposits",".","Query",":","can","you","open","a","wells","fargo","account","online","Does","the","passage","answer","the","query","?","Answer",":","No","Passage",":","You","can","open","a","Wells","Fargo","banking","account","from","your","home","or","even","online",".","It","is","really","easy","to","do",",","provided","you","have","all","of","the","appropriate","documentation",".","Wells","Fargo","has","so","many","bank","account","options","that","you","will","be","sure","to","find","one","that","works","for","you",".","They","offer","free","checking","accounts","with","free","online","banking",".","Query",":","can","you","open","a","wells","fargo","account","online","Does","the","passage","answer","the","query","?","Answer",":","Yes","Passage",":","SIZE",":","There","are","few","measurements","available","for","this","bear",".","Adult","spectacled","bears","can","weigh","between","140","and","385","pounds","(","63-173","kg",")",".","However",",","the","body","length","of","adults","is","about","150","to","180","centimeters","(","60","to","72","inches",")","and","males","may","be","30","to","40","percent","larger","than","females",".","Query",":","how","much","does","a","spectacled","bear","weigh","Does","the","passage","answer","the","query","?","Answer",":","As","a","result","of","each","request",",","the","model","would","produce","a","token","or","a","set","of","tokens",".","To","determine","the","ranking","of","a","list","of","contexts","for","a","question",",","we","create","a","separate","request","for","each","context",",","where","we","pair","the","question","with","the","context","and","ask","for","model","'s","answer",".","Then",",","in","the","corresponding","metric","for","our","scenario",",","the","contexts","are","ranked","using","the","answer","token","and","its","log","probability",".","Specifically",",","the","ordering","looks","like","the","list","given","below",",","from","good","contexts","at","the","top","to","bad","contexts","at","the","bottom",",","where","UNKNOWN_ANSWER","would","correspond","to","any","token","that","is","not","one","of","correct","or","wrong","answer","tokens",",","using","case","insensitive","match","excluding","whitespace",".","(","1",")","CORRECT_ANSWER",",","highest","log","probability","...","CORRECT_ANSWER",",","lowest","log","probability","...","WRONG_ANSWER",",","lowest","log","probability","...","WRONG_ANSWER",",","highest","log","probability","...","(","n",")","UNKNOWN_ANSWER","(","s",")","We","then","use","standard","information","retrieval","metrics",",","such","as","RR","and","nDCG",",","to","score","the","model","using","the","rankings","obtained","using","the","strategy","described","above",".","III",".","Datasets","There","are","two","ranking","tasks","in","the","MS","MARCO","benchmark",":","document","ranking","and","passage","ranking",".","Both","of","these","tasks","have","several","tracks",",","using","different","subsets","for","the","evaluation","of","the","models",".","This","scenario","currently","supports","the","passage","ranking","task","tracks",".","All","the","datasets","used","in","this","scenario","are","hosted","and","retrieved","from","one","of","the","following","repositories",":","Official","MS","MARCO","Website","|","https",":","\/\/microsoft.github.io\/msmarco\/","Benchmarking","CodaLab","Worksheet","|","https",":","\/\/worksheets.codalab.org\/worksheets\/0xf451c0dec2a6414aae0b68e8e325426c","#","noqa","TREC","Website","|","https",":","\/\/trec.nist.gov","This","scenario","makes","use","of","4","different","types","of","files",",","explanation","for","each","is","given","below",",","followed","by","a","table","listing","the","details","for","each","of","the","datasets","used",".","document",":","The","document","files","contain","all","the","documents","that","could","be","ranked","for","a","question",",","each","specified","with","an","document","ID","(","docid",")",".","For","example",",","for","the","passage","track",",","the","documents","would","be","passages",".","query",":","The","query","files","contain","the","questions","for","a","given","task",",","each","specified","with","a","query","ID","(","qid",")",".","Each","task","has","a","query","file","including","the","training","examples",".","The","validation","queries","are","determined","by","the","selected","track","of","the","task",".","Depending","on","the","task","and","split\/track",",","the","queries","read","from","the","queries","file","are","filtered","to","ensure","they","have","corresponding","qrels","and","top-k","information","before","instances","for","the","query","are","created",".","Because","of","this","filtering",",","the","number","of","queries","in","the","query","file","does","n't","directly","correspond","the","number","of","queries","for","which","instances","can","be","created",".","qrels",":","Each","query","file","is","accompanied","by","a","qrels","file",",","which","specifies","the","relationship","between","a","query","with","ID","qid","and","an","document","with","ID","docid",".","The","relevance","values","can","have","different","meanings","depending","on","the","split","and","the","track",".","Note","that","not","all","queries","would","have","corresponding","query","relevances","in","the","accompanied","file",".","Also","note","that","multiple","documents","may","have","the","same","relevance","value","with","a","qiven","query",".","topk",":","Each","query","file","is","accompanied","by","a","top-k","file",",","which","lists","the","IDs","of","the","top","k","best","documents","for","a","query","with","their","accompanied","rank",".","The","top","documents","for","each","query","were","selected","using","the","BM25","algorithm",".","The","notebook","used","to","generate","the","top-k","files","used","in","this","scenario","can","be","found","at","the","Benchmarking","CodaLab","Worksheet",".","Note","that","not","all","queries","would","have","a","corresponding","top-k","documents","in","the","accompanied","file",".","|","LOCAL","FILE","NAME","|","TRACK","|","TRACK","|","CONTENT","|","FORMAT","|","Host","|","Notes","|","#","noqa","|","passage_document.tsv","|","passage","|","-","|","8,841,823","passages","|","<","docid",">","<","text",">","|","Benchmarking","CodaLab","Worksheet","|","(","1",")","|","#","noqa","|","passage_train_queries.tsv","|","passage","|","-","|","808,731","queries","|","<","qid",">","<","text",">","|","Official","MS","MARCO","Website","|","|","#","noqa","|","passage_train_qrels.tsv","|","passage","|","-","|","532,761","query","relations","|","<","qid",">","0","<","docid",">","<","rel",">","|","Official","MS","MARCO","Website","|","(","2",")","|","#","noqa","|","passage_train_topk.tsv","|","passage","|","-","|","20","top","documents","per","query","|","<","qid",">","<","docid",">","<","rank",">","|","Benchmarking","CodaLab","Worksheet","|","(","3",")","|","#","noqa","|","passage_regular_queries.tsv","|","passage","|","regular","|","6980","queries","|","<","qid",">","<","text",">","|","Official","MS","MARCO","Website","|","(","4",")","|","#","noqa","|","passage_regular_qrels.tsv","|","passage","|","regular","|","7437","query","relations","|","<","qid",">","0","<","docid",">","<","rel",">","|","Official","MS","MARCO","Website","|","(","2",")","|","#","noqa","|","passage_regular_topk.tsv","|","passage","|","regular","|","1000","top","documents","per","query","|","<","qid",">","<","docid",">","<","rank",">","|","Benchmarking","CodaLab","Worksheet","|","|","#","noqa","|","passage_trec_queries.tsv","|","passage","|","trec","|","200","queries","|","<","qid",">","<","text",">","|","Official","MS","MARCO","Website","|","|","#","noqa","|","passage_trec_qrels.tsv","|","passage","|","trec","|","502,982","query","relations","|","<","qid",">","0","<","docid",">","<","rel",">","|","Official","MS","MARCO","Website","|","(","5",")","|","#","noqa","|","passage_trec_topk.tsv","|","passage","|","trec","|","1000","top","documents","per","query","|","<","qid",">","<","docid",">","<","rank",">","|","Benchmarking","CodaLab","Worksheet","|","|","#","noqa","Notes",":","(","1",")","We","use","a","pre-processed","version","of","the","passage","collection",",","introduced","in","(","MacAvaney",",","et",".","al",".","2021",")",",","which","greatly","improves","the","quality","of","the","passages",".","The","cleaned","collection","is","hosted","on","the","Benchmarking","CodaLab","Worksheet","as","there","is","no","other","reliable","publicly","hosted","copy",".","(","2",")","The","only","relevance","values","is","1",",","which","indicates","that","the","document","with","the","ID","the","docid","is","the","gold","match","for","the","query","with","the","ID","qid",".","(","3",")","The","number","of","top","documents","ranked","was","limited","to","20","for","the","training","set","as","we","only","generate","2","instances","per","training","query",",","one","corresponding","to","a","gold","matching","instance",",","and","the","other","one","corresponding","to","a","probable","non-matching","instance",".","(","4",")","The","labels","(","qrels",")","of","the","official","test","queries","are","not","publicly","released",".","Since","we","need","to","have","access","to","the","qrels","file","to","evaluate","the","models",",","we","instead","use","the","development","set","(","``","queries.dev.small.tsv","''",")",",","which","can","be","found","at","https",":","\/\/msmarco.blob.core.windows.net\/msmarcoranking\/collectionandqueries.tar.gz","(","5",")","The","relevance","values","for","the","TREC","task","of","the","passage","track","can","be","any","of","[","0",",","1",",","2",",","3","]",".","We","consider","[","0",",","1","]","to","be","wrong","matches","and","[","2",",","3","]","to","be","gold","matches",".","IV",".","Baselines","Currently",",","we","use","4","baselines","for","the","MS","MARCO","scenario",",","details","for","which","are","summarized","in","the","table","below",".","Baseline","|","The","baseline","name",".","#","VR","|","Number","of","validation","requests",".","For","each","effective","validation","query",",","multiple","requests","would","be","created",",","governed","by","the","provided","parameters",".","|","Baseline","|","#","VR","|","Parameters","|","regular_topk","|","10,000","|","track=regular",",","use_topk_passages=True",",","valid_topk=50","|","regular_topk_with_qrels","|","10,085","|","track=regular",",","use_qrels_passages=True",",","use_topk_passages=True",",","valid_topk=50","|","trec_topk","|","4300","|","track=trec",",","use_topk_passages=True",",","valid_topk=100","|","trec_qrels","|","9260","|","track=trec",",","use_qrels_passages=True","On","average",",","the","requests","for","the","MS","MARCO","scenario","have","~550","tokens","using","the","GPT-2","tokenizer",".","Multiplying","this","number","with","the","#","VR","column","gives","an","estimate","on","the","number","of","request","tokens","that","would","be","required","to","run","the","given","baseline","on","GPT","models",".","References","(","Bajaj","et",".","al.",",","2016",")","|","https",":","\/\/arxiv.org\/abs\/1611.09268","(","Nogueira","and","Jiang","et",".","al.",",","2020",")","|","https",":","\/\/arxiv.org\/abd\/2003.06713","(","MacAvaney",",","et",".","al",".","2021",")","|","https",":","\/\/arxiv.org\/abs\/2103.02280"]
docstr: Create and return a train instance for the given qid.
  
  References are selected as follows:
      1. We select 1 correct reference, where the documents included
         corresponds to the best document for the given train query.
      2. We create 1 wrong reference, where the document included
         corresponds to a non-gold document for the given train query.
docstr_tok: ["Create","and","return","a","train","instance","for","the","given","qid",".","References","are","selected","as","follows",":","1",".","We","select","1","correct","reference",",","where","the","documents","included","corresponds","to","the","best","document","for","the","given","train","query",".","2",".","We","create","1","wrong","reference",",","where","the","document","included","corresponds","to","a","non-gold","document","for","the","given","train","query","."]
code: def get_train_instance(self, qid: int) -> Instance:
      """Create and return a train instance for the given qid.
  
      References are selected as follows:
          1. We select 1 correct reference, where the documents included
             corresponds to the best document for the given train query.
          2. We create 1 wrong reference, where the document included
             corresponds to a non-gold document for the given train query.
      """
      # Retrieve variables for the split.
      (
          _,
          _,
          qrels_dict,
          topk,
          topk_dict,
          gold_relations,
      ) = self.get_split_variables(TRAIN_SPLIT)
  
      # Get 1 correct Document ID.
      # - Retrieve the Document IDs relevant for the given query.
      # - Sort the retrieved Document IDs by their relevance, from the
      #   highest to the lowest.
      # - Pick the most relevant Document ID as the correct Document ID.
      qrels_docids = list(qrels_dict[qid].keys())
      qrels_docids = sorted(
          qrels_docids,
          key=lambda docid: qrels_dict[qid][docid],
          reverse=True,
      )
      correct_docid = qrels_docids[0]
  
      # Get 1 wrong Document ID.
      # - Retrieve all Document IDs in the top-k dictionary for the query.
      # - Filter the Document IDs to only contain those with ranks between
      #   self.min_train_wrong_topk and self.train_topk, inclusive.
      # - Limit the filtered Document IDs to:
      #   * Those that aren't in the qrels dictionary for the query, or;
      #   * Those that are in the qrels dictionary for the query, granted
      #     that their relation to the query is not in the gold_relations
      #     list.
      # - Select the top Document ID from the filtered list as the wrong one.
      #   This ensures that the wrong documents we picked are competitive with
      #   the correct ones.
      filtered_docids = [
          docid
          for docid, k in topk_dict[qid].items()
          if self.min_train_wrong_topk <= k <= topk
      ]
      wrong_docids = [
          docid
          for docid in filtered_docids
          if docid not in qrels_dict[qid]
          or qrels_dict[qid][docid] not in gold_relations
      ]
      wrong_docid = wrong_docids[0]
  
      # Combine the selected docids in a list, then sort the list following
      # the shuffled list order, which ensures that the order of the correct
      # and wrong references are varied in the prompts. No further
      # modification is needed on the adapter side.
      docids = [correct_docid, wrong_docid]
      docids = sorted(docids, key=self.docid_index_dict.get)  # type: ignore
  
      # Create an instance and return.
      instance = self.create_instance(qid, TRAIN_SPLIT, docids)
      return instance
  
imports: import csv
  import os
  import random
  from collections import defaultdict
  from typing import Dict, List, Tuple, Optional
  from helm.common.general import ensure_file_downloaded, ensure_directory_exists
  from helm.common.hierarchical_logger import hlog
  from .scenario import Scenario, Instance, Reference, TRAIN_SPLIT, VALID_SPLIT, CORRECT_TAG, make_rank_tag, make_relevance_tag, Input, Output
label_desc: Use image node_id 13 for calling the MSMARCOScenario obj's underlying member method code with example usage: obj.get_train_instance(qid) and returns: instance