class_docstr: SimpleQA: Question-Answering on a list of texts
class_docstr_tok: ["SimpleQA",":","Question-Answering","on","a","list","of","texts"]
docstr: ```
  index documents from list.
  The procs, limitmb, and especially multisegment arguments can be used to
  speed up indexing, if it is too slow.  Please see the whoosh documentation
  for more information on these parameters:  https://whoosh.readthedocs.io/en/latest/batch.html
  Args:
    docs(list): list of strings representing documents
    index_dir(str): path to index directory (see initialize_index)
    commit_every(int): commet after adding this many documents
    breakup_docs(bool): break up documents into smaller paragraphs and treat those as the documents.
                        This can potentially improve the speed at which answers are returned by the ask method
                        when documents being searched are longer.
    procs(int): number of processors
    limitmb(int): memory limit in MB for each process
    multisegment(bool): new segments written instead of merging
    min_words(int):  minimum words for a document (or paragraph extracted from document when breakup_docs=True) to be included in index.
                     Useful for pruning contexts that are unlikely to contain useful answers
    references(list): List of strings containing a reference (e.g., file name) for each document in docs.
                      Each string is treated as a label for the document (e.g., file name, MD5 hash, etc.):
                         Example:  ['some_file.pdf', 'some_other_file,pdf', ...]
                      Strings can also be hyperlinks in which case the label and URL should be separated by a single tab character:
                         Example: ['ktrain_article        https://arxiv.org/pdf/2004.10703v4.pdf', ...]
  
                      These references will be returned in the output of the ask method.
                      If strings are  hyperlinks, then they will automatically be made clickable when the display_answers function
                      displays candidate answers in a pandas DataFRame.
  
                      If references is None, the index of element in docs is used as reference.
  ```
docstr_tok: ["``","`","index","documents","from","list",".","The","procs",",","limitmb",",","and","especially","multisegment","arguments","can","be","used","to","speed","up","indexing",",","if","it","is","too","slow",".","Please","see","the","whoosh","documentation","for","more","information","on","these","parameters",":","https",":","\/\/whoosh.readthedocs.io\/en\/latest\/batch.html","Args",":","docs","(","list",")",":","list","of","strings","representing","documents","index_dir","(","str",")",":","path","to","index","directory","(","see","initialize_index",")","commit_every","(","int",")",":","commet","after","adding","this","many","documents","breakup_docs","(","bool",")",":","break","up","documents","into","smaller","paragraphs","and","treat","those","as","the","documents",".","This","can","potentially","improve","the","speed","at","which","answers","are","returned","by","the","ask","method","when","documents","being","searched","are","longer",".","procs","(","int",")",":","number","of","processors","limitmb","(","int",")",":","memory","limit","in","MB","for","each","process","multisegment","(","bool",")",":","new","segments","written","instead","of","merging","min_words","(","int",")",":","minimum","words","for","a","document","(","or","paragraph","extracted","from","document","when","breakup_docs=True",")","to","be","included","in","index",".","Useful","for","pruning","contexts","that","are","unlikely","to","contain","useful","answers","references","(","list",")",":","List","of","strings","containing","a","reference","(","e.g.",",","file","name",")","for","each","document","in","docs",".","Each","string","is","treated","as","a","label","for","the","document","(","e.g.",",","file","name",",","MD5","hash",",","etc",".",")",":","Example",":","[","'some_file.pdf","'",",","'some_other_file",",","pdf","'",",","...","]","Strings","can","also","be","hyperlinks","in","which","case","the","label","and","URL","should","be","separated","by","a","single","tab","character",":","Example",":","[","'ktrain_article","https",":","\/\/arxiv.org\/pdf\/2004.10703v4.pdf","'",",","...","]","These","references","will","be","returned","in","the","output","of","the","ask","method",".","If","strings","are","hyperlinks",",","then","they","will","automatically","be","made","clickable","when","the","display_answers","function","displays","candidate","answers","in","a","pandas","DataFRame",".","If","references","is","None",",","the","index","of","element","in","docs","is","used","as","reference",".","``","`"]
code: def index_from_list(
      cls,
      docs,
      index_dir,
      commit_every=1024,
      breakup_docs=True,
      procs=1,
      limitmb=256,
      multisegment=False,
      min_words=20,
      references=None,
  ):
      """
      ```
      index documents from list.
      The procs, limitmb, and especially multisegment arguments can be used to
      speed up indexing, if it is too slow.  Please see the whoosh documentation
      for more information on these parameters:  https://whoosh.readthedocs.io/en/latest/batch.html
      Args:
        docs(list): list of strings representing documents
        index_dir(str): path to index directory (see initialize_index)
        commit_every(int): commet after adding this many documents
        breakup_docs(bool): break up documents into smaller paragraphs and treat those as the documents.
                            This can potentially improve the speed at which answers are returned by the ask method
                            when documents being searched are longer.
        procs(int): number of processors
        limitmb(int): memory limit in MB for each process
        multisegment(bool): new segments written instead of merging
        min_words(int):  minimum words for a document (or paragraph extracted from document when breakup_docs=True) to be included in index.
                         Useful for pruning contexts that are unlikely to contain useful answers
        references(list): List of strings containing a reference (e.g., file name) for each document in docs.
                          Each string is treated as a label for the document (e.g., file name, MD5 hash, etc.):
                             Example:  ['some_file.pdf', 'some_other_file,pdf', ...]
                          Strings can also be hyperlinks in which case the label and URL should be separated by a single tab character:
                             Example: ['ktrain_article\thttps://arxiv.org/pdf/2004.10703v4.pdf', ...]
  
                          These references will be returned in the output of the ask method.
                          If strings are  hyperlinks, then they will automatically be made clickable when the display_answers function
                          displays candidate answers in a pandas DataFRame.
  
                          If references is None, the index of element in docs is used as reference.
      ```
      """
      if not isinstance(docs, (np.ndarray, list)):
          raise ValueError("docs must be a list of strings")
      if references is not None and not isinstance(
          references, (np.ndarray, list)
      ):
          raise ValueError("references must be a list of strings")
      if references is not None and len(references) != len(docs):
          raise ValueError(
              "lengths of docs and references must be equal"
          )
  
      ix = index.open_dir(index_dir)
      writer = ix.writer(
          procs=procs, limitmb=limitmb, multisegment=multisegment
      )
  
      mb = master_bar(range(1))
      for i in mb:
          for idx, doc in enumerate(progress_bar(docs, parent=mb)):
              reference = (
                  "%s" % (idx)
                  if references is None
                  else references[idx]
              )
  
              if breakup_docs:
                  small_docs = TU.paragraph_tokenize(
                      doc, join_sentences=True, lang="en"
                  )
                  refs = [reference] * len(small_docs)
                  for i, small_doc in enumerate(small_docs):
                      if len(small_doc.split()) < min_words:
                          continue
                      content = small_doc
                      reference = refs[i]
                      writer.add_document(
                          reference=reference,
                          content=content,
                          rawtext=content,
                      )
              else:
                  if len(doc.split()) < min_words:
                      continue
                  content = doc
                  writer.add_document(
                      reference=reference,
                      content=content,
                      rawtext=content,
                  )
  
              idx += 1
              if idx % commit_every == 0:
                  writer.commit()
                  # writer = ix.writer()
                  writer = ix.writer(
                      procs=procs,
                      limitmb=limitmb,
                      multisegment=multisegment,
                  )
              mb.child.comment = f"indexing documents"
          writer.commit()
          # mb.write(f'Finished indexing documents')
      return
  
imports: from transformers import AutoModelForQuestionAnswering, AutoTokenizer, TFAutoModelForQuestionAnswering
  from whoosh import index, qparser
  from whoosh.fields import *
  from whoosh.qparser import QueryParser
  from ...None import utils
  from ...imports import *
  from ...torch_base import TorchBase
  from ..None import preprocessor
  from ..None import textutils
  from itertools import chain, zip_longest
label_desc: Use image node_id 4 for calling the SimpleQA obj's underlying member method code with example usage: obj.index_from_list(cls, docs, index_dir, commit_every, breakup_docs, procs, limitmb, multisegment, min_words, references) without return types