class_docstr: Base information retrieval dataset on which other datasets are built.
  
  Args:
      tokenizer: tokenizer
      max_query_length: maximum length of query in tokens
      max_passage_length: maximum length of passage in tokens
class_docstr_tok: ["Base","information","retrieval","dataset","on","which","other","datasets","are","built",".","Args",":","tokenizer",":","tokenizer","max_query_length",":","maximum","length","of","query","in","tokens","max_passage_length",":","maximum","length","of","passage","in","tokens"]
docstr: Function which pre-tokenizes the dataset.
docstr_tok: ["Function","which","pre-tokenizes","the","dataset","."]
code: def tokenize_dataset(self, file, max_seq_length):
      """
      Function which pre-tokenizes the dataset.
      """
      lines = open(file, "r").readlines()
      with mp.Pool() as pool:
          dataset_dict = pool.map(self.preprocess_line, lines)
      dataset_dict = {
          id_: tokens[:max_seq_length] for (id_, tokens) in dataset_dict
      }
      return dataset_dict
  
imports: import multiprocessing
  import os
  import pickle
  import random
  from typing import Optional
  import numpy
  from torch.utils.data import Dataset
  from nemo.collections.common.tokenizers.tokenizer_spec import TokenizerSpec
label_desc: Use image node_id 4 for calling the BaseInformationRetrievalDataset obj's underlying member method code with example usage: obj.tokenize_dataset(file, max_seq_length) and returns: dataset_dict