class_docstr: Break a Dataset of tokens into sentence pair blocks for next sentence
     prediction as well as masked language model.
  
     High-level logics are:
     1. break input tensor to tensor blocks
     2. pair the blocks with 50% next sentence and 50% random sentence
     3. return paired blocks as well as related segment labels
  
  Args:
      dataset (~torch.utils.data.Dataset): dataset to break into blocks
      sizes: array of sentence lengths
      dictionary: dictionary for the task
      block_size: maximum block size
      break_mode: mode for breaking copurs into block pairs. currently we support
          2 modes
          doc: respect document boundaries and each part of the pair should belong to on document
          none: don't respect any boundary and cut tokens evenly
      short_seq_prob: probability for generating shorter block pairs
      doc_break_size: Size for empty line separating documents. Typically 1 if
                      the sentences have eos, 0 otherwise.
class_docstr_tok: ["Break","a","Dataset","of","tokens","into","sentence","pair","blocks","for","next","sentence","prediction","as","well","as","masked","language","model",".","High-level","logics","are",":","1.","break","input","tensor","to","tensor","blocks","2.","pair","the","blocks","with","50","%","next","sentence","and","50","%","random","sentence","3.","return","paired","blocks","as","well","as","related","segment","labels","Args",":","dataset","(","~torch.utils.data.Dataset",")",":","dataset","to","break","into","blocks","sizes",":","array","of","sentence","lengths","dictionary",":","dictionary","for","the","task","block_size",":","maximum","block","size","break_mode",":","mode","for","breaking","copurs","into","block","pairs",".","currently","we","support","2","modes","doc",":","respect","document","boundaries","and","each","part","of","the","pair","should","belong","to","on","document","none",":","do","n't","respect","any","boundary","and","cut","tokens","evenly","short_seq_prob",":","probability","for","generating","shorter","block","pairs","doc_break_size",":","Size","for","empty","line","separating","documents",".","Typically","1","if","the","sentences","have","eos",",","0","otherwise","."]
docstr: Build index mapping block indices to the underlying dataset indices
docstr_tok: ["Build","index","mapping","block","indices","to","the","underlying","dataset","indices"]
code: def _sent_to_dataset_index(self, sent_sizes):
      """
      Build index mapping block indices to the underlying dataset indices
      """
      dataset_index = []
      ds_idx, ds_remaining = -1, 0
      for to_consume in sent_sizes:
          sent_size = to_consume
          if ds_remaining == 0:
              ds_idx += 1
              ds_remaining = sent_sizes[ds_idx]
          start_ds_idx = ds_idx
          start_offset = sent_sizes[ds_idx] - ds_remaining
          while to_consume > ds_remaining:
              to_consume -= ds_remaining
              ds_idx += 1
              ds_remaining = sent_sizes[ds_idx]
          ds_remaining -= to_consume
          dataset_index.append(
              (
                  start_ds_idx,  # starting index in dataset
                  start_offset,  # starting offset within starting index
                  ds_idx,  # ending index in dataset
                  sent_size,  # sentence length
              )
          )
      assert ds_remaining == 0
      assert ds_idx == len(self.dataset) - 1
      return dataset_index
  
file_path: fairseq/fairseq/data/legacy/block_pair_dataset.py
imports: import math
  import numpy
  import torch
  from fairseq.data import FairseqDataset
label_desc: Use image node_id 3 for calling the BlockPairDataset obj's underlying member method code with example usage: obj._sent_to_dataset_index(sent_sizes) and returns: dataset_index