class_docstr: Break a Dataset of tokens into sentence pair blocks for next sentence
     prediction as well as masked language model.
  
     High-level logics are:
     1. break input tensor to tensor blocks
     2. pair the blocks with 50% next sentence and 50% random sentence
     3. return paired blocks as well as related segment labels
  
  Args:
      dataset (~torch.utils.data.Dataset): dataset to break into blocks
      sizes: array of sentence lengths
      dictionary: dictionary for the task
      block_size: maximum block size
      break_mode: mode for breaking copurs into block pairs. currently we support
          2 modes
          doc: respect document boundaries and each part of the pair should belong to on document
          none: don't respect any boundary and cut tokens evenly
      short_seq_prob: probability for generating shorter block pairs
      doc_break_size: Size for empty line separating documents. Typically 1 if
                      the sentences have eos, 0 otherwise.
class_docstr_tok: ["Break","a","Dataset","of","tokens","into","sentence","pair","blocks","for","next","sentence","prediction","as","well","as","masked","language","model",".","High-level","logics","are",":","1.","break","input","tensor","to","tensor","blocks","2.","pair","the","blocks","with","50","%","next","sentence","and","50","%","random","sentence","3.","return","paired","blocks","as","well","as","related","segment","labels","Args",":","dataset","(","~torch.utils.data.Dataset",")",":","dataset","to","break","into","blocks","sizes",":","array","of","sentence","lengths","dictionary",":","dictionary","for","the","task","block_size",":","maximum","block","size","break_mode",":","mode","for","breaking","copurs","into","block","pairs",".","currently","we","support","2","modes","doc",":","respect","document","boundaries","and","each","part","of","the","pair","should","belong","to","on","document","none",":","do","n't","respect","any","boundary","and","cut","tokens","evenly","short_seq_prob",":","probability","for","generating","shorter","block","pairs","doc_break_size",":","Size","for","empty","line","separating","documents",".","Typically","1","if","the","sentences","have","eos",",","0","otherwise","."]
docstr: Give a list of evenly cut blocks/sentences, pair these sentences with 50%
  consecutive sentences and 50% random sentences.
  This is used for none break mode
docstr_tok: ["Give","a","list","of","evenly","cut","blocks\/sentences",",","pair","these","sentences","with","50","%","consecutive","sentences","and","50","%","random","sentences",".","This","is","used","for","none","break","mode"]
code: def _pair_sentences(self, dataset_index):
      """
      Give a list of evenly cut blocks/sentences, pair these sentences with 50%
      consecutive sentences and 50% random sentences.
      This is used for none break mode
      """
      # pair sentences
      for sent_id, sent in enumerate(dataset_index):
          next_sent_label = (
              1
              if np.random.rand() > 0.5
              and sent_id != len(dataset_index) - 1
              else 0
          )
          if next_sent_label:
              next_sent = dataset_index[sent_id + 1]
          else:
              next_sent = dataset_index[
                  self._skip_sampling(
                      len(dataset_index), [sent_id, sent_id + 1]
                  )
              ]
          self.sent_pairs.append((sent, next_sent, next_sent_label))
  
          # The current blocks don't include the special tokens but the
          # sizes already account for this
          self.sizes.append(3 + sent[3] + next_sent[3])
  
file_path: fairseq/fairseq/data/legacy/block_pair_dataset.py
imports: import math
  import numpy
  import torch
  from fairseq.data import FairseqDataset
label_desc: Use image node_id 2 for calling the BlockPairDataset obj's underlying member method code with example usage: obj._pair_sentences(dataset_index) without return types