class_docstr: A BlendableDataset implementation that uses less memory than the original implementation.
  Indices are computed algorithmically instead of storing them in memory.
  
  To test call: MemoryEfficientBlendableDataset.test_index_blending()
class_docstr_tok: ["A","BlendableDataset","implementation","that","uses","less","memory","than","the","original","implementation",".","Indices","are","computed","algorithmically","instead","of","storing","them","in","memory",".","To","test","call",":","MemoryEfficientBlendableDataset.test_index_blending","(",")"]
code: def __init__(self, datasets, weights, size, weight_bins=100):
      self.datasets = datasets
      num_datasets = len(datasets)
      assert num_datasets == len(weights)
  
      weight_bins = min(weight_bins, size)
  
      self.size = size
      self.weight_bins = weight_bins
  
      # Normalize weights.
      weights = np.array(weights, dtype=np.float64)
      assert (weights > 0.0).all()
      sum_weights = np.sum(weights)
      assert sum_weights > 0.0
      self.weights = weights / sum_weights
  
      # create ds index based on weights
      ds_index = []
      ds_bias = []
      for i, w in enumerate(self.weights):
          n = int(w * weight_bins)
          ds_index.extend([i] * n)
          ds_bias.extend(range(n))
      # make sure arrays have length of weight_bins
      n = weight_bins - len(ds_index)
      ds_index.extend([i] * n)
      ds_bias.extend(range(ds_bias[-1], ds_bias[-1] + n))
  
      self.ds_index = np.array(ds_index, dtype=np.uint32)
      self.ds_index_size = np.array(
          [(self.ds_index == i).sum() for i in range(num_datasets)],
          dtype=np.uint32,
      )
      assert (
          self.ds_index_size > 0
      ).all(), f"Some datasets have no samples in the blendable dataset, increase weight_bins or the offending weight. ds_index_size = {self.ds_index_size}"
      self.ds_bias = np.array(ds_bias, dtype=np.uint32)
  
      self.ds_size = np.array(
          [len(ds) for ds in datasets], dtype=np.uint32
      )
  
file_path: NeMo/nemo/collections/nlp/data/language_modeling/megatron/blendable_dataset.py
imports: import time
  import numpy
  import torch
  from nemo.utils import logging
  from nemo.utils.app_state import AppState
label_desc: Use image node_id 1 to create a new MemoryEfficientBlendableDataset object from inherited base classes: torch.utils.data with example: obj = MemoryEfficientBlendableDataset(datasets, weights, size, weight_bins)