class_docstr: Iterable dataset that returns constant length chunks of tokens from stream of text files.
      Args:
          tokenizer (Tokenizer): The processor used for processing the data.
          dataset (dataset.Dataset): Dataset with text files.
          seq_length (int): Length of token sequences to return.
code: def __init__(  # pylint: disable=super-init-not-called
      self,
      tokenizer,
      datasets,
      seq_length=2048,
  ):
      self.tokenizer = tokenizer
      self.concat_token_id = tokenizer.eos_token_id
      self.datasets: List[IterableDataset] = datasets
      self.seq_length = seq_length
  
      vocab_size = len(tokenizer.get_vocab())
  
      if vocab_size <= torch.iinfo(torch.int16).max:
          self.tokens_dtype = torch.int16
      elif vocab_size <= torch.iinfo(torch.int32).max:
          self.tokens_dtype = torch.int32
      else:
          self.tokens_dtype = torch.int64
  
file_path: axolotl/src/axolotl/datasets.py
imports: import logging
  import os
  from typing import List, Optional
  import torch
  from datasets import Dataset, IterableDataset
  from .prompt_tokenizers import PromptTokenizingStrategy
label_desc: Use image node_id 1 to create a new ConstantLengthDataset object from inherited base classes: IterableDataset with example: obj = ConstantLengthDataset(tokenizer, datasets, seq_length)