class_docstr: Configuration of how shards should be created.
  
  Attributes:
    num_shards: number of shards that should be used. If `None`, then the number
      of shards is computed based on the total size of the dataset and the min
      and max shard size.
    min_shard_size: minimum shard size in bytes.
    max_shard_size: maximum shard size in bytes.
    overhead: the amount of overhead when writing a file. By default, it is the
      TFRecord overhead. See
      https://github.com/tensorflow/tensorflow/blob/27325fabed898880fa1b33a04d4b125a6ef4bbc8/tensorflow/core/lib/io/record_writer.h#L104
class_docstr_tok: ["Configuration","of","how","shards","should","be","created",".","Attributes",":","num_shards",":","number","of","shards","that","should","be","used",".","If","`","None","`",",","then","the","number","of","shards","is","computed","based","on","the","total","size","of","the","dataset","and","the","min","and","max","shard","size",".","min_shard_size",":","minimum","shard","size","in","bytes",".","max_shard_size",":","maximum","shard","size","in","bytes",".","overhead",":","the","amount","of","overhead","when","writing","a","file",".","By","default",",","it","is","the","TFRecord","overhead",".","See","https",":","\/\/github.com\/tensorflow\/tensorflow\/blob\/27325fabed898880fa1b33a04d4b125a6ef4bbc8\/tensorflow\/core\/lib\/io\/record_writer.h","#","L104"]
docstr: Returns number of shards for num_examples of total_size in bytes.
  
  Each shard should be at least 128MB.
  A pod has 16*16=256 TPU devices containing 1024 TPU chips (2048 cores).
  So if the dataset is large enough, we want the number of shards to be a
  multiple of 1024, but with shards as big as possible.
  If the dataset is too small, we want the number of shards to be a power
  of two so it distributes better on smaller TPU configs (8, 16, 32, ...
  cores).
  
  Args:
    total_size: the size of the data (serialized, not couting any overhead).
    num_examples: the number of records in the data.
    uses_precise_sharding: whether a mechanism is used to exactly control how
      many examples go in each shard.
  
  Returns:
    number of shards to use.
docstr_tok: ["Returns","number","of","shards","for","num_examples","of","total_size","in","bytes",".","Each","shard","should","be","at","least","128MB",".","A","pod","has","16","*","16=256","TPU","devices","containing","1024","TPU","chips","(","2048","cores",")",".","So","if","the","dataset","is","large","enough",",","we","want","the","number","of","shards","to","be","a","multiple","of","1024",",","but","with","shards","as","big","as","possible",".","If","the","dataset","is","too","small",",","we","want","the","number","of","shards","to","be","a","power","of","two","so","it","distributes","better","on","smaller","TPU","configs","(","8",",","16",",","32",",","...","cores",")",".","Args",":","total_size",":","the","size","of","the","data","(","serialized",",","not","couting","any","overhead",")",".","num_examples",":","the","number","of","records","in","the","data",".","uses_precise_sharding",":","whether","a","mechanism","is","used","to","exactly","control","how","many","examples","go","in","each","shard",".","Returns",":","number","of","shards","to","use","."]
code: def get_number_shards(
      self,
      total_size: int,
      num_examples: int,
      uses_precise_sharding: bool = True,
  ) -> int:
      """Returns number of shards for num_examples of total_size in bytes.
  
      Each shard should be at least 128MB.
      A pod has 16*16=256 TPU devices containing 1024 TPU chips (2048 cores).
      So if the dataset is large enough, we want the number of shards to be a
      multiple of 1024, but with shards as big as possible.
      If the dataset is too small, we want the number of shards to be a power
      of two so it distributes better on smaller TPU configs (8, 16, 32, ...
      cores).
  
      Args:
        total_size: the size of the data (serialized, not couting any overhead).
        num_examples: the number of records in the data.
        uses_precise_sharding: whether a mechanism is used to exactly control how
          many examples go in each shard.
  
      Returns:
        number of shards to use.
      """
      if self.num_shards:
          return self.num_shards
  
      total_size += num_examples * self.overhead
      max_shards_number = total_size // self.min_shard_size
      if uses_precise_sharding:
          max_shard_size = self.max_shard_size
      else:
          # When the pipeline does not control exactly how many rows go into each
          # shard (called 'precise sharding' here), we use a smaller max shard size
          # so that the pipeline doesn't fail if a shard gets some more examples.
          max_shard_size = 0.9 * self.max_shard_size
      min_shards_number = total_size // max_shard_size
      if (
          min_shards_number <= 1024 <= max_shards_number
          and num_examples >= 1024
      ):
          return 1024
      elif min_shards_number > 1024:
          i = 2
          while True:
              n = 1024 * i
              if n >= min_shards_number and num_examples >= n:
                  return n
              i += 1
      else:
          for n in [512, 256, 128, 64, 32, 16, 8, 4, 2]:
              if (
                  min_shards_number <= n <= max_shards_number
                  and num_examples >= n
              ):
                  return n
      return 1
  
file_path: datasets/tensorflow_datasets/core/utils/shard_utils.py
imports: from __future__ import annotations
  import dataclasses
  import math
  import os
  from typing import Any, List, Optional, Sequence
label_desc: Use image node_id 1 for calling the ShardConfig obj's underlying member method code with example usage: obj.get_number_shards(total_size, num_examples, uses_precise_sharding) and returns: int, self, int, n, n