class_docstr: This corpus is designed for situations where you need to train a model
  on matrices, with a large number of iterations. (It should be faster than
  gensim's other IndexedCorpus implementations for this use case; check the
  `benchmark_datasets.py` script. It should also serialize faster.)
  
  The corpus stores its data in separate files called
  "shards". This is a compromise between speed (keeping the whole dataset
  in memory) and memory footprint (keeping the data on disk and reading from
  it on demand). Persistence is done using the standard gensim load/save methods.
  
  .. note::
  
    The dataset is **read-only**, there is - as opposed to gensim's Similarity
    class, which works similarly - no way of adding documents to the dataset
    (for now).
  
  You can use ShardedCorpus to serialize your data just like any other gensim
  corpus that implements serialization. However, because the data is saved
  as numpy 2-dimensional ndarrays (or scipy sparse matrices), you need to
  supply the dimension of your data to the corpus. (The dimension of word
  frequency vectors will typically be the size of the vocabulary, etc.)
  
  .. sourcecode:: pycon
  
      >>> corpus = gensim.utils.mock_data()
      >>> output_prefix = 'mydata.shdat'
      >>> ShardedCorpus.serialize(output_prefix, corpus, dim=1000)
  
  The `output_prefix` tells the ShardedCorpus where to put the data.
  Shards are saved as `output_prefix.0`, `output_prefix.1`, etc.
  All shards must be of the same size. The shards can be re-sized (which
  is essentially a re-serialization into new-size shards), but note that
  this operation will temporarily take twice as much disk space, because
  the old shards are not deleted until the new shards are safely in place.
  
  After serializing the data, the corpus will then save itself to the file
  `output_prefix`.
  
  On further initialization with the same `output_prefix`, the corpus
  will load the already built dataset unless the `overwrite` option is
  given. (A new object is "cloned" from the one saved to `output_prefix`
  previously.)
  
  To retrieve data, you can load the corpus and use it like a list:
  
  .. sourcecode:: pycon
  
      >>> sh_corpus = ShardedCorpus.load(output_prefix)
      >>> batch = sh_corpus[100:150]
  
  This will retrieve a numpy 2-dimensional array of 50 rows and 1000
  columns (1000 was the dimension of the data we supplied to the corpus).
  To retrieve gensim-style sparse vectors, set the `gensim` property:
  
  .. sourcecode:: pycon
  
      >>> sh_corpus.gensim = True
      >>> batch = sh_corpus[100:150]
  
  The batch now will be a generator of gensim vectors.
  
  Since the corpus needs the data serialized in order to be able to operate,
  it will serialize data right away on initialization. Instead of calling
  `ShardedCorpus.serialize()`, you can just initialize and use the corpus
  right away:
  
  .. sourcecode:: pycon
  
      >>> corpus = ShardedCorpus(output_prefix, corpus, dim=1000)
      >>> batch = corpus[100:150]
  
  ShardedCorpus also supports working with scipy sparse matrices, both
  during retrieval and during serialization. If you want to serialize your
  data as sparse matrices, set the `sparse_serialization` flag. For
  retrieving your data as sparse matrices, use the `sparse_retrieval`
  flag. (You can also retrieve densely serialized data as sparse matrices,
  for the sake of completeness, and vice versa.) By default, the corpus
  will retrieve numpy ndarrays even if it was serialized into sparse
  matrices.
  
  .. sourcecode:: pycon
  
      >>> sparse_prefix = 'mydata.sparse.shdat'
      >>> ShardedCorpus.serialize(sparse_prefix, corpus, dim=1000, sparse_serialization=True)
      >>> sparse_corpus = ShardedCorpus.load(sparse_prefix)
      >>> batch = sparse_corpus[100:150]
      >>> type(batch)
      <type 'numpy.ndarray'>
      >>> sparse_corpus.sparse_retrieval = True
      >>> batch = sparse_corpus[100:150]
      <class 'scipy.sparse.csr.csr_matrix'>
  
  While you *can* touch the `sparse_retrieval` attribute during the life
  of a ShardedCorpus object, you should definitely not touch `
  `sharded_serialization`! Changing the attribute will not miraculously
  re-serialize the data in the requested format.
  
  The CSR format is used for sparse data throughout.
  
  Internally, to retrieve data, the dataset keeps track of which shard is
  currently open and on a `__getitem__` request, either returns an item from
  the current shard, or opens a new one. The shard size is constant, except
  for the last shard.
docstr: Initializes the dataset. If `output_prefix` is not found,
  builds the shards.
  
  :type output_prefix: str
  :param output_prefix: The absolute path to the file from which shard
      filenames should be derived. The individual shards will be saved
      as `output_prefix.0`, `output_prefix.1`, etc.
  
      The `output_prefix` path then works as the filename to which
      the ShardedCorpus object itself will be automatically saved.
      Normally, gensim corpora do not do this, but ShardedCorpus needs
      to remember several serialization settings: namely the shard
      size and whether it was serialized in dense or sparse format. By
      saving automatically, any new ShardedCorpus with the same
      `output_prefix` will be able to find the information about the
      data serialized with the given prefix.
  
      If you want to *overwrite* your data serialized with some output
      prefix, set the `overwrite` flag to True.
  
      Of course, you can save your corpus separately as well using
      the `save()` method.
  
  :type corpus: gensim.interfaces.CorpusABC
  :param corpus: The source corpus from which to build the dataset.
  
  :type dim: int
  :param dim: Specify beforehand what the dimension of a dataset item
      should be. This is useful when initializing from a corpus that
      doesn't advertise its dimension, or when it does and you want to
      check that the corpus matches the expected dimension. **If `dim`
      is left unused and `corpus` does not provide its dimension in
      an expected manner, initialization will fail.**
  
  :type shardsize: int
  :param shardsize: How many data points should be in one shard. More
      data per shard means less shard reloading but higher memory usage
      and vice versa.
  
  :type overwrite: bool
  :param overwrite: If set, will build dataset from given corpus even
      if `output_prefix` already exists.
  
  :type sparse_serialization: bool
  :param sparse_serialization: If set, will save the data in a sparse
      form (as csr matrices). This is to speed up retrieval when you
      know you will be using sparse matrices.
  
      ..note::
  
          This property **should not change** during the lifetime of
          the dataset. (If you find out you need to change from a sparse
          to a dense representation, the best practice is to create
          another ShardedCorpus object.)
  
  :type sparse_retrieval: bool
  :param sparse_retrieval: If set, will retrieve data as sparse vectors
      (numpy csr matrices). If unset, will return ndarrays.
  
      Note that retrieval speed for this option depends on how the dataset
      was serialized. If `sparse_serialization` was set, then setting
      `sparse_retrieval` will be faster. However, if the two settings
      do not correspond, the conversion on the fly will slow the dataset
      down.
  
  :type gensim: bool
  :param gensim: If set, will convert the output to gensim
      sparse vectors (list of tuples (id, value)) to make it behave like
      any other gensim corpus. This **will** slow the dataset down.
code: def __init__(
      self,
      output_prefix,
      corpus,
      dim=None,
      shardsize=4096,
      overwrite=False,
      sparse_serialization=False,
      sparse_retrieval=False,
      gensim=False,
  ):
      """Initializes the dataset. If `output_prefix` is not found,
      builds the shards.
  
      :type output_prefix: str
      :param output_prefix: The absolute path to the file from which shard
          filenames should be derived. The individual shards will be saved
          as `output_prefix.0`, `output_prefix.1`, etc.
  
          The `output_prefix` path then works as the filename to which
          the ShardedCorpus object itself will be automatically saved.
          Normally, gensim corpora do not do this, but ShardedCorpus needs
          to remember several serialization settings: namely the shard
          size and whether it was serialized in dense or sparse format. By
          saving automatically, any new ShardedCorpus with the same
          `output_prefix` will be able to find the information about the
          data serialized with the given prefix.
  
          If you want to *overwrite* your data serialized with some output
          prefix, set the `overwrite` flag to True.
  
          Of course, you can save your corpus separately as well using
          the `save()` method.
  
      :type corpus: gensim.interfaces.CorpusABC
      :param corpus: The source corpus from which to build the dataset.
  
      :type dim: int
      :param dim: Specify beforehand what the dimension of a dataset item
          should be. This is useful when initializing from a corpus that
          doesn't advertise its dimension, or when it does and you want to
          check that the corpus matches the expected dimension. **If `dim`
          is left unused and `corpus` does not provide its dimension in
          an expected manner, initialization will fail.**
  
      :type shardsize: int
      :param shardsize: How many data points should be in one shard. More
          data per shard means less shard reloading but higher memory usage
          and vice versa.
  
      :type overwrite: bool
      :param overwrite: If set, will build dataset from given corpus even
          if `output_prefix` already exists.
  
      :type sparse_serialization: bool
      :param sparse_serialization: If set, will save the data in a sparse
          form (as csr matrices). This is to speed up retrieval when you
          know you will be using sparse matrices.
  
          ..note::
  
              This property **should not change** during the lifetime of
              the dataset. (If you find out you need to change from a sparse
              to a dense representation, the best practice is to create
              another ShardedCorpus object.)
  
      :type sparse_retrieval: bool
      :param sparse_retrieval: If set, will retrieve data as sparse vectors
          (numpy csr matrices). If unset, will return ndarrays.
  
          Note that retrieval speed for this option depends on how the dataset
          was serialized. If `sparse_serialization` was set, then setting
          `sparse_retrieval` will be faster. However, if the two settings
          do not correspond, the conversion on the fly will slow the dataset
          down.
  
      :type gensim: bool
      :param gensim: If set, will convert the output to gensim
          sparse vectors (list of tuples (id, value)) to make it behave like
          any other gensim corpus. This **will** slow the dataset down.
  
      """
      self.output_prefix = output_prefix
      self.shardsize = shardsize
  
      self.n_docs = 0
  
      self.offsets = []
      self.n_shards = 0
  
      self.dim = (
          dim  # This number may change during initialization/loading.
      )
  
      # Sparse vs. dense serialization and retrieval.
      self.sparse_serialization = sparse_serialization
      self.sparse_retrieval = sparse_retrieval
      self.gensim = gensim
  
      # The "state" of the dataset.
      self.current_shard = (
          None  # The current shard itself (numpy ndarray)
      )
      self.current_shard_n = (
          None  # Current shard is the current_shard_n-th
      )
      self.current_offset = None  # The index into the dataset which
      # corresponds to index 0 of current shard
  
      logger.info(
          "Initializing sharded corpus with prefix %s", output_prefix
      )
      if (not os.path.isfile(output_prefix)) or overwrite:
          logger.info("Building from corpus...")
          self.init_shards(output_prefix, corpus, shardsize)
  
          # Save automatically, to facilitate re-loading
          # and retain information about how the corpus
          # was serialized.
          logger.info(
              "Saving ShardedCorpus object to %s", self.output_prefix
          )
          self.save()
      else:
          logger.info("Cloning existing...")
          self.init_by_clone()
  
imports: from __future__ import print_function
  import logging
  import os
  import math
  import time
  import numpy
  import scipy.sparse
  import gensim
  from gensim.corpora import IndexedCorpus
  from gensim.interfaces import TransformedCorpus
label_desc: Use image node_id 1 to create a new ShardedCorpus object from inherited base classes: IndexedCorpus with example: obj = ShardedCorpus(output_prefix, corpus, dim, shardsize, overwrite, sparse_serialization, sparse_retrieval, gensim)