class_docstr: This corpus is designed for situations where you need to train a model
  on matrices, with a large number of iterations. (It should be faster than
  gensim's other IndexedCorpus implementations for this use case; check the
  `benchmark_datasets.py` script. It should also serialize faster.)
  
  The corpus stores its data in separate files called
  "shards". This is a compromise between speed (keeping the whole dataset
  in memory) and memory footprint (keeping the data on disk and reading from
  it on demand). Persistence is done using the standard gensim load/save methods.
  
  .. note::
  
    The dataset is **read-only**, there is - as opposed to gensim's Similarity
    class, which works similarly - no way of adding documents to the dataset
    (for now).
  
  You can use ShardedCorpus to serialize your data just like any other gensim
  corpus that implements serialization. However, because the data is saved
  as numpy 2-dimensional ndarrays (or scipy sparse matrices), you need to
  supply the dimension of your data to the corpus. (The dimension of word
  frequency vectors will typically be the size of the vocabulary, etc.)
  
  .. sourcecode:: pycon
  
      >>> corpus = gensim.utils.mock_data()
      >>> output_prefix = 'mydata.shdat'
      >>> ShardedCorpus.serialize(output_prefix, corpus, dim=1000)
  
  The `output_prefix` tells the ShardedCorpus where to put the data.
  Shards are saved as `output_prefix.0`, `output_prefix.1`, etc.
  All shards must be of the same size. The shards can be re-sized (which
  is essentially a re-serialization into new-size shards), but note that
  this operation will temporarily take twice as much disk space, because
  the old shards are not deleted until the new shards are safely in place.
  
  After serializing the data, the corpus will then save itself to the file
  `output_prefix`.
  
  On further initialization with the same `output_prefix`, the corpus
  will load the already built dataset unless the `overwrite` option is
  given. (A new object is "cloned" from the one saved to `output_prefix`
  previously.)
  
  To retrieve data, you can load the corpus and use it like a list:
  
  .. sourcecode:: pycon
  
      >>> sh_corpus = ShardedCorpus.load(output_prefix)
      >>> batch = sh_corpus[100:150]
  
  This will retrieve a numpy 2-dimensional array of 50 rows and 1000
  columns (1000 was the dimension of the data we supplied to the corpus).
  To retrieve gensim-style sparse vectors, set the `gensim` property:
  
  .. sourcecode:: pycon
  
      >>> sh_corpus.gensim = True
      >>> batch = sh_corpus[100:150]
  
  The batch now will be a generator of gensim vectors.
  
  Since the corpus needs the data serialized in order to be able to operate,
  it will serialize data right away on initialization. Instead of calling
  `ShardedCorpus.serialize()`, you can just initialize and use the corpus
  right away:
  
  .. sourcecode:: pycon
  
      >>> corpus = ShardedCorpus(output_prefix, corpus, dim=1000)
      >>> batch = corpus[100:150]
  
  ShardedCorpus also supports working with scipy sparse matrices, both
  during retrieval and during serialization. If you want to serialize your
  data as sparse matrices, set the `sparse_serialization` flag. For
  retrieving your data as sparse matrices, use the `sparse_retrieval`
  flag. (You can also retrieve densely serialized data as sparse matrices,
  for the sake of completeness, and vice versa.) By default, the corpus
  will retrieve numpy ndarrays even if it was serialized into sparse
  matrices.
  
  .. sourcecode:: pycon
  
      >>> sparse_prefix = 'mydata.sparse.shdat'
      >>> ShardedCorpus.serialize(sparse_prefix, corpus, dim=1000, sparse_serialization=True)
      >>> sparse_corpus = ShardedCorpus.load(sparse_prefix)
      >>> batch = sparse_corpus[100:150]
      >>> type(batch)
      <type 'numpy.ndarray'>
      >>> sparse_corpus.sparse_retrieval = True
      >>> batch = sparse_corpus[100:150]
      <class 'scipy.sparse.csr.csr_matrix'>
  
  While you *can* touch the `sparse_retrieval` attribute during the life
  of a ShardedCorpus object, you should definitely not touch `
  `sharded_serialization`! Changing the attribute will not miraculously
  re-serialize the data in the requested format.
  
  The CSR format is used for sparse data throughout.
  
  Internally, to retrieve data, the dataset keeps track of which shard is
  currently open and on a `__getitem__` request, either returns an item from
  the current shard, or opens a new one. The shard size is constant, except
  for the last shard.
docstr: Add rows of the current shard from `start` to `stop`
  into rows `result_start` to `result_stop` of `s_result`.
  
  Operation is based on the ``self.sparse_serialize`` setting. If the shard
  contents are dense, then s_result is assumed to be an ndarray that
  already supports row indices `result_start:result_stop`. If the shard
  contents are sparse, assumes that s_result has `result_start` rows
  and we should add them up to `result_stop`.
  
  Return the resulting ``s_result``.
code: def __add_to_slice(
      self, s_result, result_start, result_stop, start, stop
  ):
      """
      Add rows of the current shard from `start` to `stop`
      into rows `result_start` to `result_stop` of `s_result`.
  
      Operation is based on the ``self.sparse_serialize`` setting. If the shard
      contents are dense, then s_result is assumed to be an ndarray that
      already supports row indices `result_start:result_stop`. If the shard
      contents are sparse, assumes that s_result has `result_start` rows
      and we should add them up to `result_stop`.
  
      Return the resulting ``s_result``.
  
      """
      if (result_stop - result_start) != (stop - start):
          raise ValueError(
              "Result start/stop range different than stop/start range (%s - %s vs. %s - %s)"
              % (
                  result_start,
                  result_stop,
                  start,
                  stop,
              )
          )
  
      # Dense data: just copy using numpy's slice notation
      if not self.sparse_serialization:
          s_result[result_start:result_stop] = self.current_shard[
              start:stop
          ]
  
          return s_result
  
      # A bit more difficult, we're using a different structure to build the
      # result.
      if s_result.shape != (result_start, self.dim):
          raise ValueError(
              "Assuption about sparse s_result shape invalid: %s expected rows, %s real rows."
              % (
                  result_start,
                  s_result.shape[0],
              )
          )
  
      tmp_matrix = self.current_shard[start:stop]
      s_result = sparse.vstack([s_result, tmp_matrix])
      return s_result
  
imports: from __future__ import print_function
  import logging
  import os
  import math
  import time
  import numpy
  import scipy.sparse
  import gensim
  from gensim.corpora import IndexedCorpus
  from gensim.interfaces import TransformedCorpus
label_desc: Use image node_id 18 for calling the ShardedCorpus obj's underlying member method code with example usage: obj.__add_to_slice(s_result, result_start, result_stop, start, stop) and returns: s_result, s_result