class_docstr: This class defines methods for loading and iterating over text datasets.
class_docstr_tok: ["This","class","defines","methods","for","loading","and","iterating","over","text","datasets","."]
code: def _get_data(self, path, tokenizer, vocab):
      text = open(path).read()
      tokens = self.get_tokens(text, tokenizer)
  
      # make this a static method
      extra_tokens = len(tokens) % (self.be.bsz * self.seq_length)
      if extra_tokens:
          tokens = tokens[:-extra_tokens]
      self.nbatches = len(tokens) // (self.be.bsz * self.seq_length)
      self.ndata = self.nbatches * self.be.bsz  # no leftovers
  
      self.vocab = sorted(self.get_vocab(tokens, vocab))
      self.nclass = len(self.vocab)
  
      # vocab dicts
      self.token_to_index = dict(
          (t, i) for i, t in enumerate(self.vocab)
      )
      self.index_to_token = dict(
          (i, t) for i, t in enumerate(self.vocab)
      )
  
      # map tokens to indices
      X = np.asarray(
          [self.token_to_index[t] for t in tokens], dtype=np.uint32
      )
      if self.reverse_target:
          y = X.copy()
      else:
          y = np.concatenate((X[1:], X[:1]))
  
      return X, y
  
imports: import logging
  import numpy
  import os
  import h5py
  from neon.data.dataiterator import NervanaDataIterator, ArrayIterator
  from neon.data.datasets import Dataset
  from neon.data.text_preprocessing import pad_sentences, pad_data
label_desc: Use image node_id 2 for calling the Text obj's underlying member method code with example usage: obj._get_data(path, tokenizer, vocab) and returns: X,  y