class_docstr: This class defines methods for loading and iterating over text datasets.
class_docstr_tok: ["This","class","defines","methods","for","loading","and","iterating","over","text","datasets","."]
docstr: Construct vocabulary from the given tokens.
  
  Arguments:
      tokens(list): List of tokens.
      vocab:  (Default value = None)
  
  Returns:
      python.set : A set of unique tokens
docstr_tok: ["Construct","vocabulary","from","the","given","tokens",".","Arguments",":","tokens","(","list",")",":","List","of","tokens",".","vocab",":","(","Default","value","=","None",")","Returns",":","python.set",":","A","set","of","unique","tokens"]
code: def get_vocab(tokens, vocab=None):
      """
      Construct vocabulary from the given tokens.
  
      Arguments:
          tokens(list): List of tokens.
          vocab:  (Default value = None)
  
      Returns:
          python.set : A set of unique tokens
      """
      # (if vocab is not None, we check that it contains all tokens)
      if vocab is None:
          return set(tokens)
      else:
          vocab = set(vocab)
          assert vocab >= set(
              tokens
          ), "the predefined vocab must contain all the tokens"
          return vocab
  
imports: import logging
  import numpy
  import os
  import h5py
  from neon.data.dataiterator import NervanaDataIterator, ArrayIterator
  from neon.data.datasets import Dataset
  from neon.data.text_preprocessing import pad_sentences, pad_data
label_desc: Use image node_id 5 for calling the Text obj's underlying member method code with example usage: obj.get_vocab(tokens, vocab) and returns: set, vocab