class_docstr: Represents dataset with associated metadata.
  
  Holds all information about one split of a dataset (e.g. training set).
  
  Attributes:
      fields (dict): A mapping from attribute names (e.g. "left_address") to
          corresponding :class:`~data.MatchingField` objects that specify how to process
          the field.
      examples (list): A list containing all the examples (labeled tuple pairs) in this
          dataset.
      metadata (dict): Metadata about the dataset (e.g. word probabilities).
          See :meth:`~data.MatchingDataset.compute_metadata` for details.
      corresponding_field (dict): A mapping from left table attribute names
          (e.g. "left_address") to corresponding right table attribute names
          (e.g. "right_address") and vice versa.
      text_fields (dict): A mapping from canonical attribute names (e.g. "address") to
          tuples of the corresponding left and right attribute names
          (e.g. ("left_address", "right_address")).
      all_left_fields (list): A list of all left table attribute names.
      all_right_fields (list): A list of all right table attribute names.
      canonical_text_fields (list): A list of all canonical attribute names.
      label_field (str): Name of the column containing labels.
      id_field (str): Name of the column containing tuple pair ids.
class_docstr_tok: ["Represents","dataset","with","associated","metadata",".","Holds","all","information","about","one","split","of","a","dataset","(","e.g",".","training","set",")",".","Attributes",":","fields","(","dict",")",":","A","mapping","from","attribute","names","(","e.g",".","``","left_address","''",")","to","corresponding",":","class",":","`","~data.MatchingField","`","objects","that","specify","how","to","process","the","field",".","examples","(","list",")",":","A","list","containing","all","the","examples","(","labeled","tuple","pairs",")","in","this","dataset",".","metadata","(","dict",")",":","Metadata","about","the","dataset","(","e.g",".","word","probabilities",")",".","See",":","meth",":","`","~data.MatchingDataset.compute_metadata","`","for","details",".","corresponding_field","(","dict",")",":","A","mapping","from","left","table","attribute","names","(","e.g",".","``","left_address","''",")","to","corresponding","right","table","attribute","names","(","e.g",".","``","right_address","''",")","and","vice","versa",".","text_fields","(","dict",")",":","A","mapping","from","canonical","attribute","names","(","e.g",".","``","address","''",")","to","tuples","of","the","corresponding","left","and","right","attribute","names","(","e.g",".","(","``","left_address","''",",","``","right_address","''",")",")",".","all_left_fields","(","list",")",":","A","list","of","all","left","table","attribute","names",".","all_right_fields","(","list",")",":","A","list","of","all","right","table","attribute","names",".","canonical_text_fields","(","list",")",":","A","list","of","all","canonical","attribute","names",".","label_field","(","str",")",":","Name","of","the","column","containing","labels",".","id_field","(","str",")",":","Name","of","the","column","containing","tuple","pair","ids","."]
docstr: Computes metadata about the dataset.
  
  Computes the following metadata about the dataset:
  
  * ``word_probs``: For each attribute in the dataset, a mapping from words to
    word (token) probabilities.
  * ``totals``: For each attribute in the dataset, a count of the total number of
    words present in all attribute examples.
  * ``pc``: For each attribute in the dataset, the first principal component of the
    sequence embeddings for all values of that attribute. The sequence embedding of
    an attribute value is computed by taking the weighted average of its word
    embeddings, where the weight is the soft inverse word probability. Refer
    `Arora et al. (2017) <https://openreview.net/pdf?id=SyK00v5xx>`__ for details.
  
  Arguments:
      pca (bool): Whether to compute the ``pc`` metadata.
docstr_tok: ["Computes","metadata","about","the","dataset",".","Computes","the","following","metadata","about","the","dataset",":","*","``","word_probs","``",":","For","each","attribute","in","the","dataset",",","a","mapping","from","words","to","word","(","token",")","probabilities",".","*","``","totals","``",":","For","each","attribute","in","the","dataset",",","a","count","of","the","total","number","of","words","present","in","all","attribute","examples",".","*","``","pc","``",":","For","each","attribute","in","the","dataset",",","the","first","principal","component","of","the","sequence","embeddings","for","all","values","of","that","attribute",".","The","sequence","embedding","of","an","attribute","value","is","computed","by","taking","the","weighted","average","of","its","word","embeddings",",","where","the","weight","is","the","soft","inverse","word","probability",".","Refer","`","Arora","et","al",".","(","2017",")","<","https",":","\/\/openreview.net\/pdf","?","id=SyK00v5xx",">","`","__","for","details",".","Arguments",":","pca","(","bool",")",":","Whether","to","compute","the","``","pc","``","metadata","."]
code: def compute_metadata(self, pca=False):
      r"""Computes metadata about the dataset.
  
      Computes the following metadata about the dataset:
  
      * ``word_probs``: For each attribute in the dataset, a mapping from words to
        word (token) probabilities.
      * ``totals``: For each attribute in the dataset, a count of the total number of
        words present in all attribute examples.
      * ``pc``: For each attribute in the dataset, the first principal component of the
        sequence embeddings for all values of that attribute. The sequence embedding of
        an attribute value is computed by taking the weighted average of its word
        embeddings, where the weight is the soft inverse word probability. Refer
        `Arora et al. (2017) <https://openreview.net/pdf?id=SyK00v5xx>`__ for details.
  
      Arguments:
          pca (bool): Whether to compute the ``pc`` metadata.
      """
      self.metadata = {}
  
      # Create an iterator over the entire dataset.
      train_iter = MatchingIterator(
          self,
          self,
          train=False,
          batch_size=1024,
          device="cpu",
          sort_in_buckets=False,
      )
      counter = defaultdict(Counter)
  
      # For each attribute, find the number of times each word id occurs in the dataset.
      # Note that word ids here also include ``UNK`` tokens, padding tokens, etc.
      for batch in pyprind.prog_bar(
          train_iter, title="\nBuilding vocabulary"
      ):
          for name in self.all_text_fields:
              attr_input = getattr(batch, name)
              counter[name].update(
                  attr_input.data.data.view(-1).tolist()
              )
  
      word_probs = {}
      totals = {}
      for name in self.all_text_fields:
          attr_counter = counter[name]
          total = sum(attr_counter.values())
          totals[name] = total
  
          field_word_probs = {}
          for word, freq in attr_counter.items():
              field_word_probs[word] = freq / total
          word_probs[name] = field_word_probs
      self.metadata["word_probs"] = word_probs
      self.metadata["totals"] = totals
  
      if not pca:
          return
  
      # To compute principal components, we need to compute weighted sequence embeddings
      # for each attribute. To do so, for each attribute, we first construct a neural
      # network to compute word embeddings and take their weighted average.
      field_embed = {}
      embed = {}
      inv_freq_pool = Pool("inv-freq-avg")
      for name in self.all_text_fields:
          field = self.fields[name]
          if field not in field_embed:
              vectors_size = field.vocab.vectors.shape
              embed_layer = nn.Embedding(
                  vectors_size[0], vectors_size[1]
              )
              embed_layer.weight.data.copy_(field.vocab.vectors)
              embed_layer.weight.requires_grad = False
              field_embed[field] = NoMeta(embed_layer)
          embed[name] = field_embed[field]
  
      # Create an iterator over the entire dataset.
      train_iter = MatchingIterator(
          self,
          self,
          train=False,
          batch_size=1024,
          device="cpu",
          sort_in_buckets=False,
      )
      attr_embeddings = defaultdict(list)
  
      # Run the constructed neural network to compute weighted sequence embeddings
      # for each attribute of each example in the dataset.
      for batch in pyprind.prog_bar(
          train_iter, title="\nComputing principal components"
      ):
          for name in self.all_text_fields:
              attr_input = getattr(batch, name)
              embeddings = inv_freq_pool(embed[name](attr_input))
              attr_embeddings[name].append(embeddings.data.data)
  
      # Compute the first principal component of weighted sequence embeddings for each
      # attribute.
      pc = {}
      for name in self.all_text_fields:
          concatenated = torch.cat(attr_embeddings[name])
          svd = TruncatedSVD(n_components=1, n_iter=7)
          svd.fit(concatenated.numpy())
          pc[name] = svd.components_[0]
      self.metadata["pc"] = pc
  
imports: from __future__ import division
  import copy
  import logging
  import os
  import pdb
  from collections import Counter, defaultdict
  from timeit import default_timer
  import pandas
  import pyprind
  import six
  from sklearn.decomposition import TruncatedSVD
  import torch
  import torch.nn
  from torchtext.legacy import data
  from torchtext.legacy.data.example import Example
  from torchtext.utils import unicode_csv_reader
  from ..models.modules import NoMeta, Pool
  from .field import MatchingField
  from .iterator import MatchingIterator
label_desc: Use image node_id 3 for calling the MatchingDataset obj's underlying member method code with example usage: obj.compute_metadata(pca) without return types