class_docstr: Multiheaded Scaled Dot Product Attention
  
  Implements equation:
  MultiHead(Q, K, V) = Concat(head_1,...,head_h)W^O
      where head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
  
  Similarly to the above, d_k = d_v = d_model / h
  
  Inputs
    init:
      nheads : integer # of attention heads
      d_model : model dimensionality
      d_head : dimensionality of a single head
  
    forward:
      query : [batch size, sequence length, d_model]
      key: [batch size, sequence length, d_model]
      value: [batch size, sequence length, d_model]
    unseen_mask: if True, only attend to previous sequence positions
    src_lengths_mask: if True, mask padding based on src_lengths
  
  Output
    result : [batch_size, sequence length, d_model]
class_docstr_tok: ["Multiheaded","Scaled","Dot","Product","Attention","Implements","equation",":","MultiHead","(","Q",",","K",",","V",")","=","Concat","(","head_1",",","...",",","head_h",")","W^O","where","head_i","=","Attention","(","QW_i^Q",",","KW_i^K",",","VW_i^V",")","Similarly","to","the","above",",","d_k","=","d_v","=","d_model","\/","h","Inputs","init",":","nheads",":","integer","#","of","attention","heads","d_model",":","model","dimensionality","d_head",":","dimensionality","of","a","single","head","forward",":","query",":","[","batch","size",",","sequence","length",",","d_model","]","key",":","[","batch","size",",","sequence","length",",","d_model","]","value",":","[","batch","size",",","sequence","length",",","d_model","]","unseen_mask",":","if","True",",","only","attend","to","previous","sequence","positions","src_lengths_mask",":","if","True",",","mask","padding","based","on","src_lengths","Output","result",":","[","batch_size",",","sequence","length",",","d_model","]"]
docstr: Take in model size and number of heads.
docstr_tok: ["Take","in","model","size","and","number","of","heads","."]
code: def __init__(self, nheads, d_model):
      "Take in model size and number of heads."
      super(MultiheadAttention, self).__init__()
      assert d_model % nheads == 0
      self.d_head = d_model // nheads
      self.nheads = nheads
      self.Q_fc = nn.Linear(d_model, d_model, bias=False)
      self.K_fc = nn.Linear(d_model, d_model, bias=False)
      self.V_fc = nn.Linear(d_model, d_model, bias=False)
      self.output_fc = nn.Linear(d_model, d_model, bias=False)
      self.attn = None
  
file_path: translate/pytorch_translate/research/attention/multihead_attention.py
imports: import math
  import numpy
  import torch
  import torch.nn.functional
  from torch import nn
label_desc: Use image node_id 1 to create a new MultiheadAttention object from inherited base classes: nn with example: obj = MultiheadAttention(nheads, d_model)