class_docstr: Encoder layer implemented by fairseq.
  This version only removes the "args" parameter, no other changes
  
  In the original paper each operation (multi-head attention or FFN) is
  postprocessed with: `dropout -> add residual -> layernorm`.
  In the tensor2tensor code they suggest that learning is more robust when
  preprocessing each layer with layernorm and postprocessing with:
  `dropout -> add residual`. We default to the approach in the paper, but the
  tensor2tensor approach can be enabled by setting
  normalize_before to True.
class_docstr_tok: ["Encoder","layer","implemented","by","fairseq",".","This","version","only","removes","the","``","args","''","parameter",",","no","other","changes","In","the","original","paper","each","operation","(","multi-head","attention","or","FFN",")","is","postprocessed","with",":","`","dropout","-",">","add","residual","-",">","layernorm","`",".","In","the","tensor2tensor","code","they","suggest","that","learning","is","more","robust","when","preprocessing","each","layer","with","layernorm","and","postprocessing","with",":","`","dropout","-",">","add","residual","`",".","We","default","to","the","approach","in","the","paper",",","but","the","tensor2tensor","approach","can","be","enabled","by","setting","normalize_before","to","True","."]
docstr: Args:
      x (Tensor): input to the layer of shape `(batch, seq_len, embed_dim)`
      encoder_padding_mask (ByteTensor): binary ByteTensor of shape
          `(batch, seq_len)` where padding elements are indicated by ``1``.
  
  
  Returns:
      encoded output of shape `(batch, seq_len, embed_dim)`
code: def forward(self, x, encoder_padding_mask):
      """
      Args:
          x (Tensor): input to the layer of shape `(batch, seq_len, embed_dim)`
          encoder_padding_mask (ByteTensor): binary ByteTensor of shape
              `(batch, seq_len)` where padding elements are indicated by ``1``.
  
  
      Returns:
          encoded output of shape `(batch, seq_len, embed_dim)`
      """
  
      # anything in original attn_mask = 1, becomes -1e8
      # anything in original attn_mask = 0, becomes 0
      # Note that we cannot use -inf here, because at some edge cases,
      # the attention weight (before softmax) for some padded element in query
      # will become -inf, which results in NaN in model parameters
  
      x = x.transpose(0, 1)
  
      residual = x
      if self.normalize_before:
          x = self.self_attn_layer_norm(x)
      x, _ = self.self_attn(
          query=x,
          key=x,
          value=x,
          key_padding_mask=encoder_padding_mask,
      )
      x = self.dropout_module(x)
      x = self.residual_connection(x, residual)
      if not self.normalize_before:
          x = self.self_attn_layer_norm(x)
  
      residual = x
      if self.normalize_before:
          x = self.final_layer_norm(x)
  
      x = self.activation_fn(self.fc1(x))
  
      x = self.activation_dropout_module(x)
      x = self.fc2(x)
      x = self.dropout_module(x)
      x = self.residual_connection(x, residual)
      if not self.normalize_before:
          x = self.final_layer_norm(x)
  
      x = x.transpose(0, 1)
  
      return x
  
imports: import __init__
  import math
  import uuid
  from typing import Dict, Optional, List
  import torch
  from torch import Tensor, nn
  from torch.nn import Parameter, LayerNorm, Dropout
  from csrc.pytorch import util
  from csrc.pytorch.layer_base import TransformerEmbeddingLayerBase, TransformerEncoderLayerBase, TransformerDecoderLayerBase
  from csrc.pytorch.quantization import QuantLinear, TensorQuantizer, act_quant_config, weight_quant_config, out_quant_config, emb_quant_config
label_desc: Use image node_id 5 for calling the TransformerEncoderLayer obj's underlying member method code with example usage: obj.forward(x, encoder_padding_mask) and returns: x