class_docstr: TransformerEncoderLayer is made up of self-attn and feedforward network.
  This standard encoder layer is based on the paper "Attention Is All You Need".
  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
  Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in
  Neural Information Processing Systems, pages 6000-6010. Users may modify or implement
  in a different way during application.
  
  Args:
      d_model: the number of expected features in the input (required).
      nhead: the number of heads in the multiheadattention models (required).
      dim_feedforward: the dimension of the feedforward network model (default=2048).
      dropout: the dropout value (default=0.1).
      activation: the activation function of the intermediate layer, can be a string
          ("relu" or "gelu") or a unary callable. Default: relu
      layer_norm_eps: the eps value in layer normalization components (default=1e-5).
      norm_first: if ``True``, layer norm is done prior to attention and feedforward
          operations, respectivaly. Otherwise it's done after. Default: ``False`` (after).
      is_moe: if ``True``, the feedforward layer will have MOE enabled.
      num_local_experts: number of local experts for MOE.
  
  
  Examples::
      >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8)
      >>> src = torch.rand(10, 32, 512)
      >>> out = encoder_layer(src)
class_docstr_tok: ["TransformerEncoderLayer","is","made","up","of","self-attn","and","feedforward","network",".","This","standard","encoder","layer","is","based","on","the","paper","``","Attention","Is","All","You","Need","''",".","Ashish","Vaswani",",","Noam","Shazeer",",","Niki","Parmar",",","Jakob","Uszkoreit",",","Llion","Jones",",","Aidan","N","Gomez",",","Lukasz","Kaiser",",","and","Illia","Polosukhin",".","2017",".","Attention","is","all","you","need",".","In","Advances","in","Neural","Information","Processing","Systems",",","pages","6000-6010",".","Users","may","modify","or","implement","in","a","different","way","during","application",".","Args",":","d_model",":","the","number","of","expected","features","in","the","input","(","required",")",".","nhead",":","the","number","of","heads","in","the","multiheadattention","models","(","required",")",".","dim_feedforward",":","the","dimension","of","the","feedforward","network","model","(","default=2048",")",".","dropout",":","the","dropout","value","(","default=0.1",")",".","activation",":","the","activation","function","of","the","intermediate","layer",",","can","be","a","string","(","``","relu","''","or","``","gelu","''",")","or","a","unary","callable",".","Default",":","relu","layer_norm_eps",":","the","eps","value","in","layer","normalization","components","(","default=1e-5",")",".","norm_first",":","if","``","True","``",",","layer","norm","is","done","prior","to","attention","and","feedforward","operations",",","respectivaly",".","Otherwise","it","'s","done","after",".","Default",":","``","False","``","(","after",")",".","is_moe",":","if","``","True","``",",","the","feedforward","layer","will","have","MOE","enabled",".","num_local_experts",":","number","of","local","experts","for","MOE",".","Examples",":",":",">",">",">","encoder_layer","=","nn.TransformerEncoderLayer","(","d_model=512",",","nhead=8",")",">",">",">","src","=","torch.rand","(","10",",","32",",","512",")",">",">",">","out","=","encoder_layer","(","src",")"]
code: def _ff_block(self, x):
      if self.is_moe:
          return self.moe_layer(x)
      else:
          return self.ff_block(x)
  
imports: import math
  import torch
  import torch.nn
  from fairscale.nn.moe.moe_layer import MOELayer
  from fairscale.nn.moe.top2gate import Top2Gate
label_desc: Use image node_id 4 for calling the TransformerEncoderLayer obj's underlying member method code with example usage: obj._ff_block(x) and returns: self, self