class_docstr: A slight modification of torch.nn.TransformerEncoderLayer which allows for torch.jit scripting
class_docstr_tok: ["A","slight","modification","of","torch.nn.TransformerEncoderLayer","which","allows","for","torch.jit","scripting"]
code: def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1):
      super().__init__()
      self.self_attn = nn.MultiheadAttention(
          d_model, nhead, dropout=dropout
      )
      # Implementation of Feedforward model
      self.linear1 = nn.Linear(d_model, dim_feedforward)
      self.dropout = nn.Dropout(dropout)
      self.linear2 = nn.Linear(dim_feedforward, d_model)
  
      self.norm1 = nn.LayerNorm(d_model)
      self.norm2 = nn.LayerNorm(d_model)
      self.dropout1 = nn.Dropout(dropout)
      self.dropout2 = nn.Dropout(dropout)
  
      self.activation = gelu_fast
  
imports: import time
  import torch
  from torch import nn
  from hivemind.moe.server.layers.custom_experts import register_expert_class
label_desc: Use image node_id 1 to create a new TransformerEncoderLayer object from inherited base classes: nn with example: obj = TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout)