class_docstr: Transformer encoder block.
  
  From @myleott -
  There are at least three common structures.
  1) Attention is all you need had the worst one, where the layernorm came after each
      block and was in the residual path.
  2) BERT improved upon this by moving the layernorm to the beginning of each block
      (and adding an extra layernorm at the end).
  3) There's a further improved version that also moves the layernorm outside of the
      residual path, which is what this implementation does.
  
  Figure 1 of this paper compares versions 1 and 3:
      https://openreview.net/pdf?id=B1x8anVFPr
  Figure 7 of this paper compares versions 2 and 3 for BERT:
      https://arxiv.org/abs/1909.08053
class_docstr_tok: ["Transformer","encoder","block",".","From","@","myleott","-","There","are","at","least","three","common","structures",".","1",")","Attention","is","all","you","need","had","the","worst","one",",","where","the","layernorm","came","after","each","block","and","was","in","the","residual","path",".","2",")","BERT","improved","upon","this","by","moving","the","layernorm","to","the","beginning","of","each","block","(","and","adding","an","extra","layernorm","at","the","end",")",".","3",")","There","'s","a","further","improved","version","that","also","moves","the","layernorm","outside","of","the","residual","path",",","which","is","what","this","implementation","does",".","Figure","1","of","this","paper","compares","versions","1","and","3",":","https",":","\/\/openreview.net\/pdf","?","id=B1x8anVFPr","Figure","7","of","this","paper","compares","versions","2","and","3","for","BERT",":","https",":","\/\/arxiv.org\/abs\/1909.08053"]
code: def flops(self, x):
      flops = 0
      seq_len, batch_size, hidden_dim = x.shape
  
      num_elems = x.numel() // batch_size
      flops += num_elems * 6  # ln_1 (* 2), x + input, ln_2 (* 2), x + y
  
      # self_attention
      # calculations are based on the fact that head_dim * num_heads = hidden_dim
      # so we collapse (hidden_dim // num_heads) * num_heads to hidden_dim
      flops += (
          3 * seq_len * (hidden_dim + 1) * hidden_dim
      )  # projection with bias
      flops += hidden_dim * seq_len  # scaling
      flops += hidden_dim * seq_len * seq_len  # attention weights
      flops += self.num_heads * seq_len * seq_len  # softmax
      flops += hidden_dim * seq_len * seq_len  # attention application
      flops += (
          seq_len * (hidden_dim + 1) * hidden_dim
      )  # out projection with bias
  
      # mlp
      mlp_dim = self.mlp.linear_1.out_features
      flops += seq_len * (hidden_dim + 1) * mlp_dim  # linear_1
      flops += seq_len * mlp_dim  # act
      flops += seq_len * (mlp_dim + 1) * hidden_dim  # linear_2
      return flops * batch_size
  
imports: import copy
  import logging
  import math
  from collections import OrderedDict
  from functools import partial
  from typing import Dict, List, Mapping, NamedTuple, Union
  import torch
  import torch.nn
  from classy_vision.models import ClassyModel, register_model
  from .lecun_normal_init import lecun_normal_init
label_desc: Use image node_id 3 for calling the EncoderBlock obj's underlying member method code with example usage: obj.flops(x) and returns: unknown