class_docstr: Configuration class to store `BigBird` sparsity configuration.
  For more details about this sparsity config, please see `Big Bird: Transformers for Longer Sequences`: https://arxiv.org/pdf/2007.14062.pdf
  This class extends parent class of `SparsityConfig` and customizes it for `BigBird` sparsity.
docstr: Initialize the BigBird Sparsity Pattern Config.
  
  For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial
  
  Arguments:
       num_heads: required: an integer determining number of attention heads of the layer.
       block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`.
       different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability.
       num_random_blocks: optional: an integer determining the number of random blocks in each block row.
       num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding local attention window.
       num_global_blocks: optional: an integer determining how many consecutive blocks, starting from index 0, are considered as global attention. Global block tokens will be attended by all other block tokens and will attend to all other block tokens as well.
       attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure.
docstr_tok: ["Initialize","the","BigBird","Sparsity","Pattern","Config",".","For","usage","example","please","see",",","TODO","DeepSpeed","Sparse","Transformer","Tutorial","Arguments",":","num_heads",":","required",":","an","integer","determining","number","of","attention","heads","of","the","layer",".","block",":","optional",":","an","integer","determining","the","block","size",".","Current","implementation","of","sparse","self-attention","is","based","on","blocked","sparse","matrices",".","In","which","this","parameter","defines","size","of","such","blocks",",","`","Block","X","Block","`",".","different_layout_per_head",":","optional",":","a","boolean","determining","if","each","head","should","be","assigned","a","different","sparsity","layout",";","default","is","false","and","this","will","be","satisfied","based","on","availability",".","num_random_blocks",":","optional",":","an","integer","determining","the","number","of","random","blocks","in","each","block","row",".","num_sliding_window_blocks",":","optional",":","an","integer","determining","the","number","of","blocks","in","sliding","local","attention","window",".","num_global_blocks",":","optional",":","an","integer","determining","how","many","consecutive","blocks",",","starting","from","index","0",",","are","considered","as","global","attention",".","Global","block","tokens","will","be","attended","by","all","other","block","tokens","and","will","attend","to","all","other","block","tokens","as","well",".","attention",":","optional",":","a","string","determining","attention","type",".","Attention","can","be","`","unidirectional","`",",","such","as","autoregressive","models",",","in","which","tokens","attend","only","to","tokens","appear","before","them","in","the","context",".","Considering","that",",","the","upper","triangular","of","attention","matrix","is","empty","as","above","figure",".","Or","it","can","be","`","bidirectional","`",",","such","as","BERT",",","in","which","tokens","can","attend","to","any","other","tokens","before","or","after","them",".","Then",",","the","upper","triangular","part","of","the","attention","matrix","is","mirror","of","the","lower","triangular","in","the","above","figure","."]
code: def __init__(
      self,
      num_heads,
      block=16,
      different_layout_per_head=False,
      num_random_blocks=1,
      num_sliding_window_blocks=3,
      num_global_blocks=1,
      attention="bidirectional",
  ):
      """Initialize the BigBird Sparsity Pattern Config.
  
      For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial
  
      Arguments:
           num_heads: required: an integer determining number of attention heads of the layer.
           block: optional: an integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines size of such blocks, `Block X Block`.
           different_layout_per_head: optional: a boolean determining if each head should be assigned a different sparsity layout; default is false and this will be satisfied based on availability.
           num_random_blocks: optional: an integer determining the number of random blocks in each block row.
           num_sliding_window_blocks: optional: an integer determining the number of blocks in sliding local attention window.
           num_global_blocks: optional: an integer determining how many consecutive blocks, starting from index 0, are considered as global attention. Global block tokens will be attended by all other block tokens and will attend to all other block tokens as well.
           attention: optional: a string determining attention type. Attention can be `unidirectional`, such as autoregressive models, in which tokens attend only to tokens appear before them in the context. Considering that, the upper triangular of attention matrix is empty as above figure. Or it can be `bidirectional`, such as BERT, in which tokens can attend to any other tokens before or after them. Then, the upper triangular part of the attention matrix is mirror of the lower triangular in the above figure.
      """
  
      super().__init__(num_heads, block, different_layout_per_head)
  
      self.num_random_blocks = num_random_blocks
      self.num_sliding_window_blocks = num_sliding_window_blocks
      self.num_global_blocks = num_global_blocks
  
      if attention != "unidirectional" and attention != "bidirectional":
          raise NotImplementedError(
              'only "uni/bi-directional" attentions are supported for now!'
          )
      self.attention = attention
  
imports: import torch
  import random
label_desc: Use image node_id 1 to create a new BigBirdSparsityConfig object from inherited base classes: SparsityConfig with example: obj = BigBirdSparsityConfig(num_heads, block, different_layout_per_head, num_random_blocks, num_sliding_window_blocks, num_global_blocks, attention)