class_docstr: Implements Sparse Self Attention layer of Bert model based on https://github.com/microsoft/DeepSpeedExamples/blob/master/bing_bert/nvidia/modelingpreln.py#L373
  
  For more information please see, TODO DeepSpeed Sparse Transformer.
  
  For usage example please see, TODO DeepSpeed Sparse Transformer Tutorial.
docstr: Initialize the bert sparse self attention layer.
  
  Note) you can use any of the provided sparsity configs or simply add yours!
  
  Arguments:
      config: required: Bert model config
      sparsity_config: optional: this parameter determines sparsity pattern configuration; it is based on FixedSparsityConfig class.
docstr_tok: ["Initialize","the","bert","sparse","self","attention","layer",".","Note",")","you","can","use","any","of","the","provided","sparsity","configs","or","simply","add","yours","!","Arguments",":","config",":","required",":","Bert","model","config","sparsity_config",":","optional",":","this","parameter","determines","sparsity","pattern","configuration",";","it","is","based","on","FixedSparsityConfig","class","."]
code: def __init__(
      self,
      config,
      # SparsityConfig parameters needs to be set accordingly
      sparsity_config=FixedSparsityConfig(num_heads=4),
  ):
      """Initialize the bert sparse self attention layer.
  
      Note) you can use any of the provided sparsity configs or simply add yours!
  
      Arguments:
          config: required: Bert model config
          sparsity_config: optional: this parameter determines sparsity pattern configuration; it is based on FixedSparsityConfig class.
      """
  
      super(BertSparseSelfAttention, self).__init__()
      if config.hidden_size % config.num_attention_heads != 0:
          raise ValueError(
              "The hidden size (%d) is not a multiple of the number of attention "
              "heads (%d)"
              % (config.hidden_size, config.num_attention_heads)
          )
      self.num_attention_heads = config.num_attention_heads
      self.attention_head_size = int(
          config.hidden_size / config.num_attention_heads
      )
      self.all_head_size = (
          self.num_attention_heads * self.attention_head_size
      )
  
      self.query = nn.Linear(config.hidden_size, self.all_head_size)
      self.key = nn.Linear(config.hidden_size, self.all_head_size)
      self.value = nn.Linear(config.hidden_size, self.all_head_size)
  
      self.sparse_self_attention = SparseSelfAttention(sparsity_config)
  
imports: from torch import nn
  from deepspeed.ops.sparse_attention import SparseSelfAttention, FixedSparsityConfig
label_desc: Use image node_id 1 to create a new BertSparseSelfAttention object from inherited base classes: nn with example: obj = BertSparseSelfAttention(config, sparsity_config)