class_docstr: Linear layer execution with asynchronous communication and gradient accumulation fusion in backprop.
code: def forward(
      ctx,
      input: torch.Tensor,
      weight: torch.Tensor,
      bias: Optional[torch.Tensor],
      gradient_accumulation_fusion: bool,
      async_grad_allreduce: bool,
      sequence_parallel_enabled: bool,
      use_16bit_in_wgrad_accum_fusion: Optional[bool] = None,
  ):
      ctx.use_bias = bias is not None and weight.requires_grad
      ctx.gradient_accumulation_fusion = gradient_accumulation_fusion
      ctx.async_grad_allreduce = async_grad_allreduce
      ctx.sequence_parallel_enabled = sequence_parallel_enabled
      ctx.compute_weight_gradient = weight.requires_grad
  
      if use_16bit_in_wgrad_accum_fusion is not None:
          warnings.warn(
              "Deprecated option `use_16bit_in_wgrad_accum_fusion` "
              f"is set to {use_16bit_in_wgrad_accum_fusion}"
          )
  
      if ctx.compute_weight_gradient:
          ctx.save_for_backward(input, weight)
      else:
          ctx.save_for_backward(weight)
  
      if ctx.sequence_parallel_enabled:
          world_size = get_tensor_model_parallel_world_size()
          # `input` is supposed to be 3D and its order of dimension is [sequence, batch, hidden]
          shape = list(input.shape)
          shape[0] *= world_size
  
          all_gather_buffer = torch.empty(
              shape,
              dtype=input.dtype,
              device=torch.cuda.current_device(),
              requires_grad=False,
          )
          torch.distributed.all_gather_into_tensor(
              all_gather_buffer,
              input,
              group=get_tensor_model_parallel_group(),
          )
          total_input = all_gather_buffer
      else:
          total_input = input
      output = torch.matmul(total_input, weight.t())
      if bias is not None:
          output = output + bias
      return output
  
imports: from typing import Optional, Dict, Tuple, List
  import warnings
  import torch
  import torch.nn.functional
  import torch.nn.init
  from torch.nn.parameter import Parameter
  from apex._autocast_utils import _cast_if_autocast_enabled
  from apex.transformer.parallel_state import get_tensor_model_parallel_group
  from apex.transformer.parallel_state import get_tensor_model_parallel_rank
  from apex.transformer.parallel_state import get_tensor_model_parallel_world_size
  from apex.transformer.utils import divide
  from apex.transformer.tensor_parallel.mappings import copy_to_tensor_model_parallel_region
  from apex.transformer.tensor_parallel.mappings import gather_from_tensor_model_parallel_region
  from apex.transformer.tensor_parallel.mappings import reduce_from_tensor_model_parallel_region
  from apex.transformer.tensor_parallel.mappings import scatter_to_tensor_model_parallel_region
  from apex.transformer.tensor_parallel.mappings import reduce_scatter_to_sequence_parallel_region
  from apex.transformer.tensor_parallel.random import get_cuda_rng_tracker
  from apex.transformer.tensor_parallel.utils import VocabUtility
  from apex.transformer.log_util import get_transformer_logger
label_desc: Use image node_id 1 for calling the LinearWithGradAccumulationAndAsyncCommunication obj's underlying member method code with example usage: obj.forward(ctx, input, weight, bias, gradient_accumulation_fusion, async_grad_allreduce, sequence_parallel_enabled, use_16bit_in_wgrad_accum_fusion) and returns: output