class_docstr: Linear layer with row parallelism.
  
  The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its second dimension. A = transpose([A_1 .. A_p]) X = [X_1, ..., X_p]
  
  Args:
      input_size: first dimension of matrix A.
      output_size: second dimension of matrix A.
      bias: If true, add bias. Note that bias is not parallelized.
      input_is_parallel: If true, we assume that the input is already split across the GPUs and we do not split again.
      init_method: method to initialize weights. Note that bias is always set to zero.
      stride: For the strided linear layers.
      keep_master_weight_for_test: This was added for testing and should be set to False. It returns the master weights used for initialization.
      skip_bias_add: If True, do not add the bias term, instead return it to be added by the caller. This enables performance optimations where bias can be fused with other elementwise operations.
      is_expert: If True, the layer is treated as an MoE expert layer
      tp_comm_buffer_name: Communication buffer name. Not used in
                           non-Transformer-Engine modules.
      config: ModelParallelConfig object
docstr: Forward of RowParallelLinear
  
  Args:
      input_: 3D tensor whose order of dimension is [sequence, batch, hidden]
  
  Returns:
      - output
      - bias
docstr_tok: ["Forward","of","RowParallelLinear","Args",":","input_",":","3D","tensor","whose","order","of","dimension","is","[","sequence",",","batch",",","hidden","]","Returns",":","-","output","-","bias"]
code: def forward(self, input_):
      """Forward of RowParallelLinear
  
      Args:
          input_: 3D tensor whose order of dimension is [sequence, batch, hidden]
  
      Returns:
          - output
          - bias
      """
      # Set up backprop all-reduce.
      if self.input_is_parallel:
          input_parallel = input_
      else:
          assert not self.sequence_parallel
          input_parallel = scatter_to_tensor_model_parallel_region(
              input_
          )
      # Matrix multiply.
      if not self.weight.requires_grad:
          self._forward_impl = linear_with_frozen_weight
      else:
          self._forward_impl = (
              linear_with_grad_accumulation_and_async_allreduce
          )
      output_parallel = self._forward_impl(
          input=input_parallel,
          weight=self.weight,
          bias=None,
          gradient_accumulation_fusion=self.gradient_accumulation_fusion,
          async_grad_allreduce=False,
          sequence_parallel=False,
      )
  
      # All-reduce across all the partitions.
      if self.explicit_expert_comm:
          assert self.skip_bias_add
          output_ = output_parallel
      elif self.sequence_parallel:
          output_ = reduce_scatter_to_sequence_parallel_region(
              output_parallel
          )
      else:
          output_ = reduce_from_tensor_model_parallel_region(
              output_parallel
          )
      if not self.skip_bias_add:
          output = (
              (output_ + self.bias)
              if self.bias is not None
              else output_
          )
          output_bias = None
      else:
          output = output_
          output_bias = self.bias
      return output, output_bias
  
imports: import math
  import os
  import warnings
  from typing import Callable, Optional
  import torch
  import torch.nn.functional
  import torch.nn.init
  from torch.cuda.amp import custom_bwd, custom_fwd
  from torch.nn.parameter import Parameter
  from megatron.core.model_parallel_config import ModelParallelConfig
  from megatron.core.parallel_state import get_global_memory_buffer, get_tensor_model_parallel_group, get_tensor_model_parallel_rank, get_tensor_model_parallel_world_size
  from .mappings import copy_to_tensor_model_parallel_region, gather_from_sequence_parallel_region, gather_from_tensor_model_parallel_region, reduce_from_tensor_model_parallel_region, reduce_scatter_to_sequence_parallel_region, scatter_to_tensor_model_parallel_region
  from .random import get_cuda_rng_tracker, get_expert_parallel_rng_tracker_name
  from .utils import VocabUtility, divide, split_tensor_along_last_dim
label_desc: Use image node_id 2 for calling the RowParallelLinear obj's underlying member method code with example usage: obj.forward(input_) and returns: output,  output_bias