class_docstr: Gated MLP projection container.
class_docstr_tok: ["Gated","MLP","projection","container","."]
docstr: Our gated format (this is different from InferenceV1!) is to have the gate and activated neurons
  interleaved. So if we have 4 output neurons (two effective neurons) with 4 input neurons, the finalized
  parameter will look like:
  [g0_0, g0_1, g0_2, g0_3]
  [a0_0, a0_1, a0_2, a0_3]
  [g1_0, g1_1, g1_2, g1_3]
  [a1_0, a1_1, a1_2, a1_3]
  
  As a reference, in inference v1, the format is:
  [g0_0, g0_1, g0_2, g0_3]
  [g1_0, g1_1, g1_2, g1_3]
  [a0_0, a0_1, a0_2, a0_3]
  [a1_0, a1_1, a1_2, a1_3]
docstr_tok: ["Our","gated","format","(","this","is","different","from","InferenceV1","!",")","is","to","have","the","gate","and","activated","neurons","interleaved",".","So","if","we","have","4","output","neurons","(","two","effective","neurons",")","with","4","input","neurons",",","the","finalized","parameter","will","look","like",":","[","g0_0",",","g0_1",",","g0_2",",","g0_3","]","[","a0_0",",","a0_1",",","a0_2",",","a0_3","]","[","g1_0",",","g1_1",",","g1_2",",","g1_3","]","[","a1_0",",","a1_1",",","a1_2",",","a1_3","]","As","a","reference",",","in","inference","v1",",","the","format","is",":","[","g0_0",",","g0_1",",","g0_2",",","g0_3","]","[","g1_0",",","g1_1",",","g1_2",",","g1_3","]","[","a0_0",",","a0_1",",","a0_2",",","a0_3","]","[","a1_0",",","a1_1",",","a1_2",",","a1_3","]"]
code: def finalize(self) -> torch.Tensor:
      """
      Our gated format (this is different from InferenceV1!) is to have the gate and activated neurons
      interleaved. So if we have 4 output neurons (two effective neurons) with 4 input neurons, the finalized
      parameter will look like:
      [g0_0, g0_1, g0_2, g0_3]
      [a0_0, a0_1, a0_2, a0_3]
      [g1_0, g1_1, g1_2, g1_3]
      [a1_0, a1_1, a1_2, a1_3]
  
      As a reference, in inference v1, the format is:
      [g0_0, g0_1, g0_2, g0_3]
      [g1_0, g1_1, g1_2, g1_3]
      [a0_0, a0_1, a0_2, a0_3]
      [a1_0, a1_1, a1_2, a1_3]
      """
      assert (
          self.gate_params.shape[0] == self.up_params.shape[0]
      ), "Gated MLP parameters must have the same number of neurons."
      total_neurons = (
          self.gate_params.shape[0] + self.up_params.shape[0]
      )
  
      # flip the order if even with the correct tokenizer we get wrong output
      # fused_param = torch.cat([self.up_params, self.gate_params], dim=-1).reshape(total_neurons, -1)
      fused_param = torch.cat(
          [self.gate_params, self.up_params], dim=-1
      ).reshape(total_neurons, -1)
      return self.inference_model.transform_mlp_1_param(fused_param)
  
file_path: DeepSpeed/deepspeed/inference/v2/model_implementations/common_parameters/mlp_parameters.py
imports: import torch
  from ...model_implementations.parameter_base import ParameterBase
label_desc: Use image node_id 1 for calling the GatedMLPParameter obj's underlying member method code with example usage: obj.finalize() and returns: self