class_docstr: This container identifies which methods need to be overridden in addition to
  the base container to enable use in the RLHF pipeline. These methods are not
  necessary for inference alone.
  
  NOTE: If you are using this feature with a container that
  also inherits from `MetaTensorContainer`, ensure that `MetaTensorContainer`
  is inherited before `HybridEngineContainer` in the class definition.
class_docstr_tok: ["This","container","identifies","which","methods","need","to","be","overridden","in","addition","to","the","base","container","to","enable","use","in","the","RLHF","pipeline",".","These","methods","are","not","necessary","for","inference","alone",".","NOTE",":","If","you","are","using","this","feature","with","a","container","that","also","inherits","from","`","MetaTensorContainer","`",",","ensure","that","`","MetaTensorContainer","`","is","inherited","before","`","HybridEngineContainer","`","in","the","class","definition","."]
docstr: Delete module parameters if they exist and point them back to the container. The primary
  purpose of this is for TP-inference with ZeRO-3. In this scenario, we need to delete the
  parameters we've created for inference to free their memory.
docstr_tok: ["Delete","module","parameters","if","they","exist","and","point","them","back","to","the","container",".","The","primary","purpose","of","this","is","for","TP-inference","with","ZeRO-3",".","In","this","scenario",",","we","need","to","delete","the","parameters","we","'ve","created","for","inference","to","free","their","memory","."]
code: def release_memory(self):
      """
      Delete module parameters if they exist and point them back to the container. The primary
      purpose of this is for TP-inference with ZeRO-3. In this scenario, we need to delete the
      parameters we've created for inference to free their memory.
      """
      general_params = [
          (self.module.attention.attn_ow, self.dense_w),
          (self.module.attention.attn_ob, self.dense_b),
          (self.module.mlp.attn_nw, self.attn_nw),
          (self.module.mlp.attn_nb, self.attn_nb),
          (self.module.norm_w, self.input_nw),
          (self.module.norm_b, self.input_nb),
      ]
  
      self._release_params(general_params)
  
      self.release_qkv()
      self.release_mlp()
  
imports: from abc import ABC, abstractmethod
  from typing import List, Tuple
  import torch
label_desc: Use image node_id 10 for calling the HybridEngineContainer obj's underlying member method code with example usage: obj.release_memory() without return types