class_docstr: A Module that encapsulates the call to :attr:`xformers.ops.swiglu`,
  and holds the weights for the 3 linear layers
class_docstr_tok: ["A","Module","that","encapsulates","the","call","to",":","attr",":","`","xformers.ops.swiglu","`",",","and","holds","the","weights","for","the","3","linear","layers"]
docstr: Create a SwiGLU module
  
  Args:
      in_features (int): Number of features of the input
      hidden_features (int): Number of hidden features
      out_features (Optional[int], optional): Number of features of the input. Defaults to None.
      bias (bool, optional): Whether linear layers also include a bias. Defaults to True.
docstr_tok: ["Create","a","SwiGLU","module","Args",":","in_features","(","int",")",":","Number","of","features","of","the","input","hidden_features","(","int",")",":","Number","of","hidden","features","out_features","(","Optional","[","int","]",",","optional",")",":","Number","of","features","of","the","input",".","Defaults","to","None",".","bias","(","bool",",","optional",")",":","Whether","linear","layers","also","include","a","bias",".","Defaults","to","True","."]
code: def __init__(
      self,
      in_features: int,
      hidden_features: int,
      out_features: Optional[int] = None,
      bias: bool = True,
      *,
      _pack_weights: bool = True,
  ) -> None:
      """Create a SwiGLU module
  
      Args:
          in_features (int): Number of features of the input
          hidden_features (int): Number of hidden features
          out_features (Optional[int], optional): Number of features of the input. Defaults to None.
          bias (bool, optional): Whether linear layers also include a bias. Defaults to True.
      """
      super().__init__()
      out_features = out_features or in_features
      hidden_features = hidden_features or in_features
  
      self.w12: Optional[nn.Linear]
      if _pack_weights:
          self.w12 = nn.Linear(
              in_features, 2 * hidden_features, bias=bias
          )
      else:
          self.w12 = None
          self.w1 = nn.Linear(in_features, hidden_features, bias=bias)
          self.w2 = nn.Linear(in_features, hidden_features, bias=bias)
      self.w3 = nn.Linear(hidden_features, out_features, bias=bias)
  
      self.hidden_features = hidden_features
      self.out_features = out_features
      self.in_features = in_features
      self.op = None
  
imports: from dataclasses import dataclass
  from typing import Dict, Optional, Sequence, Tuple, Union
  import torch
  import torch.nn.functional
  from torch import nn
  from .common import BaseOperator, get_xformers_operator, register_operator
  from .unbind import stack_or_none, unbind
label_desc: Use image node_id 1 to create a new SwiGLU object from inherited base classes: nn with example: obj = SwiGLU(in_features, hidden_features, out_features, bias)