class_docstr: This is a wrapper class about all possible attributes and features that you can play with a model that has been
  loaded using `bitsandbytes`.
  
  This replaces `load_in_8bit` or `load_in_4bit`therefore both options are mutually exclusive.
  
  Currently only supports `LLM.int8()`, `FP4`, and `NF4` quantization. If more methods are added to `bitsandbytes`,
  then more arguments will be added to this class.
  
  Args:
      load_in_8bit (`bool`, *optional*, defaults to `False`):
          This flag is used to enable 8-bit quantization with LLM.int8().
      load_in_4bit (`bool`, *optional*, defaults to `False`):
          This flag is used to enable 4-bit quantization by replacing the Linear layers with FP4/NF4 layers from
          `bitsandbytes`.
      llm_int8_threshold (`float`, *optional*, defaults to 6.0):
          This corresponds to the outlier threshold for outlier detection as described in `LLM.int8() : 8-bit Matrix
          Multiplication for Transformers at Scale` paper: https://arxiv.org/abs/2208.07339 Any hidden states value
          that is above this threshold will be considered an outlier and the operation on those values will be done
          in fp16. Values are usually normally distributed, that is, most values are in the range [-3.5, 3.5], but
          there are some exceptional systematic outliers that are very differently distributed for large models.
          These outliers are often in the interval [-60, -6] or [6, 60]. Int8 quantization works well for values of
          magnitude ~5, but beyond that, there is a significant performance penalty. A good default threshold is 6,
          but a lower threshold might be needed for more unstable models (small models, fine-tuning).
      llm_int8_skip_modules (`List[str]`, *optional*):
          An explicit list of the modules that we do not want to convert in 8-bit. This is useful for models such as
          Jukebox that has several heads in different places and not necessarily at the last position. For example
          for `CausalLM` models, the last `lm_head` is kept in its original `dtype`.
      llm_int8_enable_fp32_cpu_offload (`bool`, *optional*, defaults to `False`):
          This flag is used for advanced use cases and users that are aware of this feature. If you want to split
          your model in different parts and run some parts in int8 on GPU and some parts in fp32 on CPU, you can use
          this flag. This is useful for offloading large models such as `google/flan-t5-xxl`. Note that the int8
          operations will not be run on CPU.
      llm_int8_has_fp16_weight (`bool`, *optional*, defaults to `False`):
          This flag runs LLM.int8() with 16-bit main weights. This is useful for fine-tuning as the weights do not
          have to be converted back and forth for the backward pass.
      bnb_4bit_compute_dtype (`torch.dtype` or str, *optional*, defaults to `torch.float32`):
          This sets the computational type which might be different than the input time. For example, inputs might be
          fp32, but computation can be set to bf16 for speedups.
      bnb_4bit_quant_type (`str`,  *optional*, defaults to `"fp4"`):
          This sets the quantization data type in the bnb.nn.Linear4Bit layers. Options are FP4 and NF4 data types
          which are specified by `fp4` or `nf4`.
      bnb_4bit_use_double_quant (`bool`, *optional*, defaults to `False`):
          This flag is used for nested quantization where the quantization constants from the first quantization are
          quantized again.
      kwargs (`Dict[str, Any]`, *optional*):
          Additional parameters from which to initialize the configuration object.
class_docstr_tok: ["This","is","a","wrapper","class","about","all","possible","attributes","and","features","that","you","can","play","with","a","model","that","has","been","loaded","using","`","bitsandbytes","`",".","This","replaces","`","load_in_8bit","`","or","`","load_in_4bit","`","therefore","both","options","are","mutually","exclusive",".","Currently","only","supports","`","LLM.int8","(",")","`",",","`","FP4","`",",","and","`","NF4","`","quantization",".","If","more","methods","are","added","to","`","bitsandbytes","`",",","then","more","arguments","will","be","added","to","this","class",".","Args",":","load_in_8bit","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","This","flag","is","used","to","enable","8-bit","quantization","with","LLM.int8","(",")",".","load_in_4bit","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","This","flag","is","used","to","enable","4-bit","quantization","by","replacing","the","Linear","layers","with","FP4\/NF4","layers","from","`","bitsandbytes","`",".","llm_int8_threshold","(","`","float","`",",","*","optional","*",",","defaults","to","6.0",")",":","This","corresponds","to","the","outlier","threshold","for","outlier","detection","as","described","in","`","LLM.int8","(",")",":","8-bit","Matrix","Multiplication","for","Transformers","at","Scale","`","paper",":","https",":","\/\/arxiv.org\/abs\/2208.07339","Any","hidden","states","value","that","is","above","this","threshold","will","be","considered","an","outlier","and","the","operation","on","those","values","will","be","done","in","fp16",".","Values","are","usually","normally","distributed",",","that","is",",","most","values","are","in","the","range","[","-3.5",",","3.5","]",",","but","there","are","some","exceptional","systematic","outliers","that","are","very","differently","distributed","for","large","models",".","These","outliers","are","often","in","the","interval","[","-60",",","-6","]","or","[","6",",","60","]",".","Int8","quantization","works","well","for","values","of","magnitude","~5",",","but","beyond","that",",","there","is","a","significant","performance","penalty",".","A","good","default","threshold","is","6",",","but","a","lower","threshold","might","be","needed","for","more","unstable","models","(","small","models",",","fine-tuning",")",".","llm_int8_skip_modules","(","`","List","[","str","]","`",",","*","optional","*",")",":","An","explicit","list","of","the","modules","that","we","do","not","want","to","convert","in","8-bit",".","This","is","useful","for","models","such","as","Jukebox","that","has","several","heads","in","different","places","and","not","necessarily","at","the","last","position",".","For","example","for","`","CausalLM","`","models",",","the","last","`","lm_head","`","is","kept","in","its","original","`","dtype","`",".","llm_int8_enable_fp32_cpu_offload","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","This","flag","is","used","for","advanced","use","cases","and","users","that","are","aware","of","this","feature",".","If","you","want","to","split","your","model","in","different","parts","and","run","some","parts","in","int8","on","GPU","and","some","parts","in","fp32","on","CPU",",","you","can","use","this","flag",".","This","is","useful","for","offloading","large","models","such","as","`","google\/flan-t5-xxl","`",".","Note","that","the","int8","operations","will","not","be","run","on","CPU",".","llm_int8_has_fp16_weight","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","This","flag","runs","LLM.int8","(",")","with","16-bit","main","weights",".","This","is","useful","for","fine-tuning","as","the","weights","do","not","have","to","be","converted","back","and","forth","for","the","backward","pass",".","bnb_4bit_compute_dtype","(","`","torch.dtype","`","or","str",",","*","optional","*",",","defaults","to","`","torch.float32","`",")",":","This","sets","the","computational","type","which","might","be","different","than","the","input","time",".","For","example",",","inputs","might","be","fp32",",","but","computation","can","be","set","to","bf16","for","speedups",".","bnb_4bit_quant_type","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","fp4","''","`",")",":","This","sets","the","quantization","data","type","in","the","bnb.nn.Linear4Bit","layers",".","Options","are","FP4","and","NF4","data","types","which","are","specified","by","`","fp4","`","or","`","nf4","`",".","bnb_4bit_use_double_quant","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","This","flag","is","used","for","nested","quantization","where","the","quantization","constants","from","the","first","quantization","are","quantized","again",".","kwargs","(","`","Dict","[","str",",","Any","]","`",",","*","optional","*",")",":","Additional","parameters","from","which","to","initialize","the","configuration","object","."]
code: def __repr__(self):
      config_dict = self.to_dict()
      return f"{self.__class__.__name__} {json.dumps(config_dict, indent=2, sort_keys=True)}\n"
  
imports: import copy
  import importlib.metadata
  import json
  import os
  from dataclasses import dataclass
  from enum import Enum
  from typing import Any, Dict, List, Optional, Union
  from packaging import version
  from ..utils import is_auto_awq_available, is_torch_available, logging
label_desc: Use image node_id 6 for calling the BitsAndBytesConfig obj's underlying member method code with example usage: obj.__repr__() and returns: self, str, json, str