jeffmeloy commited on
Commit
7eb2a19
·
verified ·
1 Parent(s): 036c14a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -12
README.md CHANGED
@@ -22,25 +22,17 @@ Model created by analyzing and selecting the optimal layers from other Qwen2.5-7
22
  - Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
23
  - Calculate maximum possible entropy: H_max = log₂(n)
24
  - Final NER score = H/H_max # normalizes score to [0,1] range
25
- - Results in value between 0 and 1
26
- # 0 = single dimension dominance, 1 = uniform dimensional utilization
27
 
28
  ## Creating Composite Model
29
 
30
  Code here: https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0/blob/main/ner_merge.py
31
 
32
- Layer Analysis:
33
- - Download base and fine-tuned models from Hugging Face Hub
34
  - Calculate Normalized Effective Rank (NER) for each layer within each model
35
-
36
- Layer Selection:
37
- - Identify common layer structures across models
38
  - Define model and layer name pairs that have highest NER for each layer based on their NER scores
39
-
40
- Model Composition:
41
- - Incrementally build a composite model using layer with highest NER from model pool.
42
-
43
- Output Generation:
44
  - Save merge reports documenting layer sources
45
  - Copy config and tokenizer files from base model
46
  - Save the composite model with complete weights # model ready to use
 
22
  - Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
23
  - Calculate maximum possible entropy: H_max = log₂(n)
24
  - Final NER score = H/H_max # normalizes score to [0,1] range
25
+ - Results in value between 0 and 1 for each model layer
 
26
 
27
  ## Creating Composite Model
28
 
29
  Code here: https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0/blob/main/ner_merge.py
30
 
31
+ Code functions:
32
+ - Download selected models from Hugging Face Hub
33
  - Calculate Normalized Effective Rank (NER) for each layer within each model
 
 
 
34
  - Define model and layer name pairs that have highest NER for each layer based on their NER scores
35
+ - Incrementally build a composite model using layer with highest NER from model pool
 
 
 
 
36
  - Save merge reports documenting layer sources
37
  - Copy config and tokenizer files from base model
38
  - Save the composite model with complete weights # model ready to use