Update README.md
Browse files
README.md
CHANGED
@@ -22,25 +22,17 @@ Model created by analyzing and selecting the optimal layers from other Qwen2.5-7
|
|
22 |
- Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
|
23 |
- Calculate maximum possible entropy: H_max = log₂(n)
|
24 |
- Final NER score = H/H_max # normalizes score to [0,1] range
|
25 |
-
- Results in value between 0 and 1
|
26 |
-
# 0 = single dimension dominance, 1 = uniform dimensional utilization
|
27 |
|
28 |
## Creating Composite Model
|
29 |
|
30 |
Code here: https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0/blob/main/ner_merge.py
|
31 |
|
32 |
-
|
33 |
-
- Download
|
34 |
- Calculate Normalized Effective Rank (NER) for each layer within each model
|
35 |
-
|
36 |
-
Layer Selection:
|
37 |
-
- Identify common layer structures across models
|
38 |
- Define model and layer name pairs that have highest NER for each layer based on their NER scores
|
39 |
-
|
40 |
-
Model Composition:
|
41 |
-
- Incrementally build a composite model using layer with highest NER from model pool.
|
42 |
-
|
43 |
-
Output Generation:
|
44 |
- Save merge reports documenting layer sources
|
45 |
- Copy config and tokenizer files from base model
|
46 |
- Save the composite model with complete weights # model ready to use
|
|
|
22 |
- Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
|
23 |
- Calculate maximum possible entropy: H_max = log₂(n)
|
24 |
- Final NER score = H/H_max # normalizes score to [0,1] range
|
25 |
+
- Results in value between 0 and 1 for each model layer
|
|
|
26 |
|
27 |
## Creating Composite Model
|
28 |
|
29 |
Code here: https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0/blob/main/ner_merge.py
|
30 |
|
31 |
+
Code functions:
|
32 |
+
- Download selected models from Hugging Face Hub
|
33 |
- Calculate Normalized Effective Rank (NER) for each layer within each model
|
|
|
|
|
|
|
34 |
- Define model and layer name pairs that have highest NER for each layer based on their NER scores
|
35 |
+
- Incrementally build a composite model using layer with highest NER from model pool
|
|
|
|
|
|
|
|
|
36 |
- Save merge reports documenting layer sources
|
37 |
- Copy config and tokenizer files from base model
|
38 |
- Save the composite model with complete weights # model ready to use
|