Update README.md
Browse files
README.md
CHANGED
|
@@ -22,25 +22,17 @@ Model created by analyzing and selecting the optimal layers from other Qwen2.5-7
|
|
| 22 |
- Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
|
| 23 |
- Calculate maximum possible entropy: H_max = log₂(n)
|
| 24 |
- Final NER score = H/H_max # normalizes score to [0,1] range
|
| 25 |
-
- Results in value between 0 and 1
|
| 26 |
-
# 0 = single dimension dominance, 1 = uniform dimensional utilization
|
| 27 |
|
| 28 |
## Creating Composite Model
|
| 29 |
|
| 30 |
Code here: https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0/blob/main/ner_merge.py
|
| 31 |
|
| 32 |
-
|
| 33 |
-
- Download
|
| 34 |
- Calculate Normalized Effective Rank (NER) for each layer within each model
|
| 35 |
-
|
| 36 |
-
Layer Selection:
|
| 37 |
-
- Identify common layer structures across models
|
| 38 |
- Define model and layer name pairs that have highest NER for each layer based on their NER scores
|
| 39 |
-
|
| 40 |
-
Model Composition:
|
| 41 |
-
- Incrementally build a composite model using layer with highest NER from model pool.
|
| 42 |
-
|
| 43 |
-
Output Generation:
|
| 44 |
- Save merge reports documenting layer sources
|
| 45 |
- Copy config and tokenizer files from base model
|
| 46 |
- Save the composite model with complete weights # model ready to use
|
|
|
|
| 22 |
- Compute Shannon entropy: H = -Σ(pᵢ * log₂(pᵢ)) # measures information content
|
| 23 |
- Calculate maximum possible entropy: H_max = log₂(n)
|
| 24 |
- Final NER score = H/H_max # normalizes score to [0,1] range
|
| 25 |
+
- Results in value between 0 and 1 for each model layer
|
|
|
|
| 26 |
|
| 27 |
## Creating Composite Model
|
| 28 |
|
| 29 |
Code here: https://huggingface.co/jeffmeloy/Qwen2.5-7B-nerd-uncensored-v1.0/blob/main/ner_merge.py
|
| 30 |
|
| 31 |
+
Code functions:
|
| 32 |
+
- Download selected models from Hugging Face Hub
|
| 33 |
- Calculate Normalized Effective Rank (NER) for each layer within each model
|
|
|
|
|
|
|
|
|
|
| 34 |
- Define model and layer name pairs that have highest NER for each layer based on their NER scores
|
| 35 |
+
- Incrementally build a composite model using layer with highest NER from model pool
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
- Save merge reports documenting layer sources
|
| 37 |
- Copy config and tokenizer files from base model
|
| 38 |
- Save the composite model with complete weights # model ready to use
|