File size: 5,036 Bytes
fd07231 df5c4aa fd07231 d4f8b5e fd07231 217f444 4e67133 217f444 a08404c 217f444 3203b11 217f444 2f49fcd 217f444 12de5b8 217f444 82d9fe3 217f444 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
---
library_name: transformers
license: llama3
base_model: meta-llama/Llama-3.2-1B-Instruct
tags:
- full
- generated_from_trainer
- nlp
model-index:
- name: stratos_unverified_mix_2nodes
results: []
datasets:
- open-thoughts/open-thoughts-114k
- open-thoughts/OpenThoughts-Unverified-173k
---
# LogiLlama
**LogiLlama** is a fine-tuned language model developed by Goppa AI. Built upon a 1B-parameter base from LLaMA, LogiLlama has been enhanced with injected knowledge and logical reasoning abilities. Our mission is to make smaller models smarter—delivering improved reasoning and problem-solving capabilities while maintaining a low memory footprint and energy efficiency for on-device applications.
---
## Model Summary
While recent trends in language models have leaned towards scaling up parameters, LogiLlama demonstrates that “less can be more.” By fine-tuning a 1B parameter base model with advanced logical reasoning techniques, LogiLlama offers:
- **Enhanced Reasoning:** Improved logical thinking and knowledge integration for more accurate and context-aware responses.
- **Efficiency:** Designed for on-device processing with a low memory and energy footprint.
- **Transparency:** Our training process and configuration files are fully open-source, reflecting our commitment to transparent and reproducible research.
LogiLlama is the first step in our journey at Goppa AI to develop efficient, intelligent, and resource-friendly models that challenge the notion that bigger is always better.
---
## Model Description
- **Model Type:** Small Language Model (SLM) fine-tuned from a 1B parameter LLaMA base
- **Architecture:**
- Hidden Size: 2048
- Hidden Layers: 16
- Attention Heads: 32
- Intermediate Size: 8192
- Special Configuration: Incorporates a customized ROPE scaling (rope_type: "llama3")
- **Tokenization:**
- Custom tokenizer with an extensive set of special tokens (defined in `special_tokens_map.json` and `tokenizer_config.json`)
- **Language:** English
- **License:** Llama 3 Community License Agreement
---
## How to Use
Below is a sample code snippet demonstrating how to use LogiLlama with the Hugging Face Transformers library:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load tokenizer and model from our repository
tokenizer = AutoTokenizer.from_pretrained("goppa-ai/Goppa-LogiLlama", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("goppa-ai/Goppa-LogiLlama", trust_remote_code=True)
model.to('cuda')
text = "When faced with a complex problem, what should one do?"
input_ids = tokenizer(text, return_tensors="pt").to('cuda').input_ids
outputs = model.generate(
input_ids,
max_length=2500,
temperature=0.6,
top_p=0.9,
repetition_penalty=1.2,
pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.batch_decode(outputs[:, input_ids.shape[1]:-1])[0].strip())
```
---
## Configuration Files
The model repository includes several key configuration files that ensure LogiLlama functions seamlessly within the Hugging Face ecosystem:
- **config.json:** Contains the model architecture settings, including hidden size, number of layers, attention heads, and other hyperparameters.
- **generation_config.json:** Defines generation parameters such as temperature, top-p sampling, and end-of-sequence tokens.
- **special_tokens_map.json:** Maps special tokens (e.g., beginning-of-text, end-of-text, padding) used during tokenization.
- **tokenizer_config.json:** Provides metadata and settings for the tokenizer, ensuring consistency with the model’s vocabulary and special tokens.
---
## Training Details
LogiLlama was fine-tuned by injecting logical reasoning and domain-specific knowledge into a 1B parameter LLaMA base. By carefully curating training data and employing specialized techniques, we enhanced the model’s capability to handle reasoning tasks without significantly increasing its size. This project marks our commitment to advancing small, efficient models that do not compromise on performance.
---
## Citation
If you use LogiLlama in your research, please cite:
```bibtex
@misc{goppa2025logillama,
title={LogiLlama: Injecting Logical Reasoning into Small Language Models},
author={Goppa AI},
year={2025},
note={https://github.com/GoppaAI/LogiLlama}
}
```
---
## License
LogiLlama is released under the [MIT License](https://opensource.org/licenses/MIT).
---
## Inference & Deployment
- **Model Size:** 1B parameters
- **Tensor Type:** float32 (F32)
- **Deployment:** Optimized for on-device inference and resource-constrained environments. Currently available for local deployment; stay tuned for updates on hosted inference solutions.
---
At Goppa AI, we are committed to pushing the boundaries of efficiency and intelligence in language models. LogiLlama is our first step towards creating small models that are not only resource-friendly but also smart enough to handle complex reasoning tasks.
Happy innovating! |