|
--- |
|
library_name: transformers |
|
tags: |
|
- gpt |
|
- distillation |
|
- mobile |
|
- embedded |
|
- onnx |
|
license: cc-by-nc-4.0 |
|
datasets: |
|
- custom |
|
- web |
|
language: en |
|
widget: |
|
- text: "In order to make pancakes, you need to" |
|
- text: "Once upon a time" |
|
--- |
|
|
|
<p align="center"> |
|
<img src="logo.png" alt="IJK Technology" width="150"> |
|
</p> |
|
|
|
<h1 align="center">IJK Technology – ByteGPT-r1</h1> |
|
|
|
|
|
**ByteGPT-r1** is a distilled version of DeepSeek's QWEN 1.5B model, optimized specifically for mobile and edge computing environments. It maintains impressive language capabilities while being designed for compute- and memory-constrained devices. |
|
|
|
## 🚀 Overview |
|
- **Model Type:** Distilled GPT-style causal language model |
|
- **Base Model:** DeepSeek's QWEN 1.5B |
|
- **Intended Use:** Edge devices, mobile phones, embedded systems |
|
- **Size:** Optimized for mobile deployment |
|
- **Training:** Knowledge distillation from QWEN 1.5B |
|
|
|
## 🧠 Why ByteGPT-r1? |
|
ByteGPT-r1 offers several advantages for mobile and edge deployment: |
|
|
|
1. **Efficient Knowledge Distillation:** |
|
Carefully distilled from DeepSeek's QWEN 1.5B model to preserve capabilities while reducing computational requirements. |
|
|
|
2. **Mobile-First Design:** |
|
Architected specifically for the constraints of mobile devices, with optimizations for both inference speed and memory usage. |
|
|
|
3. **Balanced Performance:** |
|
Maintains a good balance between model size and language generation capabilities, making it practical for real-world mobile applications. |
|
|
|
## 💡 Future Plans |
|
This model is part of our ongoing effort to bring powerful language models to edge devices. Upcoming releases will include: |
|
|
|
- **Specialized Variants:** Domain-specific versions optimized for particular use cases |
|
- **Further Optimizations:** Continued improvements in efficiency and performance |
|
- **Benchmark Results:** Comparative performance on various mobile devices |
|
- **Integration Examples:** More code samples for popular mobile frameworks |
|
|
|
## 💻 Usage |
|
|
|
### **Quick Start (with `transformers`):** |
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("ijktech/ByteGPT-r1", trust_remote_code=True) |
|
tokenizer = AutoTokenizer.from_pretrained("ijktech/ByteGPT-r1") |
|
|
|
input_text = "What is the capital of France?" |
|
inputs = tokenizer(input_text, return_tensors="pt") |
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
``` |
|
|
|
### Tokenizer |
|
|
|
The tokenizer is compatible with AutoTokenizer from Hugging Face: |
|
|
|
```python |
|
tokenizer = AutoTokenizer.from_pretrained("ijktech/ByteGPT-r1") |
|
``` |
|
|
|
### ONNX |
|
|
|
The model is also available in ONNX format, and can be used with the ONNX Runtime: |
|
|
|
```python |
|
import onnxruntime as ort |
|
import numpy as np |
|
|
|
# Create ONNX Runtime session |
|
ort_session = ort.InferenceSession("model.onnx") |
|
|
|
# Helper function to generate text using the ONNX model |
|
def generate_with_onnx(prompt_ids, max_new_tokens=50, temperature=1.0): |
|
input_ids = prompt_ids.clone() |
|
|
|
for _ in range(max_new_tokens): |
|
# Get the last block_size tokens if input is too long |
|
if input_ids.shape[1] > model.block_size: |
|
input_ids = input_ids[:, -model.block_size:] |
|
|
|
# Run inference |
|
ort_inputs = { |
|
'input': input_ids.cpu().numpy() |
|
} |
|
logits = ort_session.run(None, ort_inputs)[0] |
|
|
|
# Get predictions for the next token |
|
logits = torch.from_numpy(logits) |
|
logits = logits[:, -1, :] # Only take the last token's predictions |
|
|
|
# Apply temperature |
|
if temperature != 1.0: |
|
logits = logits / temperature |
|
|
|
# Sample from the distribution |
|
probs = torch.nn.functional.softmax(logits, dim=-1) |
|
next_token = torch.multinomial(probs, num_samples=1) |
|
|
|
# Append the new token |
|
input_ids = torch.cat([input_ids, next_token], dim=1) |
|
|
|
return input_ids |
|
|
|
# Test the generation |
|
prompt = "Hello" |
|
prompt_ids = tok(prompt, return_tensors="pt")["input_ids"] |
|
generated_ids = generate_with_onnx(prompt_ids) |
|
generated_text = tok.decode(generated_ids[0], skip_special_tokens=True) |
|
print(f"Generated text: {generated_text}") |
|
#Generated text: Hello there! How can I assist you today? I'm a helpful AI assistant trained to provide information and answer questions on a wide range of topics. |
|
``` |
|
|
|
### Android Usage |
|
|
|
Coming Soon! |
|
|
|
|
|
### iOS Usage |
|
|
|
Coming Soon! |
|
|
|
|
|
## 📜 License |
|
📍 **CC-BY-NC-4.0**: Free for non-commercial use. |
|
|
|
💼 **Commercial Use**: Contact IJK Technology Ltd for licensing at [[email protected]](mailto:[email protected]). |
|
|
|
## 🛠️ About IJK Technology Ltd |
|
IJK Technology Ltd (IJKTech) develops innovative machine learning models optimized for on-device inference. Our focus is on efficiency, privacy, and usability across mobile and embedded platforms. |
