Hugging Face's logo

Mario12355
/
swabian_german_translator

Text2Text Generation
Transformers
Safetensors
German
llama
translation
german
dialect
swabian
qlora
dpo
Eval Results
Inference Endpoints
Model card Files Community
swabian_german_translator / README.md
Mario12355's picture
Mario12355
Update README.md
feca0b6 verified
preview code
|
raw
history blame contribute delete
4.85 kB
metadata
language:
  - de
license: mit
library_name: transformers
pipeline_tag: text2text-generation
tags:
  - llama
  - translation
  - german
  - dialect
  - swabian
  - qlora
  - dpo
datasets:
  - custom
model-index:
  - name: swabian-german-translator
    results:
      - task:
          type: translation
          name: German-Swabian Translation
        metrics:
          - type: accuracy
            value: 0.8
            name: Training Loss
          - type: bleu
            value: N/A
            name: BLEU Score
metadata:
  author:
    - Your Name
  framework: pytorch
  fine_tuning_type:
    - dpo
    - qlora
base_model: llama-3.1-8b
training_data: Custom dataset based on Schwäbisch-Schwätza wordbook
training_processes:
  - sft
  - dpo

Swabian-German Translation Model (DPO-Enhanced)

This model fine-tunes LLAMA 3.1 8B for bidirectional translation between Standard German and Swabian dialect, enhanced through Direct Preference Optimization (DPO).

Model Details

  • Base Model: LLAMA 3.1 8B
  • Training Method: Two-stage fine-tuning (SFT + DPO)
  • Training Data: 12,000+ word-pair translations with contextual sentences
  • Hardware Requirements: Compatible with single-GPU setups (thanks to QLoRA)

Intended Use

  • Translating between Standard German and Swabian dialect
  • Understanding and preserving regional linguistic variations
  • Educational purposes for language learners

Usage

Basic Translation 

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "your-username/swabian-translator-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Example translation from Swabian to Standard German
def translate(text, direction="to_german"):
    if direction == "to_german":
        prompt = f"Übersetze ins Hochdeutsche: {text}"
    else:
        prompt = f"Übersetze ins Schwäbische: {text}"
    
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(**inputs, max_length=100)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example usage
swabian_text = "Du hosch ja a blaus Mol am Arm!"
german_translation = translate(swabian_text, "to_german")
print(german_translation)  # Expected: "Du hast ja einen Bluterguss am Arm!"

Translation Examples

Swabian to German:

Input: "I han koi Zeit"
Output: "Ich habe keine Zeit"

Input: "Des goht et"
Output: "Das geht nicht"

Input: "Wo bisch du her komma?"
Output: "Woher kommst du?"

German to Swabian:

Input: "Ich verstehe das nicht"
Output: "I versteh des et"

Input: "Das schmeckt sehr gut"
Output: "Des schmeckt arg guat"

Model Architecture & Training

Training Process

  1. Initial Dataset Preparation

    • Base dataset: 12,000+ word pairs from Schwäbisch-Schwätza wordbook
    • Context enhancement using LLM-generated sentences
    • Manual verification and cleanup

  2. SFT (Supervised Fine-Tuning)

    • QLoRA implementation for efficient training
    • 2 epochs on the complete dataset
    • Loss convergence at ~0.8

  3. DPO (Direct Preference Optimization)

    • 300 carefully curated preference pairs
    • 3 epochs of preference learning
    • Focus on natural and accurate translations

Technical Implementation

  • Quantized training using QLoRA
  • 4-bit precision for efficient resource usage
  • Training framework: UnslothAI
  • Single GPU training (~16GB VRAM required)

Limitations and Considerations

  1. Dialect Variations

    • Swabian varies significantly by region
    • Model focuses on common/standard Swabian expressions
    • May not capture all local variations

  2. Translation Quality

    • Best performance on common phrases and expressions
    • May struggle with very colloquial or context-dependent translations
    • Not recommended for official or legal translations

  3. Technical Limitations

    • Input length limited to 512 tokens
    • Generation speed affected by quantization
    • Memory requirements: ~8GB RAM minimum

Community and Contributions

We welcome community contributions to improve the model:

  • Additional training data
  • Regional variant documentation
  • Bug reports and fixes
  • Performance improvements

Please submit issues or pull requests through the Hugging Face repository.

Citation and Attribution 

@misc{swabian-german-translator,
  author = {[Your Name]},
  title = {Swabian-German Translation Model},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments