ilovesushiandkimchiandmalaxiangguo/shuPT-xiaohongshu-Llama3-8B-Instruct

Model Card for Model ID

Introducing 🍠(shǔ)PT-Llama3-8B-Instruct ! A SFT (Supervised Fine-Tuning) model trained on Xiaohongshu data.

Model Details

• Base Model: Meta-Llama-3-8B-Instruct
• Architecture: LLaMA 3 with LoRA fine-tuning
• Language: Chinese
• Task: Instruction following and conversational responses
• License: Same as base LLaMA 3 model.

Training Details

• Training Data: 30k crawled from Xiaohongshu (小红书)
• Framework: Hugging Face Transformers, PEFT, TRL

Training Data Details

• Data source: 30k entries crawled from Xiaohongshu (小红书), specifically targeting 'city walk' related posts
• Copyright: All content rights belong to Xiaohongshu
• Usage restrictions: Academic/research purposes only
• Format: Alpaca-style instruction tuning format
• Known issues:
• Incomplete conversation formatting in some samples
• EOS token placement inconsistencies
• Response length variations

Training Parameters

• Learning rate: 2e-4
• Weight decay: 0.001
• Max gradient norm: 0.3
• Warmup ratio: 0.03
• LR scheduler: Cosine
• Training precision: 4-bit quantization (QLoRA)
• LoRA rank (r): 64
• LoRA alpha: 16
• LoRA dropout: 0.1

Intended Use

• Chinese language instruction following
• Conversational responses
• Location-based recommendations
• City navigation assistance
• Cultural and historical information sharing

Limitations

• Limited to Chinese language understanding and generation
• Domain-specific knowledge biased towards Xiaohongshu content
• Inherits base model limitations
• May generate inconsistent responses due to temperature-based sampling

Performance

• Shows improved performance on Chinese instruction following
• Demonstrates strong capabilities in location-based recommendations

Ethical Considerations

• Model inherits potential biases from Xiaohongshu data
• Should be used in compliance with base model's usage policies
• Content generation should be monitored for accuracy and appropriateness

Optimization Opportunities

• Training parameters could be optimized for better performance
• Data cleaning and formatting could be improved

Result

• Question: 请推荐一下北京的city walk路线 (Please recommend some city walk routes in Beijing)