Hasan abusheikh's picture

Hasan abusheikh PRO

hasanabusheikh
·

AI & ML interests

None yet

Recent Activity

posted an update 5 days ago
🚀 Benchmarking Mistral Saba: Where Does It Stand? The AI race is evolving rapidly, with new models emerging to cater to regional and domain-specific needs. Mistral Saba, a 24-billion-parameter model optimized for Arabic and South Asian languages, aims to bridge linguistic gaps in AI. But does it deliver? Our latest benchmarking report reveals some critical insights: 🔹 Strengths: ✅ Cost-effective—$0.20 per million input tokens, making it budget-friendly. ✅ High throughput—Processes 150+ tokens per second, ensuring efficiency. 🔻 Major Shortcomings: ❌ Struggles with Arabic dialects—Fails to handle Egyptian, Gulf, and Levantine variations. ❌ Poor performance in Modern Standard Arabic (MSA) languages. ❌ Severe hallucinations—Generates fabricated religious content and incorrect citations. ❌ Weak logical & mathematical reasoning—Falls short in benchmarks like HellaSwag and GSM8K. ❌ Poor factual accuracy—Mistral Saba underperforms against GPT-4o and Claude 3.5 in truthfulness tests. While regional AI models are much needed, transparency, dataset curation, and ethical oversight remain crucial for their reliability. The industry must focus on community-driven dataset creation, third-party audits, and stakeholder collaboration to develop truly localized AI that serves its target populations accurately. Solution in 2 words: Hyper-Localization 💡 What are your thoughts on the need for region-specific AI models? Let’s discuss! 👇 hashtag#AI hashtag#Benchmarking hashtag#LLMs hashtag#MistralSaba hashtag#arabic hashtag#ethical hashtag#mistralhttps://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501
updated a Space almost 2 years ago
hasanabusheikh/bakrianoo-sinai-voice-ar-stt
View all activity

Organizations

CNTXT AI's profile picture

Posts 2

view post
Post
908
🚀 Benchmarking Mistral Saba: Where Does It Stand?
The AI race is evolving rapidly, with new models emerging to cater to regional and domain-specific needs. Mistral Saba, a 24-billion-parameter model optimized for Arabic and South Asian languages, aims to bridge linguistic gaps in AI. But does it deliver? Our latest benchmarking report reveals some critical insights:
🔹 Strengths:
✅ Cost-effective—$0.20 per million input tokens, making it budget-friendly.
✅ High throughput—Processes 150+ tokens per second, ensuring efficiency.

đź”» Major Shortcomings:
❌ Struggles with Arabic dialects—Fails to handle Egyptian, Gulf, and Levantine variations.
❌ Poor performance in Modern Standard Arabic (MSA) languages.
❌ Severe hallucinations—Generates fabricated religious content and incorrect citations.
❌ Weak logical & mathematical reasoning—Falls short in benchmarks like HellaSwag and GSM8K.
❌ Poor factual accuracy—Mistral Saba underperforms against GPT-4o and Claude 3.5 in truthfulness tests.
While regional AI models are much needed, transparency, dataset curation, and ethical oversight remain crucial for their reliability. The industry must focus on community-driven dataset creation, third-party audits, and stakeholder collaboration to develop truly localized AI that serves its target populations accurately.

Solution in 2 words:
Hyper-Localization

💡 What are your thoughts on the need for region-specific AI models? Let’s discuss! 👇

hashtag#AI hashtag#Benchmarking hashtag#LLMs hashtag#MistralSaba hashtag#arabic hashtag#ethical

hashtag#mistralhttps://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501

models

None public yet

datasets

None public yet