README.md

---
language:
- ar
- en
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:34436
- loss:MatryoshkaLoss
- loss:CoSENTLoss
base_model: AhmedZaky1/DIMI-embedding-v2
widget:
- source_sentence: الرجل يركب حصاناً
  sentences:
  - رجل يُبث الجبن الممزق على البيتزا
- source_sentence: المرأة تقلي لحم خنزير مشوي
  sentences:
  - امرأة تطبخ لحم خنزير مخبوز
  - طائرة طيران تقلع
- source_sentence: امرأة تحمل في ذراعها طفل كنغر
  sentences:
  - امرأة تعزف على الغيتار
  - امرأة تحمل و تحمل طفل كنغر
- source_sentence: رجل يعزف على الناي
  sentences:
  - طائرة ستقلع
  - رجل يعزف على فرقة الخيزران
- source_sentence: ثلاثة رجال يلعبون الشطرنج.
  sentences:
  - رجلين يلعبان الشطرنج
  - بعض الرجال يقاتلون
datasets:
- silma-ai/silma-arabic-english-sts-dataset-v1.0
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- pearson_cosine
- spearman_cosine
---


# DIMI Embedding model

<div align="center">

<img src="https://cdn-uploads.huggingface.co/production/uploads/65fb3ac20cfe262da2bb0fcc/uOuEn0LNhSVEBbOLwfFUu.jpeg" width="300"/>

*State-of-the-art Multilingual Sentence Embeddings for Arabic-English Semantic Similarity*

</div>

## 🚀 Model Description

**DIMI-embedding-v3-silma-sts-matryoshka** is a cutting-edge multilingual sentence embedding model specifically fine-tuned for Arabic-English semantic textual similarity tasks. Built upon the robust DIMI-embedding-v2 architecture, this model leverages **Matryoshka Representation Learning** combined with **CoSENT Loss** to deliver exceptional performance across multiple embedding dimensions.

### ✨ Key Features

- **Multi-dimensional embeddings**: Supports output dimensions of 768, 512, 256, 128, and 64
- **Bilingual expertise**: Optimized for Arabic and English text processing
- **Matryoshka architecture**: Efficient embedding computation at multiple granularities
- **State-of-the-art performance**: Fine-tuned on the comprehensive Silma Arabic-English STS dataset
- **Cosine similarity optimized**: Perfect for semantic similarity and retrieval tasks

## 📊 Model Performance

The model demonstrates exceptional performance across different embedding dimensions:

### Training Techniques

This model was trained using advanced techniques for optimal performance:

- **Matryoshka Representation Learning**: Enables efficient embeddings at multiple dimensions [768, 512, 256, 128, 64] without retraining
- **CoSENT Loss Function**: Cosine-based sentence embedding loss for superior semantic similarity learning
- **Multi-dimensional Evaluation**: Simultaneous optimization across all target dimensions during training
- **Mixed Precision Training (FP16)**: Accelerated training with maintained numerical stability
- **Warmup Learning Rate Schedule**: Gradual learning rate increase for stable convergence
- **Best Model Selection**: Automatic selection based on highest Spearman correlation on 768d embeddings

### Final Model Performance

#### Development Set Results (Silma STS Dataset)
Final evaluation on the held-out development set:

| Dimension | Pearson Correlation | Spearman Correlation |
|-----------|-------------------|---------------------|
| 768d | 0.8894 | 0.8358 |
| 512d | 0.8959 | 0.8395 |
| 256d | 0.8979 | 0.8470 |
| 128d | 0.9182 | 0.8562 |
| 64d | 0.9066 | 0.8434 |

#### MTEB STS17 Arabic Test Results
Performance on the standard MTEB STS17 (ar-ar) benchmark:

| Dimension | Pearson Correlation | Spearman Correlation |
|-----------|-------------------|---------------------|
| **768d** | **0.8205** | **0.8258** |
| **512d** | **0.8193** | **0.8227** |
| **256d** | **0.8191** | **0.8246** |
| **128d** | **0.8115** | **0.8183** |
| **64d** | **0.7962** | **0.8077** |

**Sequential Score**: 0.8077 (based on 64d performance)

## 🔧 Usage

### Basic Usage

```python
from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('AhmedZaky1/DIMI-embedding-v3-silma-sts-matryoshka', trust_remote_code=True)

# Example sentences in Arabic and English
sentences = [
    "هذا مثال جميل للذكاء الاصطناعي",  # Arabic
    "This is a beautiful example of artificial intelligence",  # English
    "التعلم الآلي يغير العالم",  # Arabic
    "Machine learning is changing the world"  # English
]

# Generate embeddings
embeddings = model.encode(sentences)
print(f"Embedding shape: {embeddings.shape}")

# Calculate cosine similarity
from sklearn.metrics.pairwise import cosine_similarity
similarity_matrix = cosine_similarity(embeddings)
print("Similarity matrix:")
print(similarity_matrix)
```

### Matryoshka Embeddings Usage

```python
# Use different embedding dimensions
dimensions = [768, 512, 256, 128, 64]

for dim in dimensions:
    # Truncate embeddings to specific dimension
    truncated_embeddings = embeddings[:, :dim]
    print(f"Dimension {dim}: {truncated_embeddings.shape}")
    
    # Calculate similarity with truncated embeddings
    similarity = cosine_similarity(truncated_embeddings)
    print(f"Average similarity at {dim}d: {similarity.mean():.4f}")
```

### Semantic Search Example

```python
import numpy as np

# Query and corpus
query = "ما هو الذكاء الاصطناعي؟"  # "What is artificial intelligence?"
corpus = [
    "الذكاء الاصطناعي هو محاكاة الذكاء البشري",
    "Machine learning is a subset of AI",
    "Deep learning uses neural networks",
    "التعلم العميق يستخدم الشبكات العصبية"
]

# Encode query and corpus
query_embedding = model.encode([query])
corpus_embeddings = model.encode(corpus)

# Find most similar documents
similarities = cosine_similarity(query_embedding, corpus_embeddings)[0]
top_indices = np.argsort(similarities)[::-1]

print(f"Query: {query}")
print("\nMost similar documents:")
for i, idx in enumerate(top_indices[:3]):
    print(f"{i+1}. {corpus[idx]} (similarity: {similarities[idx]:.4f})")
```

## 🏗️ Model Architecture

- **Base Model**: DIMI-embedding-v2
- **Training Objective**: CoSENT Loss with Matryoshka Learning
- **Supported Dimensions**: [768, 512, 256, 128, 64]
- **Max Sequence Length**: 512 tokens
- **Pooling Method**: Mean pooling
- **Similarity Function**: Cosine similarity

## 📊 Training Details

### Dataset
- **Primary Dataset**: silma-ai/silma-arabic-english-sts-dataset-v1.0
- **Evaluation Dataset**: MTEB STS17 (ar-ar)
- **Training Samples**: ~24,000+ multilingual sentence pairs
- **Evaluation Samples**: 100 held-out pairs

### Training Configuration
- **Batch Size**: 16
- **Epochs**: 4
- **Learning Rate**: Warmup ratio 0.1
- **Precision**: FP16
- **Evaluation Strategy**: Every 100 steps
- **Best Model Selection**: Highest Spearman correlation on 768d embeddings

### Hardware Requirements
- **GPU**: CUDA-compatible GPU recommended
- **Memory**: 16GB+ RAM for training
- **Storage**: 2GB+ for model weights

## 🎯 Applications

This model excels in various NLP tasks:

- **Semantic Textual Similarity**: Measure similarity between Arabic-English text pairs
- **Information Retrieval**: Find relevant documents in multilingual corpora
- **Paraphrase Detection**: Identify semantically equivalent sentences
- **Cross-lingual Search**: Search Arabic content with English queries and vice versa
- **Clustering**: Group similar multilingual documents
- **Recommendation Systems**: Content-based recommendations across languages

## 📝 Citation

If you use this model in your research, please cite:

```bibtex
@misc{dimi-embedding-v3-2024,
  title={DIMI-embedding-v3-silma-sts-matryoshka: Multilingual Sentence Embeddings for Arabic-English Semantic Similarity},
  author={Ahmed Zaky},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/AhmedZaky1/DIMI-embedding-v3-silma-sts-matryoshka}
}
```

## 📧 Contact

**Author**: Ahmed Zaky  
**Email**: [email protected]  
**GitHub**: [@AhmedZaky1](https://github.com/AhmedZaky1)

## 📄 License

This model is released under the **MIT License**.

```
MIT License

Copyright (c) 2024 Ahmed Zaky

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```

## 🙏 Acknowledgments

- **Silma AI** for providing the high-quality Arabic-English STS dataset
- **Sentence Transformers** library for the excellent framework
- **Hugging Face** for model hosting and distribution
- The **MTEB** benchmark for evaluation standards

---

<div align="center">

**Built with ❤️ by Ahmed Zaky**

*Advancing Arabic NLP through state-of-the-art embedding models*

</div>