Mizan Leaderboard

🔍 How to Use the Leaderboard:

• Search: Use the search box to find specific models
• Color Coding: Scores are color-coded from red (low) to green (high)
• Sorting: Click on column headers to sort by different metrics
• Rankings: Models are ranked by Mean (Task) score

📊 Performance Insights:

• Top Performers: Models with Mean (Task) > 65 show strong overall performance
• Specialized Models: Some models excel in specific tasks (e.g., retrieval vs classification)
• Model Size vs Performance: Larger models generally perform better but with exceptions

🚀 Submit Model for Evaluation

Submit your Turkish embedding model for evaluation on the MTEB Turkish benchmark. Authentication with Hugging Face is required to submit evaluations.

📋 Evaluation Process:

1. Sign In: First, sign in with your Hugging Face account using the button above
2. Submit Request: Fill out the form with your model details and email
3. Admin Review: Your request will be reviewed by administrators
4. Evaluation: If approved, your model will be evaluated on MTEB Turkish benchmark
5. Results: You'll receive email notifications and results will appear on the leaderboard

⚠️ Important Notes:

• Authentication Required: You must be logged in with Hugging Face to submit evaluations
• You'll receive email updates about your request status
• Make sure your model is publicly available on HuggingFace
• Valid email address is required for receiving results

📊 MTEB Turkish Dataset Overview

📈 Task Distribution:

By Task Type:

• Classification: 6 tasks (sentiment, intent, scenario, language identification)
• Retrieval: 4 tasks (FAQ, QA, historical documents, knowledge QA)
• Pair Classification: 2 tasks (natural language inference)
• Clustering: 1 task (language clustering)
• STS: 1 task (semantic textual similarity)

By Domain:

• Sentiment Analysis: Movie and product reviews
• Question Answering: FAQ, reading comprehension, and knowledge QA
• Intent/Scenario: Conversational AI applications
• Language Tasks: NLI, STS, clustering
• Multilingual: Cross-lingual evaluation capabilities

🎯 Evaluation Methodology:

Scoring:

• Each task uses task-specific metrics (accuracy, F1, recall@k, etc.)
• Mean (Task): Direct average of all individual task scores
• Mean (TaskType): Average of task category means
• Individual Categories: Performance in each task type

Model Ranking:

• Primary ranking by Mean (Task) score
• Correlation metrics provide additional insights
• Task-specific performance shows model strengths

Quality Assurance:

• Standardized evaluation protocols
• Consistent preprocessing across tasks
• Multiple metrics per task for robustness

📊 Metrics Explanation:

• Mean (Task): Average performance across all individual tasks
• Mean (TaskType): Average performance by task categories
• Classification: Performance on Turkish classification tasks
• Clustering: Performance on Turkish clustering tasks
• Pair Classification: Performance on pair classification tasks (like NLI)
• Retrieval: Performance on information retrieval tasks
• STS: Performance on Semantic Textual Similarity tasks
• Correlation: Weighted average of correlation metrics for NLI and STSB datasets
• Parameters: Number of model parameters
• Embed Dim: Embedding dimension size
• Max Seq Length: Maximum sequence length the model can process (0 = infinite/unlimited)
• Vocab Size: Size of the model's vocabulary

📖 About Mizan:

This leaderboard presents results from the Mizan benchmark, which evaluates embedding models on Turkish language tasks across multiple domains including:

• Text classification and sentiment analysis
• Information retrieval and search
• Semantic textual similarity
• Text clustering and pair classification

🚀 Submit Your Model:

Use the Submit tab to submit your Turkish embedding model for evaluation. Your request will be reviewed by administrators and you'll receive email notifications about the progress.

Contact:

For any questions or feedback, please contact info@newmind.ai

Links:

• GitHub: mteb/mteb v1.38.51 - Mizan is currently based on MTEB v1.38.51 (MTEB v2.0.0 support coming soon)