A standing series of independent evaluations of how large language models actually handle Arabic: Modern Standard Arabic, the dialects, accents, culture and pragmatics. Every article leads with published, sourced numbers and a reusable test protocol.
Ehab Saleh | techkahwa.net Model released: June 7, 2024 | Published: June 21, 2024 The numbers first Metric Result What it measures Source Declared languages 27 beyond English and Chinese The model’s scope Qwen2 blog, June 7, 2024 The “Middle East” group Arabic, Persian, Hebrew, Turkish An explicit named regional classification Same source The technical … Read more
Ehab Saleh | techkahwa.net Model released: May 23, 2024 | Published: June 6, 2024 The numbers first Metric Result What it measures Source Number of languages 23 The model’s scope Aya 23 technical report Arabic’s position on the list First Alphabetical in English, but named explicitly Same report The languages Arabic, Chinese, Czech, Dutch, English, … Read more
Ehab Saleh | techkahwa.net Leaderboard launched: May 14, 2024 | Published: June 4, 2024 The numbers first Metric Result What it measures Source Leaderboard launch date May 14, 2024 Hugging Face with the Technology Innovation Institute Launch blog The original AlGhafa benchmark Roughly a dozen datasets, mostly native Arabic The Arabic foundation Almazrouei et al., … Read more
Ehab Saleh | techkahwa.net Model released: May 13, 2024 | Published: May 30, 2024 The numbers first Metric Result What it measures Source Falcon 2 11B on Arabic ARC-C, 25-shot 25.32 A four-option test whose random baseline is 25 Technical report, arXiv:2407.14885, Table 11 On Arabic MMLU, 25-shot 28.04 General knowledge Same table On Arabic … Read more
Ehab Saleh | techkahwa.net Model released: May 13, 2024 | Published: May 27, 2024 The numbers first Metric Result What it measures Source Arabic, before the new tokenizer 53 tokens The same sample sentence GPT-4o launch page, May 13, 2024 Arabic, after the new tokenizer 26 tokens A reduction by a factor of two Same … Read more
Ehab Saleh | techkahwa.net Model released: April 4, 2024 | Published: April 18, 2024 The numbers first Metric Result What it measures Source Arabic among the optimised languages Yes, by name An explicit commitment from a Western lab Model card, Cohere Number of optimised languages 10 English, French, Spanish, Italian, German, Brazilian Portuguese, Japanese, Korean, … Read more
Ehab Saleh | techkahwa.net Model released: March 4, 2024 | Published: March 18, 2024 The numbers first Metric Result What it measures Source Release date March 4, 2024 Opus and Sonnet available the same day Anthropic announcement Multilingual results in the model card Present: MGSM multilingual math, and multilingual MMLU Section 5.6.1 Claude 3 model … Read more
Ehab Saleh | techkahwa.net Paper released: February 20, 2024 | Published: March 5, 2024 The numbers first Metric Result What it measures Source Number of questions 14,575 multiple choice The size of the exam ArabicMMLU paper, Findings of ACL 2024 Number of tasks 40 Breadth of domains Same paper Source countries 8 Arab countries Geographic … Read more
Ehab Saleh | techkahwa.net Model released: December 11, 2023 | Published: December 25, 2023 The numbers first Metric Result What it measures Source Declared languages Five: English, French, Italian, German, Spanish What the lab says about its own model Mistral announcement, December 11, 2023 Arabic mentioned in the announcement Not present Whether it was mentioned … Read more
Ehab Saleh | techkahwa.net Model released: November 6, 2023 | Published: November 20, 2023 The numbers first Metric Result What it measures Source Whisper large-v3, Modern Standard Arabic on the SADA set 27.95% error The model’s best case in Arabic Open Universal Arabic ASR Leaderboard, Interspeech 2025 Whisper large-v3, Najdi 48.58% The first stage of … Read more