ARAB-LENS

A standing series of independent evaluations of how large language models actually handle Arabic: Modern Standard Arabic, the dialects, accents, culture and pragmatics. Every article leads with published, sourced numbers and a reusable test protocol.

A Generational Leap With Not One Arabic Number In It: Reading GPT-6 Astra

by
A glass speech bubble holding three different social scenes at once

Ehab Saleh | techkahwa.net Model released: September 4, 2026 | Published: September 18, 2026 The numbers first Metric Result What it measures Source Artificial Analysis Intelligence Index 53 Composite general intelligence, level with Claude Fable 5.1 Artificial Analysis, September 2026 Terminal-Bench v4.0 59% Command line task execution Artificial Analysis AutomationBench-AA 69% Multi-step automation Artificial Analysis … Read more

Hearing Is Not Understanding: Testing Arabic Speech on Gemini 3.8 Flash

by
A studio microphone whose sound wave breaks apart into scattered particles

Ehab Saleh | techkahwa.net Model released: September 2, 2026 | Published: September 16, 2026 The numbers first Metric Result What it measures Source Artificial Analysis Intelligence Index 59 at high reasoning, up 3 points from 3.7 Flash Composite general intelligence Artificial Analysis, September 2026 Tool use 45%, up 12 points on the previous version Multi-step … Read more

MSA in a Dialect Costume: Measuring Dialect Fidelity in Claude Fable 5.1

by
A marble statue wearing an embroidered cloak draped over its shoulders

Ehab Saleh | techkahwa.net Model released: September 1, 2026 | Published: September 15, 2026 The numbers first Metric Result What it measures Source HLE 59.1% (previous best 55.5%) Extremely hard expert questions Artificial Analysis, September 2026 Terminal-Bench v2.1 91.4% Technical task execution Artificial Analysis SciCode 62.0% Scientific coding Artificial Analysis AA-Omniscience accuracy 67.2% Knowledge with … Read more

25.87: The Best Number in Arabic Speech Transcription, and What It Actually Means

by
A soundwave passing through a sieve, a quarter of it falling away as dark fragments

Model released: July 7, 2026 | Published: July 21, 2026 Ehab Saleh | techkahwa.net The numbers first System Word error rate Leaderboard or source Cohere Transcribe Arabic 25.87 Hugging Face Arabic ASR Leaderboard, July 2026 Meta OmniASR-LLM-7B 2.45 points behind the leader Same leaderboard OpenAI Whisper Large V3 About 11 points behind the leader Same … Read more

The Vowel That Flips the Meaning: DeepSeek V4 Pro on Arabic Morphology and Syntax

by
Three identical roots growing three completely different branching structures

Ehab Saleh | techkahwa.net Model released: April 24, 2026 | Published: May 8, 2026 The numbers first Metric Result What it measures Source Artificial Analysis Intelligence Index 52, second among open weights reasoning models Composite general intelligence Artificial Analysis, 2026 GDPval-AA 1554, first among open weights on agentic work Performance on real work tasks Artificial … Read more

A Million Tokens of Context, and Six Turns Is All It Takes: Claude Opus 4.6 and Arabic Conversation

by
A chain of six links turning from clear glass into rough stone

Ehab Saleh | techkahwa.net Model released: February 6, 2026 | Published: February 20, 2026 The numbers first Metric Result What it measures Source Artificial Analysis Intelligence Index First place at release Composite general intelligence Artificial Analysis, February 2026 GDPval-AA First Real work tasks Artificial Analysis TerminalBench First Coding and execution Artificial Analysis CritPT 13% Research-level … Read more

75 Percent on an Arabic Leaderboard: What the Falcon-H1-Arabic Number Measures, and What It Hides

by
A brass scale where a small glowing cube outweighs a large cloud of vapour

Ehab Saleh | techkahwa.net Model released: January 5, 2026 | Published: January 19, 2026 The numbers first Model Size Context OALL Source Falcon-H1-Arabic 34B 34B 256K tokens ~75.36% Technology Innovation Institute, January 2026 Falcon-H1-Arabic 7B 7B 256K tokens 71.47% Same source Falcon-H1-Arabic 3B 3B 128K tokens ~62% Same source And the comparisons as the developer … Read more

A Trillion Arabic Phonetic Segments: Reading Fanar 2.0, the First Arabic Multimodal Stack

by
A dark sphere emitting three braided streams of light, panels and sound rings

Ehab Saleh | techkahwa.net Model released: December 9, 2025 | Published: December 26, 2025 The numbers first Item Number Source Model size 27B parameters, triple the original Fanar’s 9B Qatar Computing Research Institute, December 2025 Text training data More than 300 billion words Same source Speech training data More than one trillion Arabic phonetic segments … Read more

Arab Culture Is Not Islamic Knowledge: What Jais 2 and the PalmX Numbers Reveal

by
Two stone archways, one opening onto a library and the other onto a warm courtyard

Ehab Saleh | techkahwa.net Model released: December 9, 2025 | Published: December 23, 2025 The numbers first Metric Result What it measures Source PalmX 2025, Arab culture 72.15% for the winning system Traditions, customs, food, history, arts across 22 countries ArabicNLP 2025 PalmX 2025, Islamic culture 84.22% for the winning system Rituals, Quran, hadith, history, … Read more

An Arab Custom That Does Not Exist: The Adversarial Set on Claude Opus 4.5

by
An ornate key with completely smooth uncut teeth in front of an old lock

Ehab Saleh | techkahwa.net Model released: November 24, 2025 | Published: December 8, 2025 The numbers first Metric Result What it measures Source Global-MMLU-Lite Arabic 91 Arabic knowledge and reasoning Artificial Analysis Arabic index Context window 200K tokens Text processed in a single pass Artificial Analysis Price $5 / $25 per million tokens Operating cost … Read more