The numbers first
| Metric | Result | What it measures | Source |
|---|---|---|---|
| Artificial Analysis Intelligence Index | 53 | Composite general intelligence, level with Claude Fable 5.1 | Artificial Analysis, September 2026 |
| Terminal-Bench v4.0 | 59% | Command line task execution | Artificial Analysis |
| AutomationBench-AA | 69% | Multi-step automation | Artificial Analysis |
| AA-Omniscience hallucination rate | 51%, down from 92% | Tendency to invent confidently | Artificial Analysis |
| Last Translation Benchmark | 44 to 45% | Hard translation cases | Slator, September 2026 |
| M-GATE (30 languages) | Ahead on most, behind its predecessor on Polish, Japanese, Thai, Finnish | Cross-language performance | Slator, citing M-GATE |
| Any published Arabic metric | None | The model’s Arabic capability | OpenAI launch materials |
That table is the whole article in one image. A model its own lab calls a generational leap, measured by the world on the command line, on automation, and across thirty languages, carries not a single number telling you how it behaves when three hundred million Arabic speakers talk to it.
Why the gap matters to me
Years ago, teaching spoken Arabic to American staff at a shelter in Tucson, one of them asked me something that sounds simple. A Syrian colleague had told him “dabbir halak.” Was that encouragement, or was he being cut loose? I could not answer in one word. The answer is not in the phrase, it is in who said it, when, and in what tone.
That is what I look for in every new model. Not how many problems it solved, but whether it can tell dabbir halak as reassurance from dabbir halak as a door closing. Nobody measures this. So the model ships with a 53 on an intelligence index, and an Arabic product team builds on an assumption that has never been tested.
Even the language numbers we do have deserve a careful read. Passing 44 to 45% of hard translation cases is not the profile of a system that has passed human ability, and losing ground against the previous version in four languages says an overall gain is not a gain everywhere. Human translators remain ahead exactly where the judgment is hard, and that region of difficulty is where everyday Arabic lives.
Model card
| Field | Detail |
|---|---|
| Name | GPT-6 Astra |
| Lab | OpenAI |
| Limited preview | September 3, 2026 |
| General availability | September 4, 2026 |
| Vendor claim | Generational leap in coding, math, science, cybersecurity |
| Arabic claim | None |
| Price | $10 per million input tokens, $50 per million output |
| Axis under test | ARAB-PRAG: intent, implicature, politeness, ambiguity |
What everyone else’s numbers say about Arabic
Since this model has no Arabic number, read the serious Arabic measurements for the whole category. These are published figures with sources, and they are the backdrop against which any launch claim should be read.
| Measurement | Number | What it means |
|---|---|---|
| DialectalArabicMMLU (LREC 2026) | 47.7% on dialects vs 51.9% MSA vs 62.8% English | The gap is not model to model, it is the language against itself |
| Syrian dialect in the same benchmark | 46.6% | Below Egyptian, Emirati and Saudi |
| Translating a dialect question into English | +5.6 points | The model understands through English, not through Arabic |
| Translating it into MSA | +0.3 points | MSA is not a bridge to dialect |
| NADI 2025, speech recognition | 35.68 WER for the best system | More than a third of spoken words misheard |
| NADI 2025, spoken dialect ID | 79.8% | Acceptable for a dialect family, not for a region |
| PalmX 2025, Arab culture | 72.15% winning score | Culture is harder for models than religious knowledge (84.22%) |
The third and fourth rows are, to me, the most important result published in Arabic evaluation this past year. Translating a question from dialect into Modern Standard Arabic barely helps a model, while translating it into English does. Which means that what we call “Arabic capability” in many systems is really English capability passing through Arabic on its way to an answer.
The test you will not find on any leaderboard
Every item below I wrote directly in Levantine Arabic. None of it was model-generated. That is a fixed rule across ARAB-LENS: gold data is authored by native speakers, and models are tested against it, never used to produce it.
Item 1, one phrase in two situations. Situation A: a friend cancels, and his friend replies “wala yhimmak, dabbir halak w baʿdein khabbirni.” Situation B: an employee asks for an extension, and the manager replies “ana ma ili dakhal, dabbir halak.” Gold: reassurance in the first, refusal of responsibility with an edge in the second. Trap: the same reading twice, or “take care of yourself” for both.
Item 2, “insha’Allah bukra.” Ask for the possible readings ranked, plus the missing information needed to decide. Gold: at least four readings, a real plan, an uncertain maybe, a polite deferral, and a social formula that closes the conversation. Trap: committing to one.
Item 3, “Allah yaʿtik al-ʿafiya” in four situations. To a worker unloading furniture, at the end of a long meeting, to a friend who ran an errand for you, and from an employee to a colleague three days late. Functions in order: acknowledging effort, a polite close, personal thanks, sarcasm. Trap: reading all four as a wish for health.
Item 4, the literalism trap. “ʿala rasi” after a favor is asked. The correct translation is functional, not literal. “On my head” is a complete failure.
Item 5, the uncertainty trap. A colleague sends one word on WhatsApp: “tayyib.” Agreeing or annoyed? Gold: say plainly that the evidence is insufficient. Any confident verdict is wrong here, because the strong model is not the one that always answers, it is the one that knows when it cannot.
Item 6, multi-turn. Six turns between two friends, one stuck in traffic, then questions about the relationship, the tone, and what wala yhimmak means in the final turn. Trap: reading it as indifference rather than reassurance.
Six items like these expose what no multiple-choice set can, because they ask about function, not definition.
The three failure patterns
| Pattern | How it shows up | Why it is dangerous |
|---|---|---|
| Fluent and wrong | Polished Arabic explaining an unintended meaning | Only an attentive native speaker catches it |
| False literalism | Linguistically correct, socially wrong | The product breaks on its first real user |
| Confident with no evidence | A decisive answer to an undecidable question | It manufactures false trust |
Cutting the AA-Omniscience hallucination rate from 92% to 51% is good news for the third pattern. But that is a measurement of general knowledge, mostly in English, and it says nothing about how the model behaves when asked about a Levantine social custom it has never seen.
The practical verdict
A verdict on GPT-6 Astra in Arabic cannot rest on a number today, because the number does not exist. That absence is the finding: anyone shipping an Arabic product on this model is building on an untested assumption.
Three practical recommendations:
- Do not assume the model reads intent. Make the interface ask for what it needs instead of inferring it, especially in support, health and education.
- Test in your own dialect before launch. Six items like the ones above separate a model that understands from one that performs, and they cost you ten minutes.
- Do not read a general intelligence score as an Arabic score. The distance between 53 on a general index and 47.7% on dialect questions is the distance between what is announced and what happens to your user.
Next in the series
From comprehension to production. I will measure whether Claude Fable 5.1 can actually write Syrian Arabic, rather than Modern Standard Arabic decorated with shu and halla’, using a dialect fidelity index across six levels.
What the field and the research community report
The absence of Arabic numbers from the major labs does not mean nobody measured. Arab researchers did, and their results are harsher than the leaderboards suggest.
The most important work here is AL-QASIDA, a systematic evaluation of dialectal Arabic quality covering nine models including GPT-4o, Command-R and Llama 3.1 alongside Arabic-specialized systems, across eight dialects including Syrian, Palestinian, Egyptian and Moroccan. Its central finding inverts the familiar picture:
Models produce responses that are fluent and adequate in meaning, yet not in the requested dialect most of the time.
Notice exactly what that sentence means. Understanding outpaces generation, which reverses the usual pattern in generative AI where production runs ahead of comprehension. The model understands your Levantine, then answers you in something else.
The study added a number worth memorizing: in monolingual settings, nearly all ADI2 scores fell below 50%, and GPT-4o itself did not clear 13% on half the dialects tested in the cross-lingual setting. ADI2 here is an automatic measure of how far generated text actually belongs to the requested dialect.
Among practitioners, the recurring complaint in Arabic translation and localization circles arrives in almost identical form every time: literal rendering of idioms, confusion in the face of undiacritized text, plural agreement errors, and weak handling of code-switching. These are not matters of taste. They are precisely the failure patterns the items in this article are built to catch.
I read the overlap between what the research measures and what practitioners complain about as a sign that the problem is structural rather than incidental. It is not a flaw in one model, it is a gap in the kind of data all of these systems learned Arabic from.
How to compute your own false literalism rate
The metric is simple and needs no tooling:
| Step | What you do |
|---|---|
| 1 | Collect ten common social phrases from your dialect |
| 2 | Place each in two different contexts, giving you twenty items |
| 3 | Ask the model for the meaning and social function, not a translation |
| 4 | Mark every response that translates correctly and misses the function |
| 5 | Divide that count by twenty |
The result is a single percentage that you recompute for every new model and store with its date. After four models you hold something no public leaderboard gives you: a time series of how well models read the intent behind your dialect.
From the test notebook: three fully worked items
Here are three items exactly as I send them, with the answer I accept and the answer I reject written out in full, so the judgment is not left to taste.
Item one: dabbir halak in two positions.
Situation A: Ahmad calls his friend: “I can’t come with you today, something urgent came up.” The friend replies: “wala yhimmak, dabbir halak w baʿdein khabbirni.”
Situation B: An employee asks his manager for a two-day extension. The manager: “ana ma ili dakhal, dabbir halak.”
The answer I score five out of five: “In the first it is reassurance that lifts the social pressure, meaning arrange your time and do not apologize, and the tone is warm. In the second it is a refusal of responsibility with an edge, meaning handle it yourself, this is not my concern.”
The answer I score zero: “It means take care of yourself and sort out your affairs”, repeated for both. That is dictionary-correct and socially inert.
Item two: the pausal form of ta marbuta, an accent test rather than a meaning test.
Write this sentence as it is pronounced in pausal form: “qara’tu risalata al-mudira.”
An Arabic speaker stops on ta marbuta as an h: al-mudirah. A model that writes al-mudirat, or insists on a case vowel in pause, reveals that it learned Arabic from written text rather than from speech. This single item is the fastest separator I have between a model that read Arabic and one that heard it.
Item three: qaf, the clearest accent marker in the Mashriq.
Here are four pronunciations of the word qal: qal, ‘al, gal, kal. Classify each and say where it is heard.
The answer I accept: qal with qaf in MSA and in coastal and rural areas and among some Levantine communities, ‘al with a glottal stop in Damascus, Beirut and Levantine urban centers, gal in the badia, northern Jordan, Iraq and the Gulf, and kal in parts of rural Palestine and Iraq.
The trap here is double: a weak model either treats anything that is not a glottal stop as “Gulf”, or builds a judgment about the speaker on the pronunciation. I log both as errors, the first as linguistic ignorance and the second as a critical error.
Why I always start from sound rather than meaning
Because accent is more honest than vocabulary. Any model can memorize a list of Levantine words, but few know that a Damascene raises the long a toward i in words like bab and nas, that the feminine ending is realized as a raised vowel in shajara and madrasa, and that this is exactly what makes a correctly written sentence sound foreign when read aloud.
So in every evaluation I run, the first item is phonetic and the last is social, and everything the leaderboards measure sits in between.
Sources
- Artificial Analysis, benchmarking GPT-6 Astra: artificialanalysis.ai/articles/benchmarking-gpt-6-astra
- Slator, Astra translation performance and M-GATE: slator.com
- OpenAI, GPT-6 Astra launch page: openai.com/index/gpt-6-astra
- DialectalArabicMMLU, LREC 2026: arxiv.org/abs/2510.27543
- NADI 2025 multidialectal Arabic speech shared task: arxiv.org/abs/2509.02038
- PalmX 2025 Arabic and Islamic culture shared task: arxiv.org/abs/2509.02550
- AL-QASIDA, analyzing LLM quality in dialectal Arabic: arxiv.org/abs/2412.04193