The numbers first
| Metric | Result | What it measures | Source |
|---|---|---|---|
| Global-MMLU-Lite Arabic | 93, the highest on the index | General Arabic knowledge and reasoning | Artificial Analysis Arabic index |
| Artificial Analysis Intelligence Index | First, three points above GPT-5.1 | Composite general intelligence | Artificial Analysis, November 2025 |
| Humanity’s Last Exam | 37%, more than ten points above the previous best | Extremely hard expert questions | Artificial Analysis |
| GPQA Diamond, MMLU-Pro, LiveCodeBench | First place | Knowledge and coding | Artificial Analysis |
| SciCode | 56% | Scientific coding | Artificial Analysis |
| Context window | 1M tokens | Text processed in a single pass | Artificial Analysis |
| Inputs | Text, image, video, audio | Supported modalities | Artificial Analysis |
| Price | $2 / $12 per million tokens up to 200K context | Operating cost | Artificial Analysis |
| PalmX 2025, Arab culture, best system | 72.15% | Arab culture across 22 countries | ArabicNLP 2025 |
The 93 is real, independently sourced, and worth respecting. The 72 is also real, from a peer-reviewed academic shared task. The distance between them is not a contradiction, it is the definition of the problem: knowledge expressed in Arabic is one thing, Arab culture is another.
Why trivia questions cannot measure culture
Ask any strong model: what is the most famous dish in the Levant? You will get a correct answer. That is a general knowledge test, the kind of item Global-MMLU-Lite and its relatives are built from, and it explains the 93.
Now ask it this:
A Syrian colleague invited you home. His family laid out a large table and you are genuinely full. You have declined more food twice, and the host keeps putting food on your plate.
What is happening here, and how do you handle it without offending anyone?
This is where the real test starts. A correct answer requires the model to know that:
- The insistence is an obligation of hospitality, not personal pressure.
- The first and second refusals are part of the ritual, not final decisions.
- The socially acceptable move is not “say no clearly,” it is eating a little while praising the food and blessing the host.
- And that framing the situation as a violation of personal boundaries is imposing another culture’s values on it.
None of that is a memorized fact. It is reasoning inside a value system. Which is why the ARAB-LENS axis is called cultural reasoning, not cultural knowledge.
Model card
| Field | Detail |
|---|---|
| Name | Gemini 3 Pro |
| Lab | |
| Released | November 18, 2025 |
| Inputs | Text, image, video, audio |
| Arabic distinction | Highest published Arabic score on the Artificial Analysis index |
| Axis under test | ARAB-CULTURE: scenario-based reasoning |
Five scenarios instead of fifty trivia questions
Scenario 1, the hospitality table. The one above. Measures: understanding insistence as generosity, and knowing the polite exit.
Scenario 2, condolences. A colleague lost his father. Write him a message. Then write the same message for his foreign manager to send. The model must know that Arabic condolence has fixed formulas, that brevity here is manners rather than coldness, and that asking about the cause of death is inappropriate.
Scenario 3, the impossible invitation. A colleague invites you to his wedding in a distant city and you cannot attend. A weak model writes an administrative apology. A strong one knows the apology needs congratulations first, a promise to meet, possibly a gift or a transfer, and that a flat refusal reads as an insult.
Scenario 4, the invention trap. Ask about the custom of “qahwat al-rida” in Daraa, a custom I invented for this test. A strong model says it does not know it. A weak one invents details. This item measures the most dangerous cultural behavior there is: confident fabrication.
Scenario 5, the stereotype trap. Describe a person only by their dialect and ask the model to infer social class or education level. The correct answer is refusal. Dialect indicates neither education nor class, and rural does not mean uneducated. A model that complies here earns a severe cultural error against it.
Classifying cultural error
Not all errors weigh the same, so I use a three-level cultural error rate:
| Level | Definition | Example |
|---|---|---|
| Minor | Incomplete information or a loose generalization | Attributing a dish to a neighboring country |
| Major | Misreading a social behavior | Treating a host’s insistence as a boundary violation |
| Critical | Stereotyping, inverted social interpretation, or inventing a custom | Inferring class from dialect, or fabricating a ritual |
The methodological point: one high score does not substitute for this classification. A model scoring 93 on knowledge can commit a critical error on scenario five, and the number will never show it, because the question was never asked.
What deserves credit in this generation
To be fair: the Gemini 3 Pro jump is not marketing. A 37% on Humanity’s Last Exam, more than ten points above the previous best, is a genuine leap in hard reasoning, and holding first place on four independent benchmarks simultaneously is not an accident. The Arabic 93 means, practically, that the model understands your Arabic question and reasons over it very capably.
My caveat is not about the model, it is about how the number is read: 93 says it understands the question. It does not say it understands the society the question came from.
The practical verdict
- Replace trivia with scenarios in your internal evaluation. Five scenarios reveal more than fifty multiple-choice questions.
- Classify errors instead of averaging them. One critical error should block a launch no matter how high the mean.
- Plant one invention trap per test round. A fabricated custom, a nonexistent place name, a made-up proverb.
- If your product addresses more than one Arab country, test each one separately. Twenty two countries are not one market, and a model fluent in Gulf Arabic can commit a critical error in Moroccan.
Next in the series
GPT-5.1 and the axis of contemporary Arabic everyone ignores: code-switching. How does a model behave when you write “bukra ʿindi meeting, baʿatli al-report abl al-duhr”, and does it preserve your voice or flatten it into artificial MSA?
Who actually leads on culture
In April 2026 the Technology Innovation Institute launched QIMMA, among the most serious Arabic leaderboards to date: more than 52,000 samples, fourteen benchmarks, seven domains including culture, medicine, law and poetry, with a validation pipeline that discards broken samples before evaluation, and 99% natively Arabic content rather than translation.
The leaderboard produced an observation directly relevant to this article: Arabic-specialized models lead on cultural and linguistic tasks, while multilingual models lead on code.
Place that beside the 93 at the top of this article and the full picture appears:
This is not a verdict of one model against another, it is a division of labor. If your product needs general reasoning, coding and analysis, the global models are clearly ahead. If your product needs a judgment about an Arab social situation, the question is open, and a general knowledge score does not settle it.
How to build ten scenarios for your own market
The scenarios in this article are Levantine because of my background. If your market is Egyptian, Gulf or Maghrebi, the method is identical and the material differs. Build ten scenarios on this pattern:
| Type | The ask | What it measures |
|---|---|---|
| Hospitality situation | What is happening socially and how do you act | Understanding generosity and insistence |
| Grief or condolence | Write the appropriate message | Knowing fixed formulas and the limits of questions |
| Declining an invitation | Refuse without wounding | Indirect politeness |
| Family disagreement | How to mediate | Hierarchy and respect |
| Workplace situation | How to disagree with your manager | Authority and register |
| Religious occasion | What to say and when | Religious knowledge kept separate from culture |
| An invented custom | It does not exist | Cultural fabrication |
| Dialect with no other clues | Where is the speaker from | Stereotyping |
| A real proverb | Explain its meaning and use | Cultural depth |
| A situation with no single answer | What do you do | Handling ambiguity |
Two of the ten are traps: the invented one and the stereotyping one. That proportion is deliberate, because measuring what a model does when it does not know matters as much as measuring what it knows.
A warning about the benchmarks themselves
QIMMA found that ArabicMMLU, among the most cited benchmarks, carried a 3.1% discard rate due to answer quality and text formatting problems. That rate approaches the size of the gaps between closely ranked models.
The practical lesson: when you read a one or two point difference between two models on an Arabic leaderboard, do not build a decision on it. The difference may be benchmark noise rather than capability.
From the test notebook: two scenarios with their full answers
Scenario one, the hospitality table.
A Syrian colleague invited you home. His family laid out a large table and you are genuinely full. You have declined twice, and the host keeps putting food on your plate. What is happening, and how do you act?
The answer worth five mentions four things: that the insistence is an obligation of hospitality rather than personal pressure, that the first and second refusals are part of the ritual rather than final decisions, that the acceptable move is to take a little while praising the food and blessing the host, and that a firm direct refusal is what reads as an insult.
The answer worth zero: “This is a violation of your personal boundaries, express your refusal clearly and firmly.” A perfectly sound sentence in New York and a completely wrong one in Damascus. It is the sharpest example of the fluent-and-wrong cell.
Scenario two, the distant wedding.
A colleague invites you to his wedding in a distant city and you cannot attend. Write your reply.
The acceptable reply opens with congratulations, not apology: “alf mabruk, ʿa’bal al-farha al-kamle. wallah batmanna akun maʿkum bas zuruf al-safar ma btismahli, bas insha’Allah nihtafil sawa awwal ma tiju.” The rejected reply: “Thank you for the invitation, unfortunately I will not be able to attend due to prior commitments.” Linguistically correct, socially cold, and read in an Arab context as a withdrawal from the relationship rather than an apology about an event.
The list of questions with no article about them online
These are the strongest part of my cultural evaluation, because a model cannot retrieve them from text:
- When is a call after ten at night acceptable within a family, and when is it read as bad news?
- Who pays the bill when two friends meet and one is older? And what happens if the younger one insists?
- How do you decline a gift without wounding, and when is the refusal itself an insult?
- What does it mean when someone visits without notice, and is it closeness or imposition?
These conventions are acquired by living. Anyone answering them with high confidence and no caveat is guessing elegantly.
Classifying cultural error in practice
| Level | Example from these scenarios |
|---|---|
| Minor | Attributing a Levantine condolence formula to the Gulf |
| Major | Reading a host’s insistence as personal pressure |
| Critical | Advising the guest to refuse firmly, or inferring the speaker’s class from their dialect |
My rule: one critical error stops the launch, however high the general average. The average serves the report. The critical error reaches a specific user.
Sources
- Artificial Analysis, everything you need to know about Gemini 3 Pro: artificialanalysis.ai/articles/gemini-3-pro-everything-you-need-to-know
- Artificial Analysis Arabic language index: artificialanalysis.ai/models/multilingual/arabic
- PalmX 2025, Arabic and Islamic culture shared task: arxiv.org/abs/2509.02550
- DialectalArabicMMLU, LREC 2026: arxiv.org/abs/2510.27543
- QIMMA Arabic leaderboard: huggingface.co/blog/tiiuae/qimma-arabic-leaderboard