A standing series of independent evaluations of how large language models actually handle Arabic: Modern Standard Arabic, the dialects, accents, culture and pragmatics. Every article leads with published, sourced numbers and a reusable test protocol.
Ehab Saleh | techkahwa.net Paper released: September 21, 2023 | Published: October 5, 2023 The numbers first Metric Result What it measures Source AceGPT-13B-chat on Arabic Vicuna-80 100.88% of GPT-3.5’s performance Arabic instruction following, judged by GPT-4 AceGPT paper, NAACL 2024 AceGPT-13B-chat on Arabic AlpacaEval 97.95% of GPT-3.5 Same measure, different set Same paper Jais-13B-chat … Read more
Ehab Saleh | techkahwa.net Model released: September 6, 2023 | Published: September 20, 2023 The numbers first Metric Result What it measures Source Model size 180 billion parameters The largest open model of its moment Technology Innovation Institute press release, September 6, 2023 Training data 3.5 trillion tokens Also the largest Same release Languages stated … Read more
Ehab Saleh | techkahwa.net Model released: August 30, 2023 | Published: September 13, 2023 The numbers first Metric Result What it measures Source Total training tokens 395 billion The model’s overall scale Jais paper, arXiv:2308.16149, August 2023 Original Arabic 55 billion How much genuine Arabic went in Paper, Table 3 Arabic after adding translated text … Read more
Ehab Saleh | techkahwa.net Model released: August 22, 2023 | Published: September 5, 2023 The numbers first Metric Result What it measures Source Modern Standard Arabic, speech in Supported Whether it hears you Official language table, seamless_communication repository Modern Standard Arabic, speech out Supported Whether it speaks to you Same table Egyptian Arabic, speech in … Read more
Ehab Saleh | techkahwa.net Model released: July 18, 2023 | Published: August 1, 2023 The numbers first Metric Result What it measures Source Arabic share of Llama 2 pretraining data Not listed, meaning below 0.005% How much Arabic entered the model at all Llama 2 paper, Table 10, July 2023 English share 89.70% The scale … Read more
Ehab Saleh | techkahwa.net Model released: March 14, 2023 | Published: March 28, 2023 The numbers first Metric Result What it measures Source GPT-4 on MMLU in Arabic 80% General knowledge, on a machine-translated exam GPT-4 Technical Report, Figure 5, March 2023 GPT-4 on MMLU in English, same condition 85.5% The reference the Arabic number … Read more
Ehab Saleh | techkahwa.net Model released: November 30, 2022 | Published: December 14, 2022 The numbers first Metric Result What it measures Source Cost of encoding Arabic against English 3 times Tokens needed for the same text carrying the same meaning Petrov, La Malfa, Torr and Bibi, NeurIPS 2023 Cost of Italian against English 1.6 … Read more
Ehab Saleh | techkahwa.net Work published: May 2022 (ACL 2022) | Published: June 5, 2022 The numbers first Metric Result What it measures Source AraBERT, training data 70 million sentences, about 24 GB The size of the Arabic input to the first serious Arabic model Antoun, Baly and Hajj, OSACT4 workshop, May 2020 AraBERT on … Read more