Naija Eval

About and method

Nigeria's own language model, N-ATLaS, tested in the open. We compare it with the Llama model it was built from and with four hosted models, on Hausa, Igbo and Yoruba speech and translation.

Extrafemi Labs · First pass, 8 October 2026

About

Numbers anyone can re-run

The N-ATLaS model card publishes only self-reported human ratings. Naija Eval publishes numbers anyone can re-run: the same public test sets, one fixed prompt, and the scripts on the Developers page. It is part of Extrafemi Research, and an Extrafemi Labs build.

Translation

  • N-ATLaS 4-bit, run locally
  • Llama 3.1 8B Instruct its base model, 4-bit, run locally
  • Claude Sonnet 5.5 hosted
  • Gemini 3.8 Flash hosted
  • GPT-6.1 Sol hosted
  • Claude Haiku 5.5 hosted

Speech to text

  • N-ATLaS Hausa, Igbo and Yoruba speech models fine-tuned Whisper Small
  • Stock Whisper Small openai/whisper-small, no Igbo
MethodReproducible

How we tested, and the limits

Data.
Google's FLEURS test set: read speech of FLORES sentences, which are also aligned across languages for translation. CC BY-SA 4.0.
Speech.
NCAIR1 Hausa, Igbo and Yoruba ASR models (fine-tuned Whisper Small) against stock openai/whisper-small, greedy decoding.
Translation.
One fixed prompt for every model, temperature 0, first line of output kept. N-ATLaS and Llama 3.1 ran locally in 4-bit on an Apple M5 with MLX; hosted models ran through OpenRouter.
Sample size.
100 items per language and direction. This is a first pass, not a leaderboard.
Quantization.
The 4-bit local models may score a point or two below full precision.
Contamination.
FLEURS and FLORES are public, so any model here may have seen them in training.
Real speech.
FLEURS is clean read speech with formal vocabulary. Street voice notes will score differently.
Scores

What the two scores mean

Speech is scored by word error rate: the share of words the transcript gets wrong, so lower is better. Translation is scored by chrF++, from 0 to 100, where higher is better. Both are also shown ignoring tone marks, so missing or wrong marks can be told apart from wrong words.