Naija Eval

Developers

Everything here can be re-run: the same public test sets, one fixed prompt for every model, and a script for each step.

Quick start

Reproduce it

Accept the NCAIR1 model terms on Hugging Face before you log in.

  1. pip install -r requirements.txt
  2. hf auth login# accept the NCAIR1 model terms first
  3. python eval/fetch.py all# about 22 GB
  4. python eval/eval_asr.py 100
  5. python eval/eval_mt.py models/natlas-4bit natlas 100
  6. python eval/eval_mt_api.py anthropic/claude-haiku-5.5 haiku55 100
  7. python eval/summarize.py
Full run

Every model, step by step

The 4-bit conversion and both local translation runs need Apple Silicon (MLX). The speech test runs anywhere PyTorch does.

  1. python3 -m venv .venv && . .venv/bin/activate
  2. pip install -r requirements.txt
  3. hf auth login# read token; accept the NCAIR1 model terms first
  4. python eval/fetch.py all# ~22 GB: ASR models, FLEURS, N-ATLaS (16 GB), Llama 3.1 4-bit
  5. mlx_lm.convert --hf-path models/N-ATLaS --mlx-path models/natlas-4bit -q --q-bits 4
  6. python eval/eval_asr.py 100
  7. python eval/eval_mt.py models/natlas-4bit natlas 100
  8. python eval/eval_mt.py models/llama31-4bit llama31 100
  9. python eval/eval_mt_api.py anthropic/claude-sonnet-5.5 sonnet55 100# hosted models: put OPENROUTER_API_KEY=... in ~/.config/naija-eval/.env
  10. python eval/summarize.py
What it takes
Downloads.
About 22 GB: the speech models, FLEURS, N-ATLaS (16 GB) and Llama 3.1 in 4-bit.
Hardware.
Local translation ran on an Apple M5 in 4-bit so the models fit in 16 GB of memory.
Hosted cost.
$1.59 for all four hosted models, 600 translations each, through OpenRouter.
Licences.
You download the models and data from their owners and must follow their licences. See Legal and licences and the terms.
Data

The results file

This is the file the report is built from: every score, with and without tone marks, and the example translations.

summary.jsonScores for 7 translation models and 3 speech languages · 60 KB
mt
One entry per model: name, where it ran, items per direction, scores per direction, scores ignoring tone marks, average, hosted cost in US dollars, and example translations.
asr
Per language: number of clips, then word and character error rates for N-ATLaS and stock Whisper, with and without tone marks, and speed as a real-time factor.
pairs
The six translation directions, for example en->ha.
scorecard.jsonThe same scores in a shared format, for tools that choose between models
scores
One row per model, task, metric and language: the score, whether higher is better, how many items it rests on, and the day it was measured. Hosted models are named by their OpenRouter id. Models run locally are named local/ and cannot be routed.
Using it
Lodestone, an open-source library for calling AI models, can read it to try the models that did best on a language first. The format is described in its repository.