Benchmark: language detection on real marketing copy
We tested the offline trigram engine behind Lingua Detective against the hardest corpus we could find: this site's own marketing copy — short strings, brand names, prices and technical terms, in 36 languages. Everything below is reproducible with one script.
Methodology
The corpus is the real i18n copy served on this site (36 locale files, every user-facing string). Two trials per language:
- Full page — all strings joined (≈ one rendered page): is the language detected at rank 1? In the top 5?
- Short string — each individual string of ≥ 25 letters (3,701 samples): is it detected at rank 1?
Brand tokens, URLs and placeholders are stripped first. Engine: franc 6.2 (offline trigrams; the same model the production /detect endpoint runs). Script and raw output are public: benchmark-detection.mjs · benchmark-detection.json. Raw JSON uses French field names; English aliases are not available. Run date: 2026-09-13.
What the numbers say
Full pages are reliable; short strings are genuinely hard. On page-length text the engine names the right language first in 29 of 36 cases, and puts it in the top 5 in 30. On isolated short strings — a button label, a tagline — top-1 accuracy drops to 64.3%. That gap is exactly why Lingua Detective never judges a page from one fragment: it reads a short sample, then the full text with the repeated template removed, and only then issues a verdict (with AI arbitration on doubtful pages).
Where the raw engine misses (full pages)
| Language | Detected as | Why it happens |
|---|---|---|
| Croatian (hr) | Serbian (srp) | Sister languages share most trigrams — the miss is between mutually intelligible neighbors, not random noise. |
| Malay (ms) | Indonesian (ind) | |
| Irish (ga) | Scots (sco) | Small training footprint + mixed English loanwords in marketing copy. |
| Icelandic (is) | Swedish (swe) | Low-resource languages against marketing vocabulary full of technical English terms. |
| Luxembourgish (lb) | Low German (nds) | |
| Māori (mi) | Swahili (swh) | |
| Maltese (mt) | Norwegian Nynorsk (nno) |
Note: franc distinguishes fine-grained codes from macrolanguages (ekk for Standard Estonian, lvs for Standard Latvian) — we count those as correct.
Why we still run an offline engine
- Privacy — your pages' text is never sent to a third-party AI service for detection.
- Speed and cost — 0.39 ms per call means a 200-page audit is detected in milliseconds, which is what makes $0.002-per-call pricing viable.
- Determinism — the same input always gives the same answer; doubtful pages escalate to the arbitration pass instead of silently guessing.
Reproduce it
npm install franc
node benchmark-detection.mjs # against a local checkout of this site's locale filesThe corpus is this site's own public locale files (/i18n/locales/{en,fr,es,…}.json). The script prints the per-language table and writes the raw JSON.