Benchmark: engine 0.2.0
Measured once on a held-out test split separated by prompt and by author. Rows in italics come from sources excluded from training entirely. Small grey figures are 95% confidence intervals.
For comparison, the hand-tuned v0.1 rules caught 46.5% of AI texts while flagging 3.7% of human texts, including 17.8% of human student essays.
Limitations. Benchmark AI text was produced in 2023 by GPT-3.5 and Claude. Performance on GPT-4o/5, Claude 3+, Gemini, DeepSeek and Llama has not yet been measured. Generalisation to unseen sources is weak. An earlier non-native-writer figure relied on corpora with restrictive licences, so it has been withdrawn until we have a commercially licensed evaluation set.
Deep Scan (deep-0.7.0, ALETHEIA V1.0)
Deep and Supreme scans combine a character- and word-level n-gram model with the standard document model. Training text comes from 120+ language models released between 2023 and 2026 (short fiction from public creative-writing benchmarks, including GPT-5.x, Claude 4–5, Gemini 2.5–4, DeepSeek, Kimi, Qwen, Grok, Llama and Mistral), real ChatGPT answers (WildChat), current-model essays, news and abstracts (DACTYL), Q&A (HC3), synthetic textbooks and guides (Cosmopedia), adversarial sets (DetectRL, RAID: paraphrasing, back-translation, synonym swaps, misspellings) and human writing from public-domain fiction, Wikipedia, learner essays and news. A small genre model sorts each text into one of three registers, each with its own threshold: academic abstracts (strictest), creative writing (calibrated on human fiction) and everything else. In V1.0 the third register gets a second model trained only on non-fiction writing; the two models vote, and the combined score was calibrated again. Every threshold was set on a calibration split so that at most 1% of human texts in any group are flagged.
Compared with deep-0.4.0, V1.0 finds more AI text in every held-out group (questions and answers 1.6% → 42.7%, real ChatGPT answers 46.6% → 55.5%, current-model news 41.5% → 56.8%) and leaves fiction and academic abstracts unchanged. Two earlier candidates that retrained a single model on all genres lost ground on GPT-5.5 fiction and were not released.
For comparison, the standard 2023-era model found only 0–13% of modern AI fiction. The Standard mode now consults the n-gram model too: it gives an AI verdict when that model clears its register's threshold, and withholds one when it disagrees. Line-numbered texts (monologues, scripts) are now read as prose; previously their lines were excluded as list items.
Weak spot: older and open models in non-fiction. Wikipedia-style articles, how-to guides and forum answers written by older or small open models (BLOOMz, Dolly, Flan-T5) are rarely detected (0–5%); ChatGPT-written ones are found 12–39% of the time. Human writing in these styles is almost never flagged (0–0.3%).
Open risk: prompt-following human writing. On Beemo, a set of answers written in 2023 by paid annotators to chatbot-style prompts, ALETHEIA V1.0 gives an AI verdict on 12% of the human stories and poems and 16% of the human rewrites (deep-0.4.0: 9% and 16%). These texts score far higher than any other human writing we have tested, so either the detector over-flags this style or some of the "human" answers were written with AI help, a known problem with crowdsourced data. Until we can tell which, treat a positive result on short writing produced to a prompt with extra caution.
What still gets through. Text run through AI "humanizers" and back-translation is detected far less often (back-translation about 30%, DIPPER paraphrasing about 50%, humanized o1-pro news articles 57%), and some highly literary single samples score in the review range rather than above the threshold. We report those as Inconclusive instead of lowering thresholds, because lowering them would flag more human writers.
AI percentage and highlighted segments (seg-0.2, ALETHEIA V1.2)
Deep and Supreme scans report an AI percentage: the share of qualifying words in sections that look AI-generated, the measure Turnitin popularised. The document is cut into overlapping segments of about 250 words; a segment model scores each one, and each sentence takes the average of the segments it belongs to. Values from 1% to 19% are shown as *%, as Turnitin does, because false positives are more likely in that range. The whole-document deep score is still shown; the two answer different questions.
V1.2 retrains the segment model with 2026 material: texts from 33 generators released up to late 2026 (wasitaigenerated benchmark), the output of 12 commercial "humanizers" (AI Humanizer Benchmark, September and October 2026), and student essays at eight levels of AI involvement (GEDE). Every test below uses generators, humanizer cycles or essay prompts the model never saw.
| Held-out test | V1.1 | V1.2 |
|---|
| 2026 AI text from unseen generators (GPT-6, Claude Opus 5.5, Gemini 3.7 …) reaching 20% AI | 22% | 29% |
| Humanizer output, unseen November 2026 cycle | 27% | 51% |
| Student essays run through a paraphrasing "humanizer" (DIPPER) | 26% | 41% |
| Human documents reaching 20% AI, 9,008 texts in 14 groups | 0.80% | 0.71% (0.11% without Beemo) |
| Human student essays (OUTFOX/PERSUADE, ELLIPSE, GEDE/BAWE) and 2026 human texts | 0–1.0% | 0–0.6% |
| Real student essays, grades 6–12, from prompts never used in training or calibration (PERSUADE 2.0) | 0 of 3,940 | 0 of 3,940 |
| Part-human, part-AI documents: highlighted words that really were AI | 88% | 91% (20% of AI words found) |
| Older (2023–24) AI test sets reaching 20% | 66.5% | 62.4% |
V1.2 trades a few points on older AI text for large gains on current models and humanizers. It flags fewer human texts overall (12 of 8,425 outside Beemo with V1.1, 9 with V1.2); one group got slightly worse (M4 human answers: 3 → 5 of 1,697). Thresholds were calibrated on human writing (at most 1% of documents in any group reaching 20%), then raised by a small fixed margin after one human résumé crossed the line in testing.
Where we stand. On short texts from the newest frontier models, independent 2026 benchmarks report Pangram and GPTZero catching about 80% at near-zero false positives, while ALETHEIA's whole-document verdict catches about 21% and its AI percentage flags 29%. Against humanizers no detector is reliable: on the same outputs GPTZero, Originality and Copyleaks range from 0% to 88% depending on the tool. ALETHEIA runs small statistical models in your browser and never sees your text on a server; closing the gap on frontier models needs a larger server-side model, which is planned. Until then, a low score on a short text is weak evidence of human authorship.
Earlier finding: academic abstracts
deep-0.2.0 flagged 25.7% of 1,834 human academic abstracts in an independent corpus (Mendeley Data, 2025), and the 2023 standard model flagged 11.1%. Both were withdrawn for that register; it now has its own, much stricter threshold.
How other detectors and "humanizers" work
Turnitin splits a submission into overlapping windows of a few hundred words, scores every sentence 0–1 with a fine-tuned transformer classifier built on an open-source foundation model, and averages the scores. It requires at least 300 words, shows scores of 1–19% only as an asterisk, and accepts missing up to about 15% of AI text to keep false positives below 1% on documents with more than 20% AI writing. A second model runs only on segments already flagged as AI and labels those that look AI-paraphrased (QuillBot and similar). Turnitin added "AI bypasser" detection for humanized text in 2025.
Humanizers (HumanizeAI.pro, Undetectable AI, StealthGPT and many more) rewrite AI output to change its statistics. An audit of 19 such tools found synonym swaps, sentence splitting, shifts in reading level, deliberate grammar and punctuation errors, typos, inserted personal pronouns and nonsense insertions; HumanizeAI.pro was among the most fluent. Detectors that rely on perplexity alone lose most of their accuracy on such text, while detectors trained on humanized examples recover most of it, provided human text run through the same tools is also labelled human so the detector learns the authorship signal rather than the rewriting.
Our gap. ALETHEIA runs entirely in your browser, with no transformer language model and no humanized training data from commercial tools. Closing that gap needs a server-side model and licensed humanizer data, which is on the roadmap below.
An honest note on time. A longer scan is more accurate only because it runs more analysis, not because it waits. Deep and Supreme use the same stronger model. Supreme adds 2,000 bootstrap re-analyses, three window sizes and paragraph influence analysis, which make the uncertainty estimate and the section map more reliable, then writes a PDF. On a laptop this takes seconds to about a minute; on a phone or with very long documents, longer.
Caveats. Human samples per group are small, so intervals are wide. Genres we have little human data for (personal essays, blogs, modern published fiction, student creative writing) use the conservative threshold, and their true false-positive rate is unmeasured. The Deep model was trained on research data whose upstream rights are still under review, so it must be retrained on commercially cleared data before a paid launch.
Data, licensed or not at all
38sources reviewed (incl. 16 from 2025–26)
6approved for commercial training
17quarantined pending source-rights review
Approved commercialWikipedia · Stack Exchange · arXiv & PMC (per-record CC licences) · AI Text Detection Benchmark 2026 · WasItAIGenerated 2026-09 (per row)
Source-rights review (quarantined)RAID · HC3 · HC3-Chinese · TXD-22 · OpenLLMText · APT-Eval · Ghostbuster · Academic Text 2026 · ITAIS v2 · IndicTextFake · GRADTEX · DeHAT · Śmigiel · QuillBot Academic · AH&AITD · Profiler · Human-AI Parallel
Non-commercialAuTexTification · PERSUADE 2.0 · OUTFOX · unified_tell · ALHD · AIGTxt
Licence unknownCOLING-2025 MGT · M4 · ArguGPT · Ateeqq · AITDNA · EditLens · Napier parliamentary · two HF mirrors
A dataset's own licence is not enough: we also check where its human text came from. Datasets built from scraped news, reviews, essays or forum posts are kept in a quarantine area until their source rights are cleared, even when the dataset itself is MIT or CC BY. Every import re-checks the licence, records checksums and versions, keeps per-record attribution, and never turns an unknown label into "human" or "AI". Human-only corpora are limited to text written before 30 November 2022.
The promotion gate
A model is promoted only if none of its training texts appear in the evaluation data, at most 1% of human texts are flagged overall and at most 2% in any writer group, and nothing regresses against production.
| Candidate | Decision | Why |
|---|
| v0.2.0 | Rejected | 864 evaluation texts overlapped its training data, so the evaluation would be inflated. |
| v0.3.0 | Rejected | Human FPR 1.36% (3 of 220; 95% CI 0.46–3.93%) exceeds the 1% limit. Production is unchanged. |
Roadmap
- NowImport the approved corpora at scale; generate in-house AI text from cleared prompts with current models; retrain on the commercial pool only.
- NowRequest licences for the most valuable quarantined sets (GRADTEX, ITAIS, DeHAT, Profiler) and for PERSUADE 2.0 (student essays).
- NextBenchmark current models (GPT-4o/5, Claude, Gemini, DeepSeek, Llama), AI-edited human text and humanizer output.
- NextServer-side transformer classifier (sentence windows, like Turnitin) plus a language-model layer; humanizer-robust training in which humanized human text is labelled human.
- NextHuman creative writing by students and modern authors under clear licences, to measure and protect the creative register.
- ThenAccounts, saved reports and paid scans (USD $2), payment provider in test mode first.
