A detector can be statistically excellent and still be wrong about the page in front of it.
Forty-four percent of a book written in 1851
For a client project on AI detection, I decided to start with a writer who could not plausibly be accused of using ChatGPT. Herman Melville published Moby-Dick in 1851 and died in 1891. Whatever disagreements the novel has inspired over the past 175 years, whether a large language model wrote it has never been one of them. So we ran the entire book through Pangram, and because the detector would not accept the whole text in a single scan, we divided it into blocks, tested them separately, and accumulated the results. When we added everything up, 44% of the book had landed on the AI side.
That sentence needs an immediate qualification. Pangram did not ingest Moby-Dick as one document and announce that Melville was 44% machine. The 44% was our cumulative figure from a sequence of scans, which means our chunk sizes, our block boundaries, and our method of adding the parts together are all baked into the number. It is a result of our procedure, not a native whole-book score.
Even with that caveat, 44% is hard to shrug off.
The client's first explanation sounded reasonable: perhaps Pangram was confused because detectors are trained on books. I was skeptical, and the reason is simple — if human books sit in a detector's training data, they should teach the system what human writing looks like, not turn books into evidence of a machine. Training on Melville should make the detector better at recognizing Melville as human.
Then we tried something harder to explain.
I wrote a completely new passage about something that does not exist. The text had never been published, could not have been copied from anywhere online, and had no prior existence for any detector to recognize. I wrote it myself; the only automated tool that touched it was Grammarly, for typo and punctuation support.
Pangram returned 100% AI.
We then handed the passage to Claude, explained what had happened, and asked it to rewrite the text with the explicit goal of getting past Pangram. The result moved in the right direction, though not in the way you might expect: Pangram now reported 74% AI and 26% human.
A piece of human writing had drawn the strongest possible AI verdict — and then an AI rewrote it and made the detector less sure it was AI.
At that point the interesting question was no longer whether detectors occasionally err. Every classifier errs. The better question was what these systems are actually measuring when they hand you a number that looks so much like a statement of authorship.
A detector receives text, and nothing else
There is a tempting mental model of an AI detector that turns out to be almost entirely wrong. It imagines the tool looking your passage up in some vast archive of ChatGPT output, or finding a hidden certificate that reads Claude wrote paragraph four. That is not what happens. The ordinary paste-and-scan detector does not know that Moby-Dick appeared in 1851, that I wrote a passage last Tuesday, or that a student's Google Doc holds three hours of revision history. It receives text and infers its origin from the words in front of it.
Some products can gather process evidence — document history, keystroke activity, writing replays — but that is a different kind of evidence collected by a different mechanism. It's worth being precise here, because the commercial products do bolt extra layers onto the core: GPTZero lists a "novel text search" component, and Pangram has retrieval-flavored features too. The classifier at the center, though, is not doing lookup. It is doing inference.
And that inference is made by a model trained to separate examples labeled human from examples labeled AI, with increasingly elaborate handling of the mixed and edited writing in between.
Pangram describes a neural classifier that tokenizes text, converts the tokens into numerical representations, and learns from large sets of human and AI examples. Its training deliberately hunts for human writing the model gets wrong, adds those hard cases back into the training data, and retrains. It also builds what it calls synthetic mirrors — AI documents generated to match human examples in topic and structure — so the classifier can't cheat by learning that certain subjects are simply more "AI" than others.
Originality describes a transformer-based discriminative model derived from the BERT family, trained on millions of examples. Same objective: learn a boundary between classes, not retrieve a stored copy.
GPTZero needs a particularly important correction, because its reputation is out of date. Its original 2023 detector became famous for perplexity and burstiness — perplexity being, loosely, how surprising a sequence of words is to a language model, and burstiness being how much that surprise varies across a document. Those ideas are still useful for understanding the history of this field. But the company states plainly that it stopped using perplexity and burstiness as its detection method in autumn 2023 and moved to a deep-learning architecture. Its 2026 technical report describes a hierarchical system that sorts text into Human, AI, and Mixed, then subdivides the AI category further.
That retirement dismantled one of my own initial framings. I had been carrying a tidy contrast in my head — GPTZero as the old perplexity generation, Pangram as the modern neural generation — and it is no longer accurate. All three leading tools have converged on learned classification.
Where perplexity still bites is in the cheaper knockoff detectors that never moved on, and those are not the tools serious buyers are choosing between.
The perplexity story nonetheless survives everywhere, and I understand why: it is wonderfully intuitive. Predictable prose looks machine-made; surprising prose looks human. It hands people a feature they can picture.
Modern classifiers are far less obliging. They learn combinations of signals that don't collapse into a neat list of stylistic rules — lexical choices, local regularities, syntactic patterns, relationships between clauses, and internal representations that have no tidy one-word name. Even the people who build these systems can describe the training procedure more easily than they can explain why one particular Melville passage crossed one particular boundary. That is a real limitation, and it bites hardest at the exact moment someone asks the question that matters most: why did it call this AI? We can explain how such an error becomes possible. Without the internal decision path for that specific scan, we cannot responsibly name the feature that caused it.
A percentage is not a unit of authorship
The interface makes this epistemic gap worse because a percentage looks far more precise than the claim it represents.
There is no shared, standardized "AI percentage" across detector companies. Originality states outright that its classic percentage is a confidence measure — a result of "40% Likely AI" does not mean 40% of the words came from a machine. Pangram 4 does something different, assigning labels at the token level and deriving an AI fraction from them, with its own definitions separating negligible contribution from genuine co-authorship. GPTZero has built yet another taxonomy around human, AI, and mixed. All of these can surface on screen wearing a percent sign while pointing at genuinely different quantities.
Our Moby-Dick number is a fourth quantity again, because we manufactured it from many scans. It should never be presented as though Pangram had measured some physical substance called "44% AI" sitting inside the novel. The honest phrasing is duller and much more useful: under our scanning and aggregation procedure, Pangram assigned enough of the tested blocks to the AI side to produce a cumulative 44%. That is a fact about our method as much as about the text.
The client was right for the wrong reason
The idea that Pangram might stumble because it was "trained on books" quietly folds two different training processes into one.
Language models like Claude and GPT are trained to generate language. Detectors are trained to classify it. A book can matter to both, but for different reasons. With an old-style perplexity detector, familiarity in the underlying language model could plausibly make a canonical passage look less surprising, which is exactly why classic literature seems like such an obvious false-positive candidate under that theory. But that theory describes a mechanism current Pangram largely does not use.
Pangram says its human training material spans many forms of human writing, while its AI side is built partly from synthetic counterparts. Human books therefore belong, conceptually, on the side that teaches the system what human writing is.
A classifier having encountered literary prose is nothing like a plagiarism checker recognizing Moby-Dick and mistaking recognition for authorship.
None of which means training is irrelevant — it matters enormously. Training data decide which variations of human and machine writing a classifier can tell apart. If some form of human prose is thinly represented, or represented oddly, or tangled up with another feature, the model can learn a crooked boundary, and machine-learning systems are notorious for exploiting correlations their designers never intended. So the client's instinct was sound: training can absolutely produce systematic errors. It was the specific claim — that a book becomes suspicious merely because books were used in training — that doesn't hold.
And there is a more inconvenient piece of evidence.
Brian Jabarian and Alex Imas of the University of Chicago Booth School of Business evaluated Pangram, Originality, and GPTZero in a 2025 working paper distributed by the National Bureau of Economic Research. Their human corpus held 1,992 pre-2020 passages across six genres — and 1,000 of those were excerpts from pre-2000 English novels on Project Gutenberg. Under the study's detector-optimized thresholds, Pangram's false-positive rate on the novel category was zero. Originality's was 0.25%. GPTZero's was 0.45%. The paper even notes that some of those old novels lack full stops — genuinely archaic, oddly punctuated prose, the exact texture of Melville — and Pangram still produced no false positives on the genre.
So the strongest independent evidence available points the opposite way from the easy story. Novels do not, as a class, fool Pangram. Which makes our result more interesting, not less.
Moby-Dick is an anomaly, not yet a theory
It would be easy to inflate "44% Moby-Dick" into proof that Pangram fails on old literature. The evidence forbids it.
Our experiment and the Booth benchmark differ in ways that have to be pulled apart before anyone reconciles the numbers. The benchmark sampled passages from a thousand different novels; we hammered on a single book with an unusually recognizable style. The researchers controlled passage construction and calibrated thresholds; we split a book by hand. And the target keeps moving — Pangram 4 shipped in July 2026, while the Booth work captured an earlier generation of these tools.
Segmentation is not an incidental detail here, either. Pangram's own technical report describes handling text in segments during training, which means context length and boundaries are part of how the system behaves.
Our manual blocks change what surrounding text the classifier sees: a sentence near a boundary is read in a different context than it would be mid-chapter, a short selection can score differently than the longer passage it came from, and the way we stitched separate scores together is one more decision the detector never made.
That does not explain the result away. It names the variables we failed to hold still — and there is a real difference between those two things. The next version of this test needs fixed passage lengths, documented split points, the exact detector version, the raw output from every scan, and more than one aggregation rule. It also needs comparison passages from other pre-LLM novels. If the pattern survives all that, we have something far sturdier than a striking screenshot. For now, I know Pangram produced an unexpected result. I do not yet know why.
Then I removed memorization from the story
The invented-text experiment earns its place for a different reason: it kills a specific explanation. The most common folk theory of AI detection is some version of it has probably seen that text before. My passage removed that at the document level. The exact text was new; there was no prior original for Pangram to retrieve. It still came back 100% AI.
That does not mean every phrase, construction, or statistical pattern in it was unprecedented — no meaningful English writing works that way. Novel authorship and novel language are different claims. But the result shows something real about how the detector operates: a passage does not need to exist in the training data to be classified as AI.
The model generalizes from characteristics it learned elsewhere and drops unfamiliar writing onto one side of a boundary.
That is exactly what a good classifier is supposed to do. A detector that could only recognize AI text it had already seen would be nearly useless. But the same generalization that lets it catch genuinely new AI output is what makes a false positive possible, because generalizing means ruling on unfamiliar examples, and sometimes the ruling is wrong.
Our experiment gives that abstraction an uncomfortable shape. I knew where the text came from. The detector knew only its statistical representation. The detector won the argument on the screen. I had the better evidence.
Grammarly is a suspect, not a verdict
My first instinct was to blame Grammarly. Only one AI-adjacent tool had touched that human passage before Pangram saw it, and it was easy to join those dots into a conclusion: light polishing nudged human prose across the line. There is real support for the general worry. The Booth paper itself flags grammar correction as one of the genuinely hard boundaries institutions face when deciding what a detector should punish. Originality has tested this directly, taking ten known-human IELTS essays and running two Grammarly conditions — a "lightly edited" pass that accepted grammar fixes but refused rephrasing and rewriting, and a "heavily edited" pass that accepted everything, rewrites included. Its finding: light, grammar-only editing left almost all the essays still classified as human, while heavier rephrasing and rewriting had a substantial effect on AI detection.
Pangram goes further and aims a number squarely at my hypothesis. Pangram 4 reports flagging AI-polished human writing as fully AI-generated only 0.009% of the time — a figure it frames as a 20-fold improvement over its previous version. In other words, the current model claims to be specifically engineered so that light polish almost never triggers a full-AI verdict.
Which cuts against the tidy story rather than for it. If our editing really was confined to that level, and if the scan ran on a model with that hardening, then a 100% AI result collides head-on with the detector's own stated behavior.
That doesn't resolve the puzzle — it sharpens it.
And there is a plain methodological problem with pinning it on Grammarly anyway: we never preserved and scanned an untouched control version of that passage before Grammarly touched it. So our experiment established that a human-origin passage with limited Grammarly assistance can draw a 100% AI result. It did not establish that Grammarly caused it. I don't want to smuggle causation into the story just because the causal version is the satisfying one.
A proper test needs two copies from one identical human source: one left untouched, one receiving only predefined spelling-and-punctuation corrections, with every accepted suggestion logged. Rephrasing, tone changes, and clarity rewrites have to be excluded from the light-edit condition — because "used Grammarly" now covers everything from a comma fix to full generative rewriting. Only then can any score difference be laid at the intervention's feet. And that distinction stops being academic the moment these tools are pointed at people instead of books.
Who decides what counts as human?
The Grammarly problem exposes something the percentages tend to hide: there is no longer an agreed line between human writing and AI writing.
Suppose I write a paragraph and software fixes two commas. Almost everyone still calls that human. Suppose it rewrites six sentences — now perhaps we say AI-assisted. Suppose I supply the argument and Claude writes every sentence; the intellectual origin is human while the visible language is machine-made.
The detectors have started building these gradations into their products precisely because binary authorship stopped describing how people actually write.
Pangram 4 separates human, AI-assisted, and AI-generated, and its report acknowledges the gap between the provenance of an idea and the provenance of the final words. GPTZero's current model contains an "AI Polished" category for text originally written by a human and later refined with AI — including uses such as improving grammar, rephrasing sentences, or enhancing clarity. Originality offers models and policy settings tuned to tolerate different amounts of AI involvement, and its own materials separate light editing from material rewriting.
The consequence is that a false-positive rate is never fully independent of how you define the positive class. Before you can ask how often a detector wrongly accuses a human, you have to decide what you're counting as human writing in the first place. That is no longer a seminar-room question. A company that allows Grammarly spelling fixes but forbids generative rewriting needs a detector whose categories and thresholds match that policy. A university may draw the line somewhere else. A publisher may care only about undisclosed generation of finished prose. Without that definition fixed in advance, a highly accurate detector will faithfully, precisely enforce the wrong rule.
One screenshot is not a study
None of my experiments licenses calling AI detectors random. The Booth paper is strong evidence for the opposite — provided it's described accurately.
It is a working paper, not a peer-reviewed article. The authors used 1,992 verified pre-2020 human passages and generated a matching 1,992 AI passages for each of four frontier models, then tested Pangram, Originality, and GPTZero across genre, length, threshold choices, very short passages, and humanized AI text. Pangram performed exceptionally. Most relevant to false accusations, it was the only detector in the study that could hold a stringent false-positive policy cap of 0.5% without losing its ability to catch AI text. That is the defensible Booth number, and it is the one to quote.
It is not the same as a universal "one in 10,000" rate; that figure isn't in the study. There is a separate and much smaller number in Pangram's own July 2026 technical report: Pangram 4 reports a 0.0041% false-positive rate — roughly one in 24,000 — on its evaluation set. That is a vendor benchmark for a newer model, and it should be labeled as one rather than blended with the independent work.
Here is the part that matters most for how anyone reads that number, and it comes from outside the company. No independent dataset yet published is large enough to test a rate as small as 0.0041%. Zero false positives across 1,992 passages only bounds the true rate at roughly 0.15% — one to two orders of magnitude short of the headline figure — and no third party has evaluated Pangram 4 at all. The published rate is also, by construction, an English, long-form, complete-sentence number; Pangram itself puts short passages, code, tables of contents, and templated writing out of scope. A chopped-up 1851 novel is not obviously a member of the population that rate was measured on.
None of these numbers actually contradict our experiment, because a false-positive rate is not a physical constant welded to a brand.
It is a property of a model version, a threshold, a test corpus, and a definition of ground truth. Tell me a detector errs once in 24,000 documents resembling its benchmark, and you've told me a great deal about expected performance on that population and almost nothing about one unusual document standing outside it. A single spectacular error, equally, says nothing about the population-wide rate. Both can be true at once — which is precisely why a false positive is so consequential. A classifier can be genuinely excellent and still be completely wrong about you.
The evidence hierarchy is upside down
The longer I worked through this, the more the whole structure of evidence looked inverted.
A text detector has less information about authorship than almost anyone else in the room. It sees the finished artifact and reasons backward to a process it never observed. Meanwhile, the forms of evidence we treat as secondary are the ones actually tied to the event we care about — how the text came to exist. A dated 1851 publication carries provenance. A version-controlled document carries provenance. Drafts show development; edit histories can distinguish a sentence typed character by character from one pasted in whole; in a controlled setting, authorship can be watched as it happens.
The detector answers a different question entirely: how closely does this text, according to this model, resemble the classes it learned to associate with AI?
That question is enormously useful. It can surface material worth reviewing at a scale no human team could reach, and serve as one signal among several in fraud detection, academic integrity, publishing, or moderation. The trouble starts only when that statistical resemblance gets promoted into provenance — when a number about how the text reads is treated as a finding about who wrote it.
Even the vendors resist that promotion. Originality, while selling a detector, tells users the score is not a final determination. The Booth researchers frame deployment as a policy problem precisely because false accusations and missed AI carry different costs. Pangram and GPTZero have both moved toward mixed-authorship categories because the old binary stopped describing reality.
So for any client weighing these tools, I now ask one question before we discuss accuracy at all: what happens when the result comes back positive?
If the answer is "we take a closer look," a detector can carry plenty of uncertainty and still earn its keep. If the answer is that someone loses a grade, a contract, a byline, a job, or their credibility, the standard of evidence has to climb to meet the stakes — and a percentage generated from linguistic resemblance cannot carry that weight alone.
Which is why our next test starts before Grammarly ever touches the page. The coming passages will be preserved in their untouched state before any tool sees them; a second copy will receive only specified spelling-and-punctuation corrections, every accepted change logged, so we can measure the Grammarly effect instead of guessing at it. Moby-Dick will be rerun with predetermined block sizes and recorded boundaries rather than collapsed into one headline figure, with other pre-LLM texts alongside it for comparison. All three detectors will see the same passages, and every version and scan date will be recorded, because these systems change too fast for an undated score to mean much.
The goal was never to make one detector look bad; the published evidence would make that a silly errand. What I want to know is where the classification starts to peel away from independently established authorship, how reproducibly it does so, and what happens when only one part of the writing process changes.
Moby-Dick gave us the first reason to look. The invented passage gave us a better one. And if a client decision ever turns on whether a particular person wrote a particular document, I would rather walk in with the history of how that document was made than a screenshot reading 74%.