# Are AI Detectors Accurate? What the Studies Show

> Not accurate enough to prove anything. Independent studies found that AI text detectors miss a large share of machine-written text, especially after light editing, and wrongly flag human writing, with false positive rates above 60 percent for some non-native English writers. A detector score can start a conversation. It should never decide a grade or a job.

- URL: https://fake-ai.com/ai-text/are-ai-detectors-accurate/
- Language: en
- Updated: 2026-10-19
- Publisher: Fake AI (OnlineBuilders LLC, https://onlinebuilders.biz)

OpenAI built its own AI text classifier and launched it in January 2023. On its test set the tool correctly flagged 26 percent of AI-written text and wrongly labelled 9 percent of human text as AI. Six months later, on 20 July 2023, OpenAI switched it off, citing its low rate of accuracy.

The company that made ChatGPT could not reliably detect ChatGPT. That should frame every claim of "99 percent accuracy" you see on a detector's homepage.

## What independent studies found

Vendor numbers are measured on vendor data. Independent researchers keep finding much weaker results.

### Bias against non-native writers

Researchers at Stanford, led by Weixin Liang, ran seven popular detectors over 91 essays written by non-native English speakers for the TOEFL test and 88 essays by US eighth-graders. The detectors were nearly perfect on the American essays. On the TOEFL essays the average false positive rate was 61.22 percent. All seven detectors agreed that 18 of the 91 human essays were AI-written, and 89 of 91 were flagged by at least one detector ([Liang et al.](https://arxiv.org/abs/2304.02819)).

### Fourteen tools, none reliable

Debora Weber-Wulff and colleagues tested 12 public tools plus Turnitin and PlagiarismCheck in 2023. Their conclusion: "the available detection tools are neither accurate nor reliable". The tools leaned towards calling text human-written, and manual editing or machine paraphrasing made detection much worse. Human texts written in another language and machine-translated into English were flagged as AI noticeably more often ([Weber-Wulff et al.](https://arxiv.org/abs/2306.15666)).

### Easy to defeat

Mike Perkins and colleagues tested six detectors on 805 text samples. Baseline accuracy was 39.5 percent. After simple adversarial edits it fell by a further 17.4 percentage points, to 22.1 percent. The authors concluded that the tools cannot currently be recommended for deciding academic integrity cases.

### Few work at low false positive rates

The RAID benchmark from the University of Pennsylvania, presented at ACL 2024, covers more than 10 million documents from 11 language models and 11 genres, with 12 kinds of adversarial attacks. When the authors fixed each detector at a 5 percent false positive rate, results varied widely, and few detectors could operate below 1 percent at all.

**Detector results from published studies**

| Item | Value |
|---|---|
| OpenAI classifier: AI text correctly flagged | 26 % |
| OpenAI classifier: human text wrongly flagged | 9 % |
| TOEFL essays wrongly flagged, average of 7 detectors | 61.22 % |
| Six detectors, accuracy on unedited AI text | 39.5 % |
| Six detectors, accuracy after simple edits | 22.1 % |

Source: OpenAI (Jan. 2023); Liang et al., Patterns 2023; Perkins et al., IJETHE 2024

## Why detectors make these mistakes

Most text detectors measure how predictable a text is to a language model. Machine-generated text tends to use the most likely next word, so it scores as very predictable. Human writing is usually less predictable, with odd word choices and uneven sentences.

That logic has two holes. First, some humans write predictably on purpose or by necessity: non-native speakers with a smaller vocabulary, technical writers following a style guide, students taught to use stock phrases, anyone who ran their draft through a grammar checker. When Liang's team had ChatGPT enrich the vocabulary of the same TOEFL essays, the false positive rate dropped sharply. Same authors, same ideas, fancier words. Second, AI text becomes less predictable as soon as someone edits it, translates it or asks the model to vary its style.

Length matters too. OpenAI warned that its own classifier was very unreliable on texts under 1,000 characters, roughly 150 to 200 words, which covers most discussion posts, emails and short answers.

Mixed texts are the hardest case. Turnitin reported that its sentence-level false positives cluster where human and AI writing meet: 54 percent of wrongly flagged sentences sat right next to actual AI text. A student who used a tool for one paragraph can therefore see neighbouring paragraphs of their own writing flagged as well.

Each new model generation shifts the patterns again. Detectors tuned on one model family can lose accuracy on the next.

## The base rate problem nobody puts on the homepage

Even a low false positive rate produces a lot of false accusations at scale. Turnitin stated in June 2023 that its document-level false positive rate was below 1 percent for documents with at least 20 percent AI writing, and that its sentence-level rate was about 4 percent. Vanderbilt University did the arithmetic when it turned the feature off in August 2023 ([Vanderbilt](https://www.vanderbilt.edu/brightspace/2023/08/16/guidance-on-ai-detection-and-why-were-disabling-turnitins-ai-detector/)).

**Vanderbilt's estimate: what a 1 percent false positive rate means**

- Papers submitted to Turnitin in 2022: 75,000
- Claimed false positive rate: 1 %
- **Papers potentially flagged wrongly: about 750**

Source: Vanderbilt University, August 2023

Seven hundred and fifty students accused of something they did not do, in one university, in one year. And the independent studies above suggest that 1 percent is optimistic for many kinds of writing.

## Questions to ask a detector vendor

If your school or company is buying a detector, the sales deck will show an accuracy figure. Ask what sits behind it.

**Before you trust a detector's accuracy claim**

- [ ] Which models generated the test texts, and are current ones included?
- [ ] What is the false positive rate on human text, measured separately?
- [ ] Was it tested on non-native writers, technical writing and short texts?
- [ ] How does accuracy change after editing, paraphrasing or translation?
- [ ] Has anyone outside the company reproduced the numbers?
- [ ] What does the vendor say a score should and should not be used for?

A vendor that answers only with a single accuracy percentage, or refuses to state a false positive rate, has told you what you need to know. The false positive rate is the number that decides how many innocent people get flagged.

## What detectors are actually good for

Detectors are not useless. They work as a rough triage tool when the stakes are low and a human makes the decision: an editor checking whether a batch of freelance articles needs a closer look, or a teacher deciding which essays to discuss in class.

**Where an AI detector score is a reasonable input**

| | Reasonable use? | Why |
|---|---|---|
| Deciding a grade or failing a student | no | False positives fall on real people |
| Firing or rejecting a writer | no | Same problem, higher stakes |
| Picking texts for a closer human read | partly | Only with other evidence |
| Spotting spam floods on a website | yes | Errors on single texts matter less |
| Checking your own text before submitting | partly | Can calm or create false worry |

Turnitin's own guidance points the same way. Its guide to the AI writing report says the score should not be the sole basis for adverse action against a student and calls for human judgement under the institution's own policies.

## Better ways to check authorship

If you need to know whether someone wrote a text, look at the process rather than the product. Draft history in Google Docs or Word, earlier versions, notes, sources the writer can explain, and a short conversation about the argument tell you far more than any score. Watermarks may help in future, but they only cover text from providers that embed them, as our guide to the [AI text watermark](https://fake-ai.com/ai-text/ai-text-watermark/) explains.

If you have been flagged yourself, our step-by-step guide on what to do when [falsely accused of using AI](https://fake-ai.com/ai-text/falsely-accused-of-using-ai/) covers the evidence that helps. Organisations writing an AI policy can find the legal labelling rules in our explainer on [EU AI Act Article 50](https://fake-ai.com/ai-text/eu-ai-act-article-50/). And if you are building a tool that needs to know where text came from, ask a developer such as [OnlineBuilders](https://onlinebuilders.biz/solutions/machine-learning) about provenance logging instead of after-the-fact detection.

## FAQ

### Which AI detector is the most accurate?

Rankings change with every new language model, and vendors test on their own data. Independent benchmarks such as RAID show that some detectors do well on familiar text at a 5 percent false positive rate, but few can operate below 1 percent, which is the range that matters when you accuse someone.

### Can AI detectors detect ChatGPT?

Sometimes. Unedited output from a model the detector was trained on is caught more often. Edited, translated or mixed text is caught much less reliably, and newer models are harder to detect than older ones.

### Why did my own writing get flagged as AI?

Detectors look for predictable word choices. Plain, formulaic or heavily edited writing, technical prose and text by non-native speakers can look predictable to them. A flag says your text resembles their idea of AI output, not that you used AI.

### Does Turnitin's AI score prove cheating?

No, and Turnitin says so itself. Its guide to the AI writing report warns that the model can misidentify text and should not be used as the sole basis for adverse action against a student.

Photo: lecroitg, CC0 (https://commons.wikimedia.org/wiki/File:Test-986769_640.jpg)