How AI Detectors Actually Work

Most articles about AI detectors stop at "they check for patterns." That is like saying a car works by "moving forward." Let us actually open the hood.

The two classic metrics

Perplexity

Imagine feeding a sentence one word at a time into a language model and asking it to guess the next word. If the model assigns a high probability to the correct next word at every step, the overall perplexity of the sentence is low. Low perplexity means the text is predictable. Language models produce predictable text by design, so low perplexity is a signal that the text may be machine-generated.

The formula is straightforward:

1. Take the average log probability of each token 2. Negate it 3. Exponentiate the result

A perplexity of 5 means the model was on average choosing among five equally likely candidates for each word. A perplexity of 50 means the model was genuinely uncertain. Human writing typically scores much higher than AI writing on this metric.

Burstiness

This metric looks at the distribution of sentence lengths and the variation in syntactic structures. Humans produce bursts:

• A short punchy sentence
• Then a subordinate clause that stretches across three lines
• Then a fragment

The standard deviation of sentence length is one common proxy for burstiness. Language models, especially earlier ones, tended toward uniform sentence lengths because they optimize for coherence at the token level. A high burstiness score indicates the kind of variation that human writers naturally produce.

Think of it this way: If every sentence in a paragraph is roughly the same length, something is off. Humans don't write that way.

The newer signal: token probability

Recent research has found that the raw probabilities the model assigns to its own output carry more information than perplexity alone.

In 2024, a team at the University of Rochester published a study showing that log probability distributions at the token level achieve classification accuracy above 95 percent on several benchmarks, compared with roughly 88 percent for perplexity-based methods.

The intuition is simple: a model producing its own text is highly confident about each token. A human writer, even a good one, produces text that looks less like what the model would have chosen.

How commercial tools use this

• GPTZero now reports an "entropy" score alongside its probability labels
• Originality.ai uses token-level features internally
• OpenAI shelved its own classifier in 2023 partly because it struggled with short texts, but the underlying approach of using token probabilities was sound—the problem was calibration, not the signal itself

The classifier

The raw features—perplexity, burstiness, token probability distributions—feed into a supervised classifier. Historically this was logistic regression. More recently, tools have moved to gradient-boosted trees or small neural networks.

The classifier was trained on labeled datasets: thousands or tens of thousands of paired examples of human-written and AI-generated text.

The choice of training data matters enormously. A detector trained primarily on academic essays will struggle with marketing copy, and vice versa.

Why calibration is the hard part

Building the classifier is the easy part. Calibrating it so that it produces reliable scores across every domain is the hard part.

The false-positive problem

The problem shows up as false positives:

• Legal briefs flagged because their formulaic structure resembles AI output
• ESL essays flagged because the writer's grammar is unusually clean
• Press releases flagged because they follow a rigid inverted-pyramid structure

These are not trivial numbers when the stakes include academic expulsion.

The model-generalization problem

Calibration also breaks down across models. A detector trained on GPT-3.5 output may not generalize to Claude 3 or Gemini. Each language model has its own statistical fingerprint. Detectors must continuously retrain or fine-tune on new model outputs, creating a moving-target problem.

The ensemble approach

Some tools now use ensemble methods: multiple classifiers trained on different model outputs and different domains, with a meta-learner combining their predictions.

• This improves robustness
• But adds complexity and latency
• Makes the system harder to audit because no single model is responsible for the final score

What this means in practice

Understanding the mechanics tells you where to trust the tool and where to be skeptical.

The detector is not reading for meaning. It is reading for statistical regularities. Those regularities are useful, but they are not the same as evidence of misconduct.

Ensemble methods in detail

Ensemble methods represent the most significant advancement in detection accuracy since the original perplexity-based approaches. Rather than relying on a single classifier, ensemble systems combine multiple models to produce a more robust prediction.

How ensembles work

An ensemble typically consists of three to five specialized classifiers, each trained on different data or using different algorithms:

• Classifier A trained on perplexity and burstiness features
• Classifier B trained on token probability distributions
• Classifier C trained on syntactic patterns and n-gram frequencies
• Classifier D trained on model-specific outputs (GPT-4o, Claude, Gemini)

A meta-learner then combines these predictions, weighting each classifier based on its historical performance on similar texts. The meta-learner might use a simple voting mechanism or a more sophisticated stacking approach where it learns which classifiers are most reliable for different text types.

Why ensembles outperform single classifiers

Single classifiers are vulnerable to distribution shift—when the text they encounter differs from their training data. A classifier trained on academic essays may struggle with marketing copy. An ensemble mitigates this by spreading the risk across multiple specialized models.

Independent detector evaluations generally find that ensembles can be more robust than a single classifier, especially when the writing domain or source model changes. The important point is not a universal percentage gain; it is that combining signals can reduce dependence on one brittle pattern.

The tradeoff

Ensembles are usually slower. Each additional classifier adds processing time and infrastructure cost. For most review workflows, that latency is acceptable. For real-time applications like streaming text analysis, it can be a constraint.

How detectors handle different AI models

Each language model produces text with a distinct statistical fingerprint. Detectors must account for these differences, and their ability to do so varies significantly.

GPT-4o

GPT-4o produces text with high predictability at the token level. Its training on massive datasets means it defaults to the most statistically probable word choices. This makes GPT-4o text relatively easy to detect—it has low perplexity and uniform burstiness. Turnitin detects GPT-4o output at 99.1% accuracy.

Claude 3.5

Claude 3.5 produces more varied phrasing than GPT models. Anthropic's training approach encourages Claude to avoid repetitive patterns, which increases the text's statistical distance from typical AI output. GPTZero's accuracy on Claude 3.5 drops to 86.9%, compared to 97.8% on GPT-3.5.

Gemini 1.5

Google's Gemini 1.5 sits between GPT-4o and Claude 3.5 in terms of detectability. Its output is less predictable than GPT-4o but more structured than Claude 3.5. Detection accuracy on Gemini 1.5 ranges from 89% to 94% depending on the tool.

Open-source models (Llama 3, Mistral)

Open-source models present the greatest challenge for detectors. Their outputs vary more widely because they are fine-tuned by different communities for different purposes. A Llama 3 model fine-tuned on academic papers produces very different text than one fine-tuned on Reddit comments. Detection accuracy on open-source models ranges from 74% to 93%, depending on the specific model and fine-tuning.

The retraining cycle

Detectors must continuously retrain on new model outputs. This creates a moving-target problem: by the time a detector has been retrained on GPT-4o output, a new model like GPT-5 or Claude 4 may be released with different statistical properties. The retraining cycle for major detectors is typically three to six months, which means there is always a window where new models are harder to detect.

The arms race between detectors and humanizers

As detectors improve, humanizers improve in response. This creates a continuous cycle of adaptation.

The humanizer challenge

Humanizers work by increasing the statistical distance between AI-generated text and the detector's model of AI output. Early humanizers performed simple word substitution, which was easy for detectors to identify. Modern humanizers use semantic reconstruction—encoding text into a meaning space and generating new text from that representation.

The detector response

Detectors respond by incorporating adversarial training. They add humanized text to their training data and learn to recognize the patterns that humanizers introduce. This is similar to how spam filters adapt to new spam techniques.

The current state

In mid-2026, humanizers continue to improve, but detectors adapt just as quickly. The arms race means that no humanizer or detector provides a permanent advantage.

Implications for users

The arms race means that no humanizer or detector provides a permanent advantage. What works today may not work in six months. Users should:

• Avoid relying on any single tool as a permanent solution
• Test their approach against multiple detectors
• Stay informed about new developments in both detection and humanization

Future of detection technology

Several emerging approaches could reshape how AI detection works in the coming years.

Watermarking

AI developers can embed subtle statistical patterns in their model output that are invisible to humans but detectable by algorithms. Google and OpenAI have both published research on watermarking techniques. If widely adopted, watermarking could provide a more reliable detection method than current statistical approaches.

The challenge is adoption. Watermarking requires AI developers to implement it voluntarily, and there is no guarantee that all developers will cooperate. Open-source models, in particular, can be modified to remove watermarks.

Behavioral analysis

Rather than analyzing the text itself, behavioral analysis examines the process that produced it. Did the writer produce a complete draft unusually quickly? Did the text appear in its final form without revisions? Did the writing session show no signs of the typical human process—deletions, backtracking, restructuring?

Canvas and Blackboard are beginning to incorporate keystroke logging and revision tracking, which could provide behavioral signals that complement text-based detection.

Stylometric profiling

Stylometric analysis builds a statistical profile of an individual writer's style. If a student's baseline writing shows a particular distribution of sentence lengths, vocabulary choices, and syntactic patterns, a dramatic deviation could indicate AI assistance.

This approach is more reliable than generic AI detection because it is calibrated to the individual. However, it requires access to the writer's previous work and raises privacy concerns.

Multi-modal detection

Future detectors may analyze not just the text but also the context in which it was produced. Did the writer reference sources that exist? Did they include personal anecdotes? Did they respond to feedback in drafts? Multi-modal approaches combine text analysis with metadata to produce a more complete picture.

Key takeaways

• Ensemble methods combine multiple classifiers to improve accuracy and robustness across different text types
• Different AI models produce text with distinct statistical fingerprints—GPT-4o is easiest to detect, open-source models are hardest
• The arms race between detectors and humanizers means no tool provides a permanent advantage
• Watermarking could provide more reliable detection but requires AI developer cooperation
• Behavioral analysis and stylometric profiling offer complementary approaches to text-based detection
• The best strategy is to use multiple signals rather than relying on any single detection method

Frequently asked questions

Why do detectors perform differently on different AI models?

Each language model has a distinct statistical fingerprint based on its training data and architecture. GPT-4o produces highly predictable text because it was trained on massive datasets and defaults to statistically probable word choices. Claude 3.5 produces more varied phrasing because Anthropic's training encourages it to avoid repetitive patterns. Detectors trained primarily on GPT output may struggle with text from other models.

What is adversarial training in AI detection?

Adversarial training is a technique where detectors are trained on humanized text—text that has been modified to evade detection. By including humanized examples in their training data, detectors learn to recognize the patterns that humanizers introduce. This helps detectors maintain accuracy even as humanization tools improve.

Will watermarking make current detection methods obsolete?

Watermarking could provide more reliable detection than statistical methods, but it is not a complete solution. Watermarks can be removed or degraded by humanization tools. Open-source models can be modified to remove watermarks entirely. Watermarking is most effective as one component of a multi-signal detection approach.

How often do detectors need to retrain?

Major detectors retrain every three to six months to keep up with new AI models and humanization techniques. Some detectors use continuous learning, updating their models incrementally as new data arrives. The retraining cycle creates a window where new AI models may be harder to detect.

Can behavioral analysis replace text-based detection?

Behavioral analysis provides complementary signals but cannot fully replace text-based detection. It requires access to writing process data (keystroke logs, revision history) which is not always available. It also raises privacy concerns. The most effective approach combines behavioral analysis with text-based detection for a more complete picture.

Try it yourself

Want to see how detection works in practice? Test your text with Vortixy's free AI detector and get an instant analysis of whether content reads as human or AI-generated. If you need to adjust flagged text, the AI humanizer can help you rewrite it naturally.