Can AI-Written Fiction Be Caught by How It's Built, Not How It Sounds?

The problem: AI style is getting harder to catch

In early 2026, a major publisher pulled a horror novel from shelves after detection tools flagged it as mostly AI-generated the first commercial book pulled over AI authorship concerns. Around the same time, detection services found that roughly a fifth of self-published novels on Amazon showed heavy signs of AI generation, a figure that had jumped sharply year over year.

The usual way researchers catch AI writing is by looking at style: word choice, sentence rhythm, overused words like "delve" or "tapestry," excessive em-dashes. The trouble is that style is a moving target. Newer model versions quietly drop these tics, and a bit of fine-tuning or light editing can crash detection accuracy from near-perfect to almost nothing.

A new paper, StoryScope: Investigating Idiosyncrasies in AI Fiction, accepted at COLM 2026, asks a different question: instead of how a story sounds, can you tell AI writing apart from human writing by how it's constructed its plot shape, its handling of time, how it reveals information, how its characters are built? These are the underlying narrative choices, not the surface prose, and the researchers argue they're much harder to fake without a full structural rewrite.

The basic setup

The team built a large parallel dataset: 10,272 writing prompts, each answered by both a human author and five current LLMs Claude, GPT, Gemini, DeepSeek, and Kimi. That's six versions of essentially the same story premise. In total, that's 61,608 stories, averaging around 5,000 words each long enough to actually contain flashbacks, subplots, and slow-burn revelations, which is exactly the kind of structure this study cares about.

From that corpus, they extracted 304 distinct narrative features spanning ten dimensions of storytelling: character agency, social relationships, plot structure, setting, time, and how information is revealed to the reader.

How they pulled it off

They call their pipeline STORYSCOPE, and it runs in three stages:

1. Structured representation every story is converted into a JSON-style template capturing its narrative content (who the characters are, what happens, how time moves) while stripping away the actual prose. This forces later stages to compare stories on substance, not wording.
2. Cross-source comparison for a sample of prompts, all six versions of a story (human + 5 AI) are compared side-by-side by an LLM judge, which writes up observations about where the sources diverge who uses gossip as a plot device, who explains the moral outright, and so on.
3. Feature discovery and scoring those observations get turned into concrete, closed-form questions (e.g., "Is the resolution driven by the protagonist's choices or by outside events?") that can be answered consistently for every story. Each of the 61,608 stories is then scored on all 304 features by an LLM annotator, producing a feature vector for classification.

The team trained XGBoost classifiers on these vectors for two tasks: telling human from AI writing, and identifying which of the five AI models (or a human) wrote a given piece.

What actually separates human and AI writing

Using narrative structure alone with stylistic cues deliberately excluded the classifier hit 93.2% accuracy (macro-F1) at distinguishing human from AI stories, capturing 97% of the performance of a model that also had access to style. For the harder task of guessing exactly which of six sources wrote a story, narrative features alone reached 68.4% accuracy.

A few consistent patterns emerged:

• AI over-explains. AI narrators spell out the story's theme or moral explicitly far more often than human narrators do (77% vs. 52% of the time), and AI dialogue leans into open philosophical debate more than human dialogue does.
• AI likes tidy plots. AI stories favor a single, protagonist-driven throughline with few subplots and clean resolutions. Human stories are messier more subplots, more ambiguous endings, more willingness to leave threads loose.
• AI plays it chronologically straight. Humans use flashbacks, time jumps, and nonlinear structure far more often to delay a reveal; AI tends to tell events in the order they happened.
• AI shows feelings through the body. Where a human writer might just say a character "felt afraid," AI reaches for physical description a tight chest, cold sweat at a much higher rate (81% vs. 38%).
• Humans talk to the reader and the world. Human stories break the fourth wall and address the reader directly far more often, and reference real, specific outside works and authors almost twice as often as AI, which tends to stick to vague allusions.
• AI stories cluster together. Plotted in narrative-feature space, the five AI models sit in one tight, overlapping region, while human stories spread out much more widely and occupy a distinct region and are measurably "rarer" in their combinations of narrative choices.

Notably, the researchers also tested what happens after an AI story is stylistically rewritten to remove typical AI prose artifacts. The narrative-based classifier barely lost accuracy dropping less than 2 points because editing sentence-level style doesn't touch the underlying plot architecture.

Every model has its own narrative "handwriting"

Beyond the human/AI split, each AI model showed distinguishable habits:

• Claude is the most restrained and distinct of the five. It tends to escalate events gently rather than building to a dramatic peak, favors quiet epilogue-style endings, avoids dream sequences, and leans into honoring familiar storytelling conventions rather than subverting them.
• GPT gravitates toward socially-driven plots, using gossip and rumor as a mechanism more than any other source, and is more willing than other AI models to subvert reader expectations or leave a resolution ambiguous.
• Gemini, DeepSeek, and Kimi form a more confused, harder-to-tell-apart cluster, but each still has quirks: DeepSeek tends to front-load context that other sources reveal gradually; Gemini favors especially tidy endings and notably bleak settings; Kimi comes across as the most generic of the five, with the fewest distinguishing traits.

Conclusion

The headline takeaway is that AI-generated fiction has a detectable shape, not just a detectable voice. As surface style keeps getting easier for newer models (or a quick editing pass) to disguise, the way a story is put together its causal chains, its handling of time, how explicit it is about meaning looks like a sturdier signal, because faking it convincingly would require rewriting the story's architecture, not just swapping words.

The paper frames this against an emerging legal and creative-industry question: as courts and copyright offices increasingly focus on how much genuine human creative control went into a work, "how was this conceived" may end up mattering more than "how was this written." The researchers have released their prompts, a large set of AI-generated stories, and the narrative-feature data to support further work on AI authorship and narrative analysis.