The Problem With AI Detection Tools

Substack's integration of Pangram introduces suspicion into the reading experience.

I don’t want to read 100% AI-generated articles. If I wanted to read those, I’d have AI generate them for me and read them there. However, 100% AI-generated articles are already easily detected with my own personal LLM that I’ve spent the last ~40 years training (i.e., my brain)… and I don’t mean “I can reliably identify all AI text by reading it” by saying that; context matters as well. One of my connections has what some might describe as an AI-sounding ‘voice’, but I’ve read her book that came out before AI was a thing. It’s just how she writes. Notably, her writing is also good. An article that has an AI cadence to it isn’t necessarily bad just because it has an AI cadence. AI was trained on human writing; thus, it has also adopted the best of human phrasing in its processes (including that “not this, but that” construct I just used in those two sentences).

Which brings me to the point of this article…

An AI Detector Has Been Introduced On Substack

Substack recently integrated the AI detection tool Pangram into its app, whereby anyone can screen any article, note, or comment over 100 words for AI-generated and AI-assisted text. Pangram is supposedly more accurate than other AI-detection tools, indicating a 1-in-10,000 false-positive rate for most types of writing. It also claims that it doesn’t store screened text or use it to train its models, so your writing shouldn’t be going into Pangram’s next algorithm. I’m skeptical of this, I must admit. We should be aware that scanning our own drafts (which is currently the only way to turn Pangram’s scanning off for Substack readers—we have to scan our text first to get that option) means that our text is being processed by a third-party company, regardless of how that company states it’s being used.

Anyway, I can get behind trying to preserve trust and authenticity in writing and understand the reasoning behind introducing Pangram on Substack. However, the decision also introduces substantial reputational risks for writers, creating new categories of collateral damage and changing how we approach reading in ways that may actually harm those whom Substack is trying to protect.

I don’t use AI to write, but I know people who do; they are using writing for a different purpose from me or need AI as an assist for certain parts of the process that I, for various reasons that don’t include being better than anyone else, don’t need help with (for example, I saw a post on here yesterday from a writer who thinks and writes in German and uses AI to translate his articles into English for his English-speaking audience). Those individuals often have defensible reasons for bringing AI tools into their process. Would I want a German writer to only write in German instead of using AI to make their thoughts accessible to those of us who only speak English? No.

I am also extremely concerned that the blanket encouragement to check everything we read for AI using detectors that are known to produce false positives (even if the rate claimed by the company is lower than that for other detectors) is damaging the reading experience and unfairly jeopardising the reputations of creators, including those who have never touched a generative AI tool.

The Reputational Risks Of AI Detection

Pangram, which doesn’t assess text based on how predictable the next word in a sentence is, incorporates a broader range of linguistic signals, and it does perform well compared with competitors in controlled laboratory settings (Turnitin’s false positive rate is ~1 in 200 for academic writing, for example). But laboratory performance is not the real-world. Journalist Tim Requarth investigated Pangram’s role in several high-profile publishing controversies and commented that “Pangram’s CEO is wielding it as a weapon against individuals”. In his article, Requarth discusses computer scientist Arvind Narayanan’s statistical calculation that even at a false positive rate of 1 in 10,000, if every instructor used an AI detector on all student submissions, 5–10% of students would be falsely accused at some point during their undergraduate studies.

So, the most immediate danger of integrating Pangram into Substack is the reputational risk it poses to writers, particularly when the tool produces a false positive. That false positive is is public accusation of inauthenticity, and having an AI detector embedded and ready to scan our writing for anyone who chooses to do so makes writing and publishing itself a reputational risk that we don’t have full control over (“Just turn it off”… no, because turning it off gives the impression we have something to hide and may also result in our audience assuming our writing is AI). That accusation can be devastating for creators, and it can happen and follow them regardless of whether they actually used AI. There’s a creator on TikTok who is currently being accused of supporting generative AI (I have yet to find evidence of this), and under every post they make and every post other creators make about them, there are comments advising against supporting the creator because they support AI. It just takes one false positive.

Emma Alpern documented the effects of false AI accusations in the New York Times: a day-care worker berated in front of children for supposedly using AI to write an incident report, a Kenyan writer spent decades learning formal English to be told he sounded like a machine. AI detectors look for statistical patterns that distinguish AI from human writing, like predictable word choices and uniform sentence length. But these patterns also characterise legitimate human writing.

Non-native English speakers are the most systematically disadvantaged. In a 2023 study, AI detectors tested on 91 TOEFL essays and 88 written by US eighth-graders accurately classified nearly all of the US student essays, but misclassified over half of the TOEFL essays as AI-generated, with a 61.3% false-positive rate. Non-native speakers often use a more limited vocabulary and more predictable sentence structures because they’re taught using the formal, edited English text that AI is trained on. Pangram has made efforts to address this bias, and its own benchmarks on ESL datasets show significantly lower false-positive rates than competitors. But the broader bias is reality, and Pangram’s own acknowledgement that it performs less well on “niche cases” like poetry and recipes is a reminder that no tool is universal.

Neurodivergent writers are also disproportionately affected. Autistic writers often write in a highly precise, formal, and structured manner, avoiding the filler words, emotional hedging, and grammatical imprecision that characterise casual human writing. AI models have scraped enormous quantities of clear, explanatory internet text, much of which was written by neurodivergent people who communicate most comfortably in writing; thus, the models emulate that style. In Alpern’s NYT article, an autistic ESL teacher shares that “AI language sounds the way it does because it borrowed from autistic people first”. So neurodivergent writers are accused of sounding robotic for writing the way they naturally communicate.

On Substack, false positives carry serious consequences, as a writer who is publicly flagged (even by a single reader) risks losing subscribers, being targeted by online mobs, and having their reputation permanently tainted by an accusation that, in the current climate, is morally charged in a way that is disproportionate to the actual harm of AI-assisted writing. Being labelled as a producer of ‘slop’, even falsely, can be career-ending for a creator whose livelihood depends on reader trust.

This creates a concerning asymmetry in which the people who are most likely to be falsely accused are those who write with the most care: non-native speakers who have worked hardest to master formal English, neurodivergent writers who communicate with unusual precision, and professional writers who have internalized the conventions of their craft. Meanwhile, the people actually using AI to generate content can often evade detection by using “humanizer” tools or by mixing AI-generated text with human text in ways that confuse classifiers.

Responding To AI Allegations On Substack

If you’re accused of using AI on Substack, regardless of whether you have or not, how you respond matters.

If you do use AI, be transparent about it (you can get ahead of this by using the “How I Make This” statement feature that Substack has introduced alongside the Pangram integration). So, if you use it for brainstorming, translation, outlining, grammar checking etc., you can say so there. Being up-front about it may minimize backlash from a detection tool result showing that you have used AI tools in your process. However. Be mindful that disclosing AI authorship might, in itself, damage perceptions of trustworthiness, caring, competence, and likability, especially in interpersonal and emotional writing, depending on the audience’s AI literacy.

If you don’t use AI and the detector has thrown a false positive, the most important thing is not to get excessively defensive about it. I know you will want to get defensive, because being falsely accused naturally makes us want to defend ourselves, but being defensive also makes people think we have something to hide, even if they themselves would react that way. Address the accusation calmly and directly; use it as an opportunity to educate your audience about the limitations of AI detection tools, even ones that claim accuracy to the degree that Pangram does.

Pangram and similar tools are statistical inferencers, not forensic detectors, so they make probabilistic guesses based on patterns, and those guesses are wrong in predictable ways. Explain that your writing style, precision, formal grammar, non-native English background, or whatever fits your circumstances best may have triggered the tool, and then stand your ground. If you have drafts, notes, outlines, etc. that you’ve saved, you can share those, although I know a lot of Substack writers won’t have such things (basing this on my personal approach); many of us don’t keep detailed notes on online blog articles. I write my articles partially in Word, partially in Notepad, and partially in the Substack app itself, copy/pasting and editing as I go. If you use a similar approach, you can take screenshots of the version history on Substack itself:

It will show you the timestamps of the changes you made to the article. I scrolled all the way down; it shows I started this at around 1:14 pm and it’s now 3:20 PM (no, I’m not that slow, I took a break or two):

Of course, this won’t be feasible if you write entirely in Word or another program and copy/paste over (which, by the way, is a completely reasonable and normal way to write an article, so if someone says that’s what they did, it’s not reasonable to assume they’re lying and must be using AI…)

What AI Detector Integration Does To Reading

I might write a more detailed breakdown of the reputational risks and responses to AI false positives later, but in this article I wanted to get into a second point as well because I think it’s more timely and relevant given the recent addition of Pangram to this platform: what AI detector integration does to reading.

There’s a lot of trust involved in reading writers on Substack. When we open an article, we’re giving the writer our attention in exchange for their honest perspective. Introducing an AI detector into this reader–writer relationship brings in a third party that claims to tell us whether the writer can be trusted, rather than the writer doing that job or us relying on our own discernment to make the decision. I don’t like this aspect of it. Reading is a leisure activity to me, and the “use this tool to see if what you’re reading is human!” aspect makes reading feel more restless. I don’t want to be encouraged to approach every article with suspicion. I want to think about the ideas and arguments, the emotional resonance, the thought that went into it, the value the writer wants to provide.

If we read with “is this AI?” in mind, even if just subconsciously, we’re scanning for ‘tells’ instead of engaging with the words and ideas themselves. Lecturer of Writing and Rhetoric at the University of Mississippi Marc Watkins spent a week using Pangram’s Chrome extension and found that the stoplight-style color coding (red for AI, yellow for mixed, green for human) changed his engagement habits completely. He wrote in his Substack post:

I stopped interacting and reading posts and instead focused on labels. When we allow a company to place labels on our social media interactions, we cede some agency to opaque systems, ultimately giving them a great deal of power over our interactions.

Hypervigilance like this sucks all the joy out of reading and makes it harder to receive a piece of writing on its own terms. If a beautifully crafted paragraph moves us, but a scanner tells us it’s 44% AI-assisted, does that sentence really lose its meaning?

A piece of writing can be genuinely useful and AI-assisted, and we don’t need to know whether it’s AI-assisted or not to make a judgment on whether it’s useful. We can use our own discernment and the context around what is being communicated. Blindly checking everything for AI and using that score to decide whether a piece is worth reading misses the nuance of human communication. A personal apology, a grief memoir, a letter to a community about a difficult decision are contexts where the writing is the human connection, and if those were generated by AI, that makes the sentiment hollow, so the context demands that a human actually wrote it. But an essay on how the First World War started, for example… the context of that is in the accuracy and clarity of the information. If a writer used AI to help structure their argument, check their facts, or improve the clarity of a technical explanation, the value of the information doesn’t change, and the writer’s ideas, curation, and editorial judgment remain. Generative AI can organize information and improve structural clarity relatively well.

But a blanket AI detector can’t make the distinction between whether the 30% of a post that Pangram flags as “AI-assisted” is a structural edit to a paragraph the writer agonised over for hours or whether the entire thing was generated in 0.01 seconds and thrown in with no consideration. It also doesn’t distinguish between AI-generated text and AI-translated text, or whether the writer has just internalised a formal writing style that happens to resemble AI’s phrasing. You get a number and have to draw your own conclusions, and that number is being provided to anyone who wants it in an environment where the conclusions being made are increasingly uncharitable.

There’s also somewhat of a double standard here because ghostwriting has been a thing for decades. We’ve been reading text that wasn’t actually written by the person in the byline for centuries, and we’ve largely been OK with this. We know that the majority of celebrity memoirs are ghostwritten and that politicians and leaders employ professional copywriters to structure their ideas in their voice. The named person provides the concepts, life experience, and anecdotes; the hired professional handles the syntax, structure, and research. We’re fine with this and don’t question the person’s authenticity because of it. There’s no mass movement to scan books for ghostwriting, and nobody is developing a browser extension that labels a politician’s speech as “PR-assisted. It’s so normalized that it barely registers as a topic of conversation. If, as Chris Best said in his Substack Post article introducing the Pangram integration, we “access human-made art because [we] know there’s a human behind it and that’s what [we’re] looking for, other humans, showing me in art what they hide in their selves”… Why do we accept ghostwriters (because there is still a human mind interpreting the author’s intent, however mediated) but not, potentially, the work of a Japanese writer who has AI-translated her work into English?

The value we place on writing has never only been about who typed the words. We’re interested in people’s ideas and the context of those; the relationship between the writer and reader, which can’t be diminished into a number provided by a robot looking for patterns in phrasing.

Focus On Ideas, Not Words

Substack’s decision to integrate Pangram is an understandable response to a real problem, as the internet is being flooded with AI-generated content that, in many cases, isn’t actually worth reading. The concern that readers might invest their time and money in a newsletter written by no one is legitimate. But the solution Substack has chosen creates as many problems as it solves. Giving readers a tool to audit the statistical patterns of every piece of writing on the platform is introducing a culture of suspicion that will disproportionately harm the writers that Substack, presumably, wants to keep on its platform.

And it changes the experience of reading on here in ways that are difficult to walk back, because once readers start using a tool to check whether an author can be trusted instead of relying on their own discernment, it’s hard for them to stop using it and, by extension, stop being suspicious of everything they read. Essentially, the integration of Pangram has introduced suspicion into the reading experience, which takes something away from the reader–writer relationship that is difficult to restore.

The value of this platform is in the trust between writers and readers and the sense that you are hearing from a real person with real ideas and real skin in the game. An AI detector can’t tell you if an idea is worth your time or if a writer has something genuine to say. That’s a job for your brain. Use it.