Gergely Orosz

Hacker News

Peter Yang

Substack shipped an AI detection feature this week, powered by a company called Pangram, letting readers scan any post, note, or comment over 100 words for an estimate of how much was written by AI versus a human. CEO Chris Best framed it as a transparency play, not a ban: “Anti-slop is a pro-AI stance.” Writers can scan their own drafts before publishing and add a voluntary disclosure note. The platform even coined a term for what it’s fighting: “Claudefishing” — content that mimics a human voice but was generated by AI.
The reception has been warm in some corners. Gergely Orosz called it a great initiative, noting he simply stops reading anything he suspects is AI-written. Peter Yang plugged in accounts of prolific X posters and found what he expected. The general vibe: finally, someone’s doing something.
Here’s the problem. Freddie deBoer, a writer who got flagged by Pangram after someone accused him of AI-writing a post, spent time actually stress-testing the tool — and the results are not reassuring. The same 300-word passage Pangram called 100% AI-written sits inside a 5,000-word essay Pangram calls 100% human-written. Sub-sections of the flagged passage? Also 100% human. The percentage meter seems to only return polar results — 100% one way or the other — which is a strange behavior for a probabilistic tool. He also says he induced false positives in human-written text in about 15 minutes of deliberate effort.
DeBoer’s core argument isn’t that Pangram is useless. It’s that using it as a one-shot verdict is epistemologically reckless. Even Pangram’s own stated false-positive rate produces statistically expected errors at scale. When the professional consequences for writers are real, that matters.
Substack is betting that human-verified writing becomes a scarce, valuable commodity. That’s a reasonable bet. But the tool they’ve chosen to enforce that bet has a documented brittleness problem — context-dependent, gameable, and prone to confident wrong answers. “Claudefishing” is a catchy term. Whether Pangram can actually catch it is a different question.

Trends

Substack launched an AI detection tool powered by Pangram, letting readers scan posts and comments for AI involvement. CEO Chris Best frames it as 'anti-slop is a pro-AI stance' — he's not against AI assistance but wants to prevent Substack from becoming a content farm. The move is a veiled jab at LinkedIn's AI-saturated feed. The tool is a transparency mechanism, not an enforcement one, giving readers a probabilistic signal rather than hard bans. Critics note AI detection is unreliable and risks flagging non-native English writers as AI-generated.

Substack's AI detector is a shot at LinkedIn, not AI itself

We’ve AI detection since it’s early days. Few years now

In today’s rapidly evolving, technologically sophisticated digital landscape—it has become increasingly difficult to ascertain, with absolute confidence, whether a given piece of written content was produced by a human being or generated by an artificial intelligence system.

To be fair this is my take on AI content in general. AI itself is OK AMD beneficial but slop is not

Thanks for sharing this information.