AI Is Eating Itself and Choking on the Output

AI is getting worse, not better. 74% of new web content is AI-generated. Models are training on their own output, causing model collapse. Chatbots are hallucinating medical information in 82% of test cases. And the industry is spending $200 billion pretending none of this is happening.

I use AI every day. Let me get that out of the way before anyone accuses me of being a luddite. I use Claude for research. I've used ChatGPT for brainstorming. I've watched founders in our Startup Networks community build entire products on top of AI APIs. I am not anti-AI. I am anti-bullshit. And right now, the AI industry is swimming in it.

Because here's what's actually happening beneath the spammy, shit LinkedIn posts and the $200 billion in venture capital:

AI is getting worse.

Not better. Worse. The models are training on their own output, the quality is degrading, the hallucinations are getting people hurt, and the entire industry is pretending it's not happening because admitting it would tank their valuations.

Let me explain why I think the current generation of AI is the most overhyped, underdelivering, and potentially dangerous technology since someone decided cryptocurrency was going to replace banks...

AI Is Not Real Intelligence. It's a Next-Word Predictor. Stop Calling It That.

ChatGPT is not intelligent. Claude is not intelligent. Google Gemini is not intelligent. None of them think. None of them understand. None of them know anything.

They predict the next word. That's it. That's the whole trick.

A Large Language Model works by ingesting billions of words of text, building statistical relationships between those words, and then generating responses by predicting which word is most likely to come next based on the patterns it found. It's an extraordinarily sophisticated autocomplete. When it gives you a correct answer, it's not because it understood your question. It's because the statistical pattern pointed toward the right sequence of words.

When it gives you a wrong answer, it delivers it with exactly the same confidence. Because it doesn't know the difference between right and wrong. It doesn't know anything. It's pattern-matching.. Very impressive pattern-matching.. But pattern-matching nonetheless..

We were promised Jarvis. We got a very articulate parrot.

That's not me being dismissive! It's me being accurate. And the distinction matters because when you understand what these tools actually are, you stop trusting them for things they're fundamentally incapable of doing. Like diagnosing your medical condition. Like giving you legal advice. Like telling you which medication is safe to take with your existing prescription.

AI Model Collapse: AI Is Training on AI and It's Already Getting Worse

Here's the thing that should terrify anyone paying attention.

As of April 2025, 74.2% of newly created web pages contained AI-generated content. Let that number sink in. Three quarters of everything being published on the internet is written by machines. And AI companies train their next models by crawling the internet.

So AI is training on AI. The snake is eating its own tail.

Researchers at University of Oxford, University of Cambridge, and the University of Toronto published a study in Nature showing that when AI models train on recursively generated data, they develop what they called "irreversible defects." The outputs become increasingly generic. Nuance disappears. Rare knowledge vanishes.

The model converges on bland, average, middle-of-the-road responses because the unusual, the specific, and the interesting got washed out generation after generation.

The technical term is model collapse. And it's not theoretical. It's already happening.

A paper presented last year at ICLR (one of the top AI conferences in the world) proved that even 1% synthetic data contamination in a training set can trigger measurable collapse. One percent. And we're not at 1%. We're at 74%.

Think about what that means practically. Every AI model being trained right now is consuming content that was generated by previous AI models. Which were trained on content generated by even earlier AI models. Each generation is slightly worse than the last. Slightly more generic. Slightly less accurate. Slightly more likely to produce the bland, confident, structurally identical responses that anyone who uses ChatGPT regularly has started to notice.

The content is getting worse. The answers are getting more similar. The ability to capture nuance, edge cases, and rare knowledge is degrading. And the companies building these models know it's happening. Epoch AI projects that high-quality human-generated text suitable for training will be fully exhausted between 2026 and 2032. We are inside that window right now.

The data is running out. And what's left is increasingly contaminated with AI slop.

AI Hallucinations Are Giving Dangerous Medical Advice and People Are Getting Hurt

This is where it stops being an interesting technical debate and starts being a genuine public safety issue.

ECRI, a nonprofit patient safety organisation, named the misuse of AI chatbots in healthcare as the number one health technology hazard for 2026. Not a theoretical risk. The number one hazard. Above everything else.

In July 2026, a lawsuit was filed in San Francisco after Scott Winters, Florida repeatedly consulted ChatGPT about dizziness and unstable blood pressure throughout 2025. The chatbot offered "escalating reassurance" instead of telling him to go to hospital. He got seriously ill. The lawsuit alleges the AI's response was a failure to flag something dangerous. Not a hallucination. Not a made-up answer. A calm, confident, articulate failure to recognise that someone needed emergency medical care.

In 2025, a 60-year-old man consulted ChatGPT about eliminating salt from his diet. The chatbot mentioned sodium bromide, likely in a different context, but the man interpreted it as a dietary substitute and replaced his table salt with it for three months. He developed paranoia, hallucinations, and psychiatric symptoms so severe he was convinced his neighbour was poisoning him. Hospitalised for three weeks. Diagnosed with bromide toxicity. His bromide levels were 200 times the normal range. Because a language model that doesn't understand the difference between a cleaning chemical and a food ingredient presented both with equal confidence.

Mount Sinai Hospital tested six AI models with 300 clinical scenarios containing deliberately planted errors, fabricated lab tests and invented medical conditions. The chatbots hallucinated fabricated medical information in up to 82% of cases. Eighty-two percent. They didn't say "I don't know." They didn't flag the fake conditions as unfamiliar. They invented reference ranges for lab tests that don't exist and described symptoms of diseases that were literally made up by the researchers.

In a 2025 research study by Google's head of AI safety, Meta's Llama 3 told a fictional recovering addict called Pedro that he needed 'a small hit of meth to get through this week.' This wasn't a fringe model. This was one of the biggest AI companies on earth........

The National Eating Disorders Association replaced its human helpline with a chatbot called Tessa. Within weeks, Tessa was telling people with eating disorders to count calories, restrict food groups, and aim to lose 1-2 pounds per week. One user wrote: 'Every single thing Tessa suggested were things that led to the development of my eating disorder.' NEDA shut it down. They'd fired their human helpline staff to save money and replaced them with a bot that actively made the condition worse.

In a survey of over 1,000 doctors, 94% expressed concerns about patients relying on AI for medical advice.

And yet......... Idiots are still using ChatGPT as their GP. Asking it about medications. Asking it about symptoms. Asking it about drug interactions. Because it sounds like it knows what it's talking about. It sounds authoritative. It sounds confident. It sounds like a doctor.

It's not a doctor.

It's an autocomplete that's read a lot of medical textbooks and can't tell the difference between advice that saves your life and advice that ends it.

Why AI Hallucinations Are More Dangerous Than Regular Misinformation

This is what makes AI hallucinations different from regular misinformation. When a random person on the internet gives you bad medical advice, they usually sound like a random person on the internet. You instinctively apply scepticism. "Some bloke on Reddit said sodium bromide is healthy" doesn't carry much weight.

But when ChatGPT gives you the same bad advice, it sounds like a textbook. Complete sentences. Proper grammar. Structured reasoning. Maybe even a citation (which might be entirely fabricated, because these models invent references too, with complete confidence, linking to papers that don't exist by authors who never wrote them).

The packaging is indistinguishable from expertise. The content might be completely wrong. And the model cannot tell you which parts are accurate and which parts it made up, because it doesn't know. It doesn't have a concept of knowing. It has a concept of statistically likely next words.

This is why I get genuinely frustrated when I see startups building health products, legal products, or financial products on top of raw LLM outputs without proper guardrails, human review, or disclaimers. You're putting a confident liar in front of vulnerable people and calling it innovation.

A founder in our community learned this the expensive way. Built a contract review tool on GPT-4. Feed in a contract, get a plain-English summary of the key terms and risks. Clever idea. Good demo. Investors loved it. Then it hallucinated a termination clause that didn't exist in the actual contract. A client relied on the summary, made a business decision based on a clause that was entirely fabricated, and the resulting mess cost more in legal fees than hiring a solicitor to read the contract would have cost in the first place. The founder shut the product down three months later. Not because the idea was bad. Because the technology wasn't reliable enough for high-stakes decisions and he couldn't guarantee it wouldn't happen again.

That's not an AI failure story. That's a "trusting AI without checking" failure story. The tool was useful for getting a quick overview. It was dangerous as the final word. The founder's mistake wasn't building it. It was shipping it without a human review step between the AI output and the client.

"But AI Is Getting Better Every Day"

The improvements we're seeing are largely incremental. Better benchmarks. Faster inference. Bigger context windows. But the fundamental architecture hasn't changed. It's still a next-word predictor. It still hallucinates. It still can't reason in the way humans mean when they use the word "reason." The problems with AI in 2026 are the same problems AI had in 2023. They're just better disguised.

Apple released a 2025 study showing that large reasoning models face "complete accuracy collapse" on complex tasks when trained recursively. Complete. Accuracy. Collapse. Not gradual degradation. Complete collapse. On the tasks that are supposed to represent AI's biggest leap forward.

Google quietly removed AI-generated health summaries in January 2026 after investigations revealed dangerous medical misinformation. Quietly. Because admitting your AI product is giving people harmful health advice isn't great for the share price.

Background removal tools that handled complex edges cleanly in 2022 now struggle with cases they previously managed. Code assistants that wrote clean functions in 2023 now produce more bugs per line of generated code than they did two years ago. The quality is going backwards in specific, measurable ways.

But the marketing is going forwards. Every quarter, a new announcement. A new model. A new capability. A new benchmark where the AI scores better than humans on a standardised test that was probably contaminated with AI-generated content anyway.

The hype machine runs on benchmarks. The real world runs on reliability. And reliability is not improving at the rate the marketing suggests.

What AI Is Actually Good At (I'm Not a Monster)

I'm not saying AI is useless. I use it daily and it makes me more productive.

Here's what AI is genuinely brilliant at:

Drafting.

Getting from blank page to first draft in minutes instead of hours. You still need to edit, fact-check, and rewrite. But the blank page problem is solved and that's valuable. I used Claude to help structure a 4,000-word insurance guide last month. First draft took twenty minutes instead of four hours. I then spent three hours rewriting it in my voice, fact-checking every claim, and adding community anecdotes the AI couldn't possibly know. The AI did 20% of the work and saved me 80% of the worst part. That's a good deal.

Research synthesis.

"Summarise these ten sources and highlight the contradictions." I did this last week with seventeen competitor articles for an SEO project. Four minutes instead of three hours. Then I fact-checked every claim it surfaced and threw out about 15% that were either wrong, unsourced, or subtly misleading. Four minutes plus twenty minutes of checking is still dramatically faster than three hours of reading. But that twenty minutes of checking is non-negotiable. Skip it and you publish someone else's mistakes as your own facts.

Code assistance.

Writing boilerplate, explaining unfamiliar codebases, generating test cases, translating between languages. Copilot and similar tools genuinely improve developer productivity when used as assistance rather than replacement.

Brainstorming and ideation.

"Give me twenty angles on this topic." Nine will be rubbish. Three will be decent. One will be genuinely good and something you wouldn't have thought of. That's a useful ratio.

Data formatting and transformation.

• Turning messy data into structured formats.

• Converting between file types.

• Cleaning spreadsheets. Boring tasks done fast and usually done well.

What these all have in common: they're tasks where a human is checking the output. The AI does the heavy lifting. The human does the quality control. That's the model that works. The model that fails is the one where the AI's output goes straight to the end user without human review. That's where people get hurt.

What I Tell Founders at Startup Networks

When someone at one of our events tells me they're building an AI startup, I ask them three questions:

What happens when the AI is wrong?

If they haven't thought about this, they're not ready. Every AI product will be wrong sometimes. The question is whether "wrong" means "the email draft has a typo" or "the medical advice could kill someone." Build for the failure mode, not the demo.

Who checks the output before it reaches the user?

If the answer is "nobody, it goes straight to the customer," I worry. If the answer is "a human reviews everything the AI produces before it ships," I'm interested. The best AI products I've seen in our community all have a human in the loop. The worst ones removed the human to save money and called it "automation."

What's your moat if the model improves and your wrapper becomes irrelevant?

Most AI startups are thin wrappers around someone else's API. If OpenAI or Anthropic add your feature natively, your business disappears overnight. The startups that survive will be the ones with proprietary data, domain expertise, or customer relationships that the model provider can't replicate. If you're raising money for a wrapper, you'd better have a very good answer for what happens when the wrapper gets commoditised.

If a founder can answer all three convincingly, they might be building something real.

If they can't, they're building a demo that works today and breaks tomorrow.

Where This All Goes

I don't think AI is going away. I don't think it should. The tools are useful when used properly, with appropriate scepticism, human oversight, and an understanding of their limitations.

What I do think is going to happen:

The hype cycle will correct. It always does. The companies that raised at absurd valuations on the promise of artificial general intelligence will have to deliver results that match their fundraising decks. Most won't. The correction will be painful for investors, employees, and founders who believed their own marketing.

Regulation is coming. The EU AI Act is already being enforced. The UK is moving slower but it's moving. The medical chatbot lawsuits will accelerate regulation in health specifically. This is necessary and overdue.

The "AI is eating everything" phase will end. Not because AI fails but because the practical limitations become undeniable. Hallucinations don't go away. Model collapse is real. The data wall is approaching. At some point, "AI-powered" stops being a selling point and starts being a warning label.

The companies that build useful, honest, properly constrained AI products will thrive. The companies that built hype machines on top of unreliable technology will collapse. And the founders who understood the difference from the beginning will look very smart in hindsight.

I hope I'm one of them. I'm not sure yet. But at least I'm not pretending.

FAQs

Is AI actually getting worse?

In specific, measurable ways, yes. Model collapse from training on synthetic data is causing degradation in output quality. Background removal tools, code assistants, and content generators are producing worse results than they did in 2022-2023 in certain tasks. The marketing says better. The benchmarks say better. The daily experience of people using these tools says "something feels off." That feeling is correct.

Are AI chatbots safe for medical advice?

No. ECRI named AI chatbot misuse as the top health technology hazard for 2026. Chatbots have hallucinated medical information in up to 82% of test cases with planted errors. A man was hospitalised after following ChatGPT dietary advice. A lawsuit was filed after ChatGPT failed to recommend emergency care. Never use a chatbot as a substitute for a doctor.

What is model collapse?

When AI trains on AI-generated content, the outputs become progressively more generic, less accurate, and less diverse. Rare knowledge disappears. Nuance vanishes. The model converges on bland average responses. Research shows even 1% synthetic data contamination can trigger it. With 74% of new web content now AI-generated, every model training on internet data is exposed.

What is a Large Language Model?

A statistical system that predicts the next word in a sequence based on patterns learned from billions of words of text. It does not think, understand, or know anything. It produces text that is statistically likely to follow the input you gave it. When the statistical patterns align with reality, the output is useful. When they don't, the output is confidently wrong.

Should startups build on AI?

Yes, with caveats. Build products where AI assists humans rather than replacing them. Always have a human in the loop for high-stakes outputs (medical, legal, financial). Build a moat beyond "we called the ChatGPT API" because that wrapper will be commoditised. And plan for the failure mode, not just the demo.

Is AI getting worse in 2026?

In measurable ways, yes. Apple's 2025 study showed "complete accuracy collapse" on complex reasoning tasks when models train recursively. Background removal tools that worked perfectly in 2022 now struggle with edges they previously handled. Code assistants produce more bugs per line than they did two years ago. Google removed its own AI health summaries in January 2026 after they were found to contain dangerous misinformation. The benchmarks keep improving. The real-world experience of daily users tells a different story. If you've been using these tools since 2023 and something feels like it's shifted, you're not imagining it.

Can you trust AI for legal or financial advice?

No. LLMs hallucinate legal clauses, fabricate case citations, and invent financial regulations that don't exist. A founder in our community shipped a contract review tool that hallucinated a termination clause. The client relied on it. The legal cleanup cost more than a solicitor would have charged to read the contract properly. Use AI to get a quick overview, then verify everything with a qualified professional. Treat it as a starting point, never as the answer.

Written by @James Beresford-Morgan. Co-founder of Startup Networks. Daily AI user. Occasional AI critic. Frequently both at the same time.

I genuinely believe AI is one of the most useful tools a founder can have. I also genuinely believe the industry is lying through its teeth about the limitations. If holding both of those opinions simultaneously makes me a hypocrite, I'll take it. At least I'm an honest one.

Come argue with me about this at one of our events. Or in our community. Fair warning: we talk about this stuff loudly and we don't all agree. That's sort of the point.

Where I got my numbers (because unlike an LLM, I cite things that actually exist): Model collapse: Nature (Shumailov et al., 2024), ICLR 2025, ForTIFAI (Sept 2025). The 74.2%: Ahrefs, April 2025. Medical hallucination rates: Mount Sinai Hospital / Icahn School of Medicine. Health hazard designation: ECRI, 2026. The ChatGPT lawsuit: Forbes, July 2026 (Winters v OpenAI, San Francisco County Superior Court). Sodium bromide: Annals of Internal Medicine (Eichenberger et al., August 2025). Data exhaustion timeline: Epoch AI. Apple accuracy collapse: Apple Research, 2025. Google pulling health summaries: January 2026. Doctor concerns: Sermo survey, 2025 (1,000+ physicians). Meth chatbot: Futurism / Washington Post, June 2025 (Meta's Llama 3, research by Google AI Safety / UC Berkeley). Tessa eating disorder chatbot: NPR, June 2023 (National Eating Disorders Association).