Category: Confident and Wrong

Series: Confident and Wrong — Article 7 of 7

Intro:

The only thing a brand can do about AI’s confidence problem is build better evidence. Not fix the problem — the problem is structural and cannot be fixed from outside the training architecture. But build conditions under which the model is less likely to be confidently wrong about you specifically. That is what Entity Confidence is. It is risk management, not a solution.

Confident and Wrong — Article 7 of 7.

To read the whole series from the start click here

Back to previous article

What This Series Has Established

This series has documented a structural argument across six articles. The summary is blunt.

AI is trained to sound certain rather than to be accurate. The training incentive is clear: human raters reward confident, fluent outputs over hedged, appropriately uncertain ones. The result is systematic miscalibration — models that express more certainty than their evidence warrants, at every output, by design. A second compounding mechanism, ownership bias, adds up to 26% additional confidence inflation when a model responds in its own voice — which is every standard commercial interaction. Neither mechanism is corrected in any default commercial deployment. The fix exists. It is not applied.

Fabrication is accelerating, not stabilising. The Lancet data is unambiguous: one fabricated citation in every 277 academic papers in early 2026, up from one in 2,828 in 2023. A sixfold increase in two years, with no mechanism in the data that would cause it to level off. Academic publishing is the most adversarial environment for AI fabrication — with professional incentives, peer review, and editorial oversight. The commercial context has none of those checks. The 1-in-277 figure is a lower bound for brand misrepresentation, not an upper bound.

The standard defence — SEO rank — is weakening. Google AI Overviews cited sources from the top 10 organic results 76% of the time in July 2025. By January 2026 — six months later — that figure was 38%. The mechanism is query fan-out: AI engines decompose queries into multiple sub-queries, each retrieving from a different slice of the web, so the set of sources cited is much wider than the set that traditional SEO optimises for. A top-10 ranking provides less protection against AI misrepresentation than it did six months ago. The trajectory is not reversing.

Content credentials certify history, not truth. C2PA — the most serious provenance infrastructure in existence, backed by the Five Eyes agencies, mandated by the EU AI Act, shipping on the Google Pixel 10 — answers who made a piece of content, when, and through what channel. It cannot answer whether the underlying claims are accurate. The Five Eyes advisory said this explicitly: Content Credentials “can answer the who, when, where, and how parameters of trust, but not the why.” Provenance is a floor. It is not a ceiling.

Training data provenance is architecturally impossible to trace. No AI vendor can tell you which specific training documents produced a specific model belief, because large-scale neural network training does not work that way. Beliefs are distributed across billions of parameters, shaped by the entire corpus, not by isolable source documents. This is not a gap that audits, disclosure requirements, or data cards will close — the mechanism of distributed representation is what makes the tracing impossible. There is no chain of custody between an AI claim about your brand and any specific source document that might have generated it.

These are not bugs awaiting a patch. They are architectural properties of how large language models are built, trained, and deployed. A better model, a new regulation, or a commercial partnership with an AI vendor does not change any of them.

What Brands Cannot Do

The structural argument matters because it rules out a category of responses that brands might otherwise invest in.

Brands cannot reach the training layer. When a model has already learned something inaccurate about your brand — from a poorly-sourced news article, a competitor’s claim, a fabricated citation that entered the training corpus — there is no correction pathway analogous to calling a journalist and asking for a correction. The belief is distributed across the model’s parameters. There is no specific location to update.

Brands cannot file a correction with an AI system the way they can with a publication. Editors can issue corrections. Journalists can retract. An AI system has no equivalent process. The outputs of a deployed model are a function of its training, which is fixed. A prompt-level correction (“that’s wrong, here’s the right information”) may update the current conversation. It does not update the model. The next user receives the same pre-training representation.

Brands cannot buy their way to accurate AI representation. [The Partnership Paradox](/articles/partnership-paradox) — documented elsewhere on this site — established this empirically. ChatGPT correctly identified one out of ten San Francisco Chronicle excerpts despite Hearst’s formal OpenAI content partnership. Muck Rack’s analysis of over a million AI citations found that the Financial Times, Time, and Axios — all with OpenAI licensing agreements — are cited more frequently by Gemini, which has no agreements with any of them. Commercial relationships operate at the executive level. Citation engines operate on the editorial record. The two are separate systems.

Brands cannot use content credentials to establish that their AI representation is accurate. C2PA can certify that a brand’s content was produced by verified tools at a verified time. It cannot certify that what the AI says about the brand is true.

The One Layer Brands Can Influence

There is one layer of the AI stack that is accessible from outside: the retrieval layer.

Not all AI responses are generated from pre-training alone. Many commercial AI systems — including Google’s AI Overviews, Perplexity, and increasingly ChatGPT in browsing mode — retrieve sources at inference time and incorporate them into their response. This is Retrieval-Augmented Generation (RAG): the model pulls relevant documents at the moment of query, uses them to ground its response, and may cite them as sources.

The retrieval layer is where most mid-sized brands’ AI representation is actually determined. The reason is scale. Pre-training corpora are dominated by the most-linked, most-covered, most-discussed entities on the internet — predominantly large enterprises, global brands, and organisations with decades of web presence. An SME that does not have strong pre-training representation is not, in most AI queries, being answered from what the model learned in training. It is being answered from what the retrieval system finds at inference time.

This distinction has a practical consequence: the retrieval layer responds to inputs that brands can build.

Earned editorial coverage in sources AI engines trust. The Muck Rack analysis found 82% of AI citations come from earned media — independent editorial decisions by credible outlets. ConvertMate’s GEO Benchmark Study (12,500 queries, 8,000 domains) found brands are 6.5 times more likely to be cited via third-party sources than via their own domain. YouTube now accounts for 5.6% of AI Overview citations — a channel that did not register in earlier citation analyses. The editorial record, not the brand’s own publishing, is what the retrieval architecture reads.

Consistent entity signals across platforms. Structured data (schema markup), Wikidata presence, consistent NAP (name, address, phone) data, and verified business listings are signals that AI retrieval systems use to resolve entity identity. When signals are consistent and corroborating across multiple independent platforms, the model’s confidence that it has correctly identified the entity is better-founded. When signals are inconsistent or absent, the model is filling gaps — which is where fabrication risk is highest.

Content that AI engines can parse and retrieve. BLUF structure (Bottom Line Up Front), direct definitional language, topical depth, and clear section openings are not styling choices — they are functional requirements for retrieval-mode citation. A page that answers a specific question in its first sentence is structurally more likely to be retrieved and cited for that question than one that contextualises before answering.

What Entity Confidence Is — and Is Not

Entity Confidence is a proxy metric for how well-grounded an AI’s knowledge of a specific brand is.

A high EC score means the brand’s AI representation is backed by consistent, corroborating, independently produced evidence from multiple credible sources. Earned media coverage, consistent entity signals, primary source documentation, structured data, third-party editorial attribution — these are the inputs. The score is a read on the evidentiary quality of what AI systems have available when they generate claims about the brand.

It is not a guarantee of accurate representation. The confidence gap — the structural miscalibration between AI’s expressed certainty and the accuracy of its outputs — cannot be closed from the outside. A brand with a high EC score will still sometimes be misrepresented. The miscalibration mechanism is in the model, not in the evidence.

What a high EC score changes is the probability distribution. A model working from rich, consistent, independently corroborated evidence has less gap for fabrication to fill. The fabrication mechanism — generating plausible-sounding text when evidence is absent or thin — is less likely to activate when evidence is present and strong. This is not a theoretical claim about how models should work. It is the same logic that explains why the Lancet data shows lower fabrication rates in journals with stronger editorial oversight: more rigorous checking reduces the propagation of fabrication. More evidence reduces the gap that fabrication fills.

Entity Confidence is therefore risk management, not a solution. The honest framing is this: you cannot make AI accurate about your brand. You can make it harder for AI to be wrong about your brand. Those are different claims, and the distinction matters.

EC does not compete with the confidence gap. It manages exposure to it.

The Honest Argument for Building It Anyway

The series has established that the confidence gap is widening: overconfidence is trained in, fabrication is accelerating, standard defences are weakening, provenance cannot certify truth, and training data cannot be traced. None of that is likely to improve on the timescales that matter for a brand building its position now.

In that environment, there are two positions a brand can take.

The first is to rely on the hope that AI happens to have accurate information about you from pre-training, or that its retrieval system happens to find good sources, or that users happen to verify what AI says before acting on it. The first two are functions of how much independently credible material about your brand exists in the places AI looks. The third is not happening at scale — the Deloitte finding that 38% of business executives made wrong decisions based on hallucinated AI output in 2024 is an operational measure of how rarely verification happens in practice.

The second position is to build the evidentiary record that makes accurate AI representation more likely. Not certain. More likely. Earned coverage in credible outlets. Consistent entity signals across platforms. Primary sources that can be retrieved and cited. Content structured for AI parsability. A Wikidata entry. A schema markup foundation. These are the signals that the retrieval layer reads. They are the only signals that brands can build.

Every brand that does not build this record is implicitly accepting whatever AI happens to say, with whatever confidence AI happens to express it. The confidence, as this series has documented, is structural and inflated. The accuracy, as the Lancet data and the CJR study both establish, is unreliable. A brand’s decision not to invest in its entity evidence is not a neutral decision — it is a decision to leave AI’s representation of it ungoverned.

The Closing Argument for the Series

The Confident and Wrong series has documented a widening gap between what AI asserts and what is actually true. The gap is widening because all four of its structural drivers are moving in the same direction simultaneously: overconfidence is trained in and not corrected, fabrication is accelerating through a self-reinforcing data loop, standard defences are weakening as query fan-out dilutes ranking’s protective effect, and architecture prevents the kind of provenance tracing that might enable systematic correction.

These are not temporary problems. They are features of the current generation of AI systems, and the mechanisms that produce them are not being systematically addressed. The training incentive that rewards fluent overconfidence has not been removed. The ownership bias fix is not deployed. The fabrication trend is not reversing. The SEO-to-AI-citation relationship is not restoring. The training data tracing problem is not solved.

Entity Confidence is not the solution to this. There is no solution to this from outside the model. What Entity Confidence® is — the only honest description — is the best available response: a systematic approach to building the evidentiary infrastructure that gives AI the best possible material to work from, so that when it generates confident claims about your brand, those claims are more likely to be grounded in something accurate.

That is the counter. It is not a complete defence. It is the only lever available.

The choice for any brand is not between accurate AI representation and inaccurate AI representation. It is between leaving AI’s representation of you to chance and doing the only available work to shift the odds. The confidence gap cannot be closed. The risk it represents can be managed.

Build the evidence. It is the only counter there is.

*Sources: Ghafouri, Yao et al., “Epistemic Integrity in Large Language Models,” arXiv:2411.06528 (November 2024); arXiv:2606.03437, “Large Language Models Are Overconfident in Their Own Responses” (June 2026); Topaz et al., “Fabricated Citations in Scientific Literature,” The Lancet (May 2026); CJR / Tow Center for Digital Journalism, “We Compared Eight AI Search Engines. They’re All Bad at Citing News.” (2025); Deloitte, AI in the Enterprise (2024); Muck Rack Generative Pulse (via Nieman Lab, July 2025); Ahrefs Brand Radar, AI Overview citation analysis (February 2026, data collected January 2026); ConvertMate GEO Benchmark Study 2026; NSA/ASD/CCCS/NCSC-UK Five Eyes Content Credentials Advisory (January 2025); OpenAI o3/o4-mini system card.*

Series: Confident and Wrong — Article 6 of 7

Intro:

When an AI engine makes a confident claim about your brand, there is no chain of custody from that claim back to a verified source. This is not a gap that providers have chosen not to close. It is a structural consequence of how large language models are built — and the architecture that creates it is stable. It is not going to be fixed by better practices, more careful documentation, or the maturation of provenance standards.

Confident and Wrong — Article 6 of 7.

To read the whole series from the start click here

Two Types of Provenance — and Only One of Them Is Addressable

Provenance, in the AI context, has two distinct meanings. Conflating them produces the wrong conclusion about what can be fixed and what cannot.

Output traceability is the question of whether we can identify which model generated a specific piece of content. This is addressable. Watermarking, digital signatures, cryptographic attestation, and model fingerprinting all address this layer. The C2PA standard — covered in Article 5 of this series — addresses this layer for AI-generated content: it can tell you who created something and when, attaching a signed provenance chain to the output.

Input traceability is the question of whether we can identify what training data produced a specific claim. This is not addressable. There is no mechanism — technical, legal, or procedural — by which a user, a brand, or a regulator can trace an AI’s confident assertion about a brand back to the specific training documents that produced it. The chain of custody was never established. It cannot be reconstructed retrospectively.

Article 5 showed that C2PA certifies history, not truth. This article goes upstream of that problem. Before we even reach the question of verifying truth, we cannot trace the claim to its origin. C2PA addresses the output layer — content that was created and signed. This article addresses the input layer — the training data that shaped what the model knows. These are separate problems, and only one of them has a technical solution.

Why Input Traceability Is Structurally Impossible

A 95-paper systematic review published on arXiv in December 2025 (arXiv:2601.14311, submitted to ACM Computing Surveys) establishes this directly. The survey spans a decade of research across data provenance, transparency, traceability, bias, privacy, and tooling in large language models. Its headline finding is not that providers have been careless about documenting their training data. It is that the opacity of LLM training data is architectural, not incidental.

The authors identify “the inherent trade-off between transparency and opacity” as the central structural tension. Full transparency about training data would expose proprietary datasets, copyright-sensitive material, and privacy-sensitive content. These are not hypothetical risks — they are concrete legal and commercial liabilities for every major provider. Opacity is therefore the stable commercial equilibrium. Not a shortcut taken by corner-cutting providers. The equilibrium.

The scale of LLM training compounds the structural problem. Modern large language models are trained on corpora containing hundreds of billions of tokens drawn from trillions of web documents, books, code repositories, and other sources. At that scale, the tracking infrastructure required to maintain a chain of custody from every output claim back to every input document does not exist and has no realistic path to existence. The training process is not a lookup operation with a traceable query log. It is a compression of patterns across an incomprehensibly large corpus into model weights — and once those weights are set, the specific documents that shaped them cannot be reconstructed from the outputs they produce.

This is the architecture. The survey’s contribution is to establish that this opacity is systematic across the field, not a choice by any one provider. Individual companies are not simply failing to disclose; they are operating within an architecture that makes the disclosure structurally unavailable.

What Watermarking Does and Does Not Solve

The survey covers watermarking as a technical partial remedy. Understanding what watermarking addresses — and what it does not — is important for assessing whether the chain-of-custody problem has any current technical solution.

Watermarking addresses output traceability: embedding a signal in generated text that allows the output to be attributed to a specific model or provider. This is a genuine technical advance. If an AI-generated piece of content is found to contain errors or fabrications, watermarking can confirm which model produced it. That is a meaningful accountability mechanism at the output layer.

Watermarking does not address input traceability. A watermark confirms that this text came from this model. It says nothing about what training data shaped the model’s knowledge of the subject. The claim about your brand embedded in watermarked text is still a claim with no traceable lineage — we now know which model made it, but not what the model drew on to make it. The chain of custody still ends at training time and is not reconstructable.

The two problems are distinct. Only one has a technical solution. The technical solution to output traceability — watermarking, C2PA, digital signatures — does not transfer to the input traceability problem. It does not narrow the gap. It addresses a different problem entirely.

The Two Routes AI Takes for Brand Knowledge

Understanding why the chain-of-custody problem is intractable requires understanding how AI systems acquire brand knowledge in the first place. There are two routes, and neither provides verified primary provenance.

Route one: training data. During pre-training, the model absorbs claims about your brand from text on the web as it existed at training time. The documents that contained those claims may no longer exist. The quality of those documents was never evaluated. The information may be outdated — sometimes by years. And the process by which the model weighted and compressed those claims into its parameters is not documented or recoverable. The chain of custody ends at training time and is gone.

Route two: retrieved content (RAG). At inference time, retrieval-augmented generation systems fetch documents from the web in response to a query. The source document is, in principle, identifiable — this is the closest thing to traceable provenance that currently exists in AI systems. But traceable does not mean verified. The AI may not disclose which documents it retrieved. The documents it retrieves may themselves be secondary or tertiary material — a blog post summarising a press release summarising a study. And retrieval operates on semantic relevance, not source quality: a well-optimised marketing page may outrank a peer-reviewed study because it is more similar to the query, not because it is more accurate.

Neither route establishes verified primary provenance. The training route produces claims with no recoverable lineage. The retrieval route produces claims with a recoverable but unverified source — and one that is weighted by optimisation, not evidence quality.

The confident tone the AI applies to outputs from both routes is not a reliability signal. It is a training artefact — the product of the same overconfidence mechanisms documented in Articles 2 and 3 of this series. The model does not express more confidence when drawing on better-sourced material. It expresses the same confidence regardless.

Why Retrieval Is Not a Fix — and Why It Still Matters

RAG is often positioned as the solution to training-data opacity. The argument is that if the model retrieves documents at inference time, the sources are knowable and the chain of custody is re-established.

This argument is partially right and partially wrong in ways that matter.

RAG does re-establish a link between the AI’s output and a specific document that was retrieved. That is real, and it is meaningfully different from pure training-data reliance. A claim grounded in a retrieved document is, in principle, more auditable than a claim from pre-training.

But the retrieved document is not the chain of custody. It is one link in a chain that starts from a verified primary source — and that link may be secondary material. Semantic relevance scoring determines what gets retrieved; source quality does not directly determine retrieval rank. The AI selects the documents that are most relevant to the query, which is not the same as the documents that are most accurate about the subject.

For brands, this distinction has a practical implication. The retrieval layer — what AI systems retrieve at inference time — is the only layer brands can currently influence. A brand that earns coverage in high-authority outlets, publishes original structured content, and maintains consistent entity signals across the web is building the content pool that retrieval systems draw on. This is the channel EC addresses.

But influence over the retrieval layer is not the same as chain-of-custody provenance. What brands are doing when they optimise for AI retrieval is making it more likely that the AI retrieves their material, not establishing that the retrieved material will be treated as primary evidence. The AI’s confidence in the output is still a structural artefact, not a signal of the quality of the evidence chain behind it.

Why Brands Cannot Correct What They Cannot See

The absence of input traceability has a direct commercial consequence: brands have no ability to audit the training data that produced the AI’s confident statements about them. They cannot identify which documents shaped the model’s knowledge. They cannot submit corrections to specific training-data errors. They cannot request that the model revise a specific claim based on an inaccurate source — because the source is not identifiable.

This is not a temporary operational gap. It is the structural position of every brand with respect to every major AI provider. The architecture does not support the transparency that would be required for audit-and-correct workflows.

What brands can do is operate at the retrieval layer: shaping what the AI retrieves at inference time through earned media, structured entity data, original content, and consistent cross-platform signals. This is not access to the training layer. It is influence over the retrieval layer — a partial substitute that addresses the secondary channel, not the primary one.

For the training channel — the pre-training data that established the model’s baseline knowledge of your brand — the only available strategy is long-term. If a brand generates enough high-quality, frequently retrieved content over time, that content may enter future training corpora. But the timeline is model training cycles, not content publication cycles. And the specific documents that shape training remain unknown.

Where This Leaves the Series Arc

The first five articles in this series established a sequence.

Article 1 showed that premium models produce more convincingly wrong outputs — the confidence is consistent, the accuracy is not. Article 2 explained why: AI was trained to sound certain, and the training objective was fluency, not calibration. Article 3 quantified the failure rate: one fabricated citation in every 277, and accelerating. Article 4 showed that ranking-based defences are weakening as AI citation patterns decouple from organic search. Article 5 established that content credentials — C2PA and its equivalents — certify history, not truth. They can tell you who signed something and when, not whether it is accurate.

This article shows why the problem cannot be fixed at the infrastructure level: there is no chain of custody from an AI’s confident claim to a verified training source, by design, and the architecture is stable. The opacity is the equilibrium.

That leaves one question. If the training layer is opaque, the confidence mechanism is broken, the fabrication rate is rising, and the infrastructure cannot certify truth — what is the only counter?

Article 7 answers that.

Forward to next article

*Sources: arXiv:2601.14311, “Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMs” (submitted December 2025, under review at ACM Computing Surveys); NSA/ASD/CCCS/NCSC-UK Five Eyes Content Credentials Advisory (January 2025); TrueScreen, “C2PA: History, Limitations, and What’s Missing” (2026).*

Provenance Certifies History, Not Truth — And That’s a Brand Problem

Series: Confident and Wrong — Article 5 of 7

Intro:

In January 2025, the intelligence agencies of the United States, United Kingdom, Canada, and Australia published a joint advisory endorsing C2PA — the Coalition for Content Provenance and Authenticity — as the recommended technical standard for media provenance in national security contexts.

The document is the highest-authority institutional endorsement the standard has received. And buried in its technical framing is an admission that matters for anyone thinking about provenance and brands:

Content Credentials, the advisory states, “can answer the who, when, where, and how parameters of trust, but not the *why*.”

The most serious provenance infrastructure in existence — backed by over 6,000 organisations, mandated by EU law, shipping by default on the Google Pixel 10 — cannot tell you whether a single claim is true.

That is not a design flaw. It is the ceiling of what provenance infrastructure can achieve. And understanding where that ceiling sits changes how you think about what brands can and cannot build their AI credibility on.

Confident and Wrong — Article 5 of 7.

To read the whole series click here

What C2PA Actually Does

C2PA embeds a cryptographically signed manifest inside a digital file. The manifest records who created or modified the content, with what tools, at what time — creating a chain-of-custody log that accumulates as the file passes through different hands. Verification involves checking the cryptographic signatures, validating the content hashes, and reconstructing the chain.

The standard was built from two converging projects: Adobe’s Content Authenticity Initiative (focused on creator attribution) and the BBC/Microsoft Project Origin (focused on news disinformation). It merged in 2021 and now has a steering committee that includes Adobe, Amazon, BBC, Google, Intel, Meta, Microsoft, OpenAI, Sony, and Truepic.

Its institutional momentum is real. The EU AI Act mandates it for AI-generated content disclosure (Article 50, effective August 2026). It is being fast-tracked to become ISO standard 22144. The Five Eyes advisory explicitly frames it as the structural response to deepfakes, influence operations, and media impersonation — a provenance layer that makes it harder to successfully fabricate the origin of content.

All of that is what C2PA does when it works. The question is what it cannot do even when it works perfectly.

The Trust Gap

RAND Corporation’s analysis of the standard is direct: “C2PA signing tools do not verify metadata accuracy — manifests only confirm that specific tools made specific claims.”

This is what the Five Eyes document calls the trust gap. A staged photograph taken on a C2PA-enabled camera produces a technically valid manifest. A document that accurately records a false statement carries a signed provenance chain. A selectively edited video that removes the thirty seconds that would change its meaning is, from a C2PA perspective, unimpeachable — if it was captured and processed by verified tools.

Provenance answers: *who said this, and when, and through what channel?*

It does not answer: *is this true?*

That distinction matters because it is easy to conflate the two. A signed provenance chain feels like a marker of reliability. In the context of physical evidence — a chain of custody for a legal document, a provenance record for a piece of art — it often is. But physical chain-of-custody records work because the underlying evidence is directly examinable. Provenance for digital media certifies the pathway; it says nothing about what the content at the end of that pathway actually represents.

Four Limitations That Compound the Gap

The trust gap is the deepest limitation, but there are four structural problems that constrain C2PA’s reach before you even get to the truth question.

Metadata stripping eliminates C2PA manifests entirely when content is screenshotted or recompressed through social media platforms. This is so fundamental that the EU AI Act’s own regulatory text explicitly prescribes supplementary watermarking and logging requirements alongside the C2PA mandate — the regulation acknowledges that metadata stripping makes C2PA alone insufficient. RAND: “success depends on end-to-end ecosystem compliance, unrealistic in open systems.”

Forgery risk inverts the trust signal. Hacker Factor documented that “authenticated forgeries can be created in minutes using standard tools.” X.509 certificates — the cryptographic backbone of C2PA signing — are accessible to anyone prepared to pay for them. A valid manifest is evidence that verified tools were used; it is not evidence that the person using those tools had any claim to the identity they signed with. As C2PA verification becomes a trust shortcut, it also becomes an attack surface.

Economic exclusion creates a two-tier system. X.509 certificates cost approximately $289 per year through recognised Certificate Authorities — there is no free equivalent (unlike TLS certificates, which Let’s Encrypt made universal). Independent journalists, small publishers, and SMEs cannot access the trusted-credential tier on the same terms as large organisations. The standard structurally privileges well-resourced entities, which maps onto the same problem EC identifies with training-data pre-training bias: the infrastructure favours the already-advantaged.

Privacy exposure creates a structural tension with the journalism and whistleblowing contexts where provenance most matters. Content Credentials embed creator identity by default. Pseudonymous certificate options exist in the specification but are rarely implemented. For any creator whose safety depends on anonymity, C2PA as currently deployed is a liability, not a tool — which means the contexts where verified provenance would be most valuable are precisely the contexts where the standard is least adoptable.

The Ceiling for Brands

The trust gap is not only a C2PA problem. It describes the ceiling of what any provenance infrastructure can achieve — and that ceiling applies directly to brand entity signals.

Structured data tells AI systems *that* your brand has a consistent name, description, and category across verified sources. Wikidata entries and schema markup confirm *that* certain facts about your brand have been recorded in recognised authority graphs. Earned media placement tells AI *that* independent editors have decided to cover your brand in credible outlets.

None of these signals confirm whether the underlying claims are true. They confirm that the claims have been made, through verified channels, by identifiable sources. An AI system processing these signals can build a high-confidence representation of your brand — and that representation can be confidently wrong. The confidence is derived from provenance; the accuracy is derived from something else entirely.

This is what the Five Eyes document names explicitly and what most brand-entity optimisation frameworks leave unstated. Consistent entity signals, strong earned media coverage, and structured provenance data raise the floor — they establish that your brand exists, has been corroborated by independent sources, and has a verifiable history in public information spaces. They do not raise the ceiling. The ceiling is determined by the quality of the underlying claims.

What Raises the Ceiling

The only inputs that add truth-value above the provenance floor are substance-level: original research, first-hand documented experience, and human editorial judgement about what is actually accurate.

A proprietary study your company conducted and published is primary evidence. Its provenance is verifiable (you produced it), but more importantly, its content is examinable — a journalist or analyst who covers it can assess whether the methodology is sound, whether the conclusions follow from the data, whether the findings hold up. That is truth-value, not just provenance.

A first-hand case study with specific, documented outcomes contains claims that are, in principle, verifiable by anyone who cares to check. That is different from a brand description that has been signed, structured, and distributed across every authority source in the AI’s training corpus.

Human editorial judgement — the decision by a credible journalist or analyst to include a brand in a piece, having applied their own standards of accuracy — is the closest thing to a truth-signal that exists in the AI citation ecosystem. This is why earned media is not just a provenance play. An editorial act is not just a provenance signal; it is an accuracy signal, because the editor’s decision involves a content judgement, not just a source judgement.

The Four-Faceted framework the Five Eyes advisory applies to C2PA — Provenance, Education, Policy, Detection — names Detection as “always a cat and mouse game as technology evolves.” The same logic applies at the brand level. Provenance infrastructure (entity signals, structured data, earned coverage) sets the floor and maintains it. But it is the substance — the claims that are actually true and can be shown to be true — that determines what the AI can correctly say about you.

The Liar’s Dividend

One further implication, borrowed from the Five Eyes framing: as synthetic content proliferates, bad actors can weaponise the erosion of trust itself. They do not have to prove that your content is fabricated — they only have to make the claim plausibly enough to sow doubt.

Content Credentials are partly a defence against this. A brand that has consistently signed its content with verifiable provenance credentials makes it harder to credibly claim that content is fabricated. A brand that has built strong, consistent entity signals across authority sources makes false AI representations of it harder to sustain alongside the verifiable record.

Interpol projects that up to 90% of online content will be at least partly synthetic by 2027. In that environment, a brand’s verifiable provenance record — not just what AI knows about it, but what can be shown to have been consistently and accurately stated over time — becomes a resilience asset. The Liar’s Dividend gets harder to collect when the target has an unambiguous provenance footprint.

What This Means in Practice

The practical conclusions are layered.

Provenance infrastructure matters and should be built: consistent entity signals, verified structured data, Wikipedia presence, earned media in the outlets AI systems trust. These are the floor. Without them, AI systems have nothing reliable to draw on.

But the floor is not the ceiling. Brands that invest only in provenance signals — claiming consistent, well-distributed entity data without investing in the substance underneath — are building a well-signed container with unremarkable contents. The AI will represent them consistently. It will not represent them usefully.

The ceiling is raised by original research and documented expertise — the things that can be examined and verified independently of the channel through which they arrived. That is what human editorial standards have always been designed to surface. And it is what C2PA, structured data, and entity optimisation, for all their value, are structurally unable to substitute for.

The Next Article in This Series

If provenance infrastructure can certify who published a claim but not whether it is true, the next question is whether AI providers can at least trace a confident claim back to its source. Article 6 shows they cannot: there is no chain of custody from an AI’s output back to the training data that produced it.

Forward to next article

*Sources: NSA/ASD/CCCS/NCSC-UK Five Eyes Content Credentials Advisory (January 2025); TrueScreen, “C2PA: History, Limitations, and What’s Missing” (2026), citing RAND Corporation, Hacker Factor, and Fortune; Content Authenticity Initiative/C2PA (2026); EU AI Act Article 50.*

Series: Confident and Wrong — Article 4 of 7

Intro:

In July 2025, 76% of Google AI Overview citations came from pages ranking in the top 10 of organic search results. By January 2026 — six months later — that figure had fallen to 38%. The most widely relied-upon signal for AI citation predictability collapsed by half in six months, and the mechanism that caused it is not going to reverse. The most counterintuitive consequence: YouTube, not any editorial publisher, is now the single most-cited domain in AI Overviews.

Confident and Wrong — Article 4 of 7.

To read the whole series click here

The 76% to 38% Collapse Is the Story

The Ahrefs Brand Radar analysis is the most comprehensive dataset available on Google AI Overview citation behaviour. It covers 863,000 keyword SERPs and 4 million AI Overview URLs, with January 2026 data drawn from AI Overviews powered by Gemini 3. The headline finding is not subtle.

In July 2025, approximately 76% of AI Overview citations came from pages ranking in the top 10 of organic search. By January 2026, that figure was approximately 38%. A 38-percentage-point collapse in six months.

The remaining citations split nearly evenly. In the January 2026 data: 31.2% of AI Overview citations came from pages ranking positions 11–100, and 31.0% came from pages beyond the top 100 entirely — not ranked in the visible search results at all for the query in question. When you isolate organic blue links and exclude SERP features (news boxes, featured snippets, shopping units), the distribution is: 37.1% top-10, 26.2% positions 11–100, and 36.7% outside the top 100.

The practical translation: the AI Overviews index is now sourcing roughly the same volume of citations from pages that dominate Google Search as it is from pages that do not appear in Google Search at all for the same query. The top-10 is no longer an enclosure. It is one of three comparably weighted pools.

The Prior Assumption Was Built on the 76% Figure

Article 2 in this series described the structural overconfidence built into AI systems. Article 3 documented citation failure rates. This article addresses a third failure: the failure of the strategic defence.

The implicit logic of most AI visibility management goes: rank highly in Google, and AI Overviews will cite you. The logic was never watertight, but it was defensible. When 76% of citations come from the top 10, a high organic rank is a strong predictive signal. Not perfect — but strong enough to be worth managing.

When that figure is 38%, the logic is not defensible. A coin flip has a 50% success rate. Top-10 organic ranking as a predictor of AI Overview citation is now performing below that threshold.

As established in our earlier article on ChatGPT’s search dependency, 88% of ChatGPT citations come from the search channel — pages indexed and accessible via traditional web search. That finding remains valid and distinct. Being indexed is still the prerequisite for citation eligibility. What the Ahrefs Brand Radar data demonstrates is something more specific: that holding the top-10 position, once highly predictive of AI Overview citation specifically, is now less than half as predictive as it was six months ago.

SEO rank is the entry ticket. It puts your content into the pool of pages the system can reach. What has changed is the mechanism that determines which pages in that pool get selected.

Query Fan-Out Is the Structural Explanation

Google AI Overviews do not work by looking up a user’s query and citing the pages that rank for it. They generate a set of related sub-queries — fan-out queries — and retrieve pages from those expanded searches. The AI is assembling an answer to a complex question by breaking it into components and sourcing each component from wherever that component ranks best.

This means: a page that ranks top-10 for the user’s stated query may not appear in the citations, because the AI is answering sub-questions that the page doesn’t rank for. A page that ranks nowhere for the original query may be cited, because it ranks for a fan-out sub-query.

Query fan-out is not a bug or a temporary experimental behaviour. It is the architectural mechanism by which AI Overviews answer questions comprehensively. The system is getting more sophisticated at fan-out decomposition, not less. And as fan-out becomes more sophisticated, the pool of potentially citable pages widens further — because the system is effectively running dozens of implicit sub-queries for any given user question, each with its own independent ranking competition.

The practical consequence: a brand that ranks top-10 for “best accounting software for small businesses” is not necessarily cited in an AI Overview responding to that query, because the AI may be generating sub-queries like “accounting software with the most integrations,” “accounting software security certifications,” “small business accounting software comparison 2026,” and “accounting software migration from spreadsheets.” If the brand’s content does not rank in those adjacent searches, it may not appear in the cited pool at all — regardless of its position for the primary query.

YouTube Is the Signal Most Brands Have Missed

The January 2026 data contains a finding that is largely absent from how brands think about AI visibility: YouTube now accounts for 5.6% of all AI Overview citations.

Of citations from pages outside the top 100 in organic search, 18.2% are YouTube URLs. YouTube is the most-cited single domain in AI Overviews — ahead of any editorial publisher, ahead of any knowledge base, ahead of any news outlet. YouTube citations grew 34% in the six months covered by this analysis.

The mechanism is fan-out. Many of the sub-queries generated from complex user questions are informational: how does X work, what does Y look like, how do I do Z. These sub-queries frequently resolve to video content. A brand that has invested heavily in written content and no investment in video has built a strategy that is structurally blind to the fastest-growing citation source in Google AI Overviews.

This is not a prediction about the future of video. It is a current data point. YouTube’s 18.2% share of non-ranking citations is happening now. And it is happening through the same fan-out mechanism that is pulling citations away from top-10 ranked pages generally. The brands that appear in AI Overviews for informational sub-queries are, increasingly, the brands with video content that answers those sub-questions — not just written content that ranks for the primary query.

The Collapse Is Structural, Not Cyclical

The 76% to 38% movement could be interpreted as a calibration phase — early AI Overviews leaned heavily on top-ranked pages while the system was new, and over time it has expanded its sourcing. Under that interpretation, the decline levels off and stabilises.

That interpretation is wrong about the mechanism. The decline is not a calibration artefact. It is a direct consequence of fan-out becoming more sophisticated. The more complex the sub-query decomposition, the more diverse the potential citation pool — because different sub-queries will have different page-level winners, and those winners will be scattered across the ranking distribution rather than concentrated at the top of any single query’s results.

As AI Overviews answer increasingly complex questions — and the queries being routed to AI Overviews are getting more complex, not simpler — the fan-out decomposition expands. A richer fan-out means a more dispersed citation pool. The trend away from top-10 concentration is the predictable output of a more capable system, not a temporary anomaly in a less capable one.

This is why the 38% figure is not the floor. It is the current reading on a declining trend with no structural reversal in sight.

What Brands That Rank Well Are Now Missing

The brands most exposed to this collapse are the ones that have been most successful at traditional SEO. They rank in the top 10. They have been told — and the 76% figure previously supported the claim — that their ranking protects their AI Overview visibility. They have not invested in what the 38% figure now demands.

What the new citation distribution rewards is topical breadth, not primary query rank. A brand that dominates the primary keyword but has thin or absent content on adjacent topics is not covered for the fan-out sub-queries that now drive 62% of AI Overview citations. A brand that publishes written content exclusively is not covered for the YouTube citations that drive 18.2% of non-ranking citations. A brand that tracks its AI visibility by tracking its organic rankings is measuring a signal that now explains only 38% of citation outcomes.

Three things separate the brands that are adapting from the brands that are relying on a weakened defence:

Adjacent topic coverage. Fan-out sub-queries pull from wherever the sub-topic ranks best. Brands that have built out content on secondary, supporting, and adjacent topics — not just their primary keyword targets — appear in more of those sub-query citation pools. Topical depth across a subject area outperforms narrow depth on a single query.

Video content in Google’s ecosystem. YouTube’s 18.2% share of non-ranking citations is not addressable through written content, regardless of how well-structured or how highly ranked. Informational sub-queries that resolve to video require video content. A written guide to the same topic will not substitute for a video answer to the fan-out sub-question.

Direct AI citation tracking. Inferring AI Overview citation from organic ranking is now less than a coin-flip predictor. Brands that are tracking their actual AI Overview presence — directly, not inferred from rank — have the signal they need to manage a 38% world. Brands that are tracking rank and inferring AI visibility from that are managing a world that existed six months ago.

The Implication for This Series

This series began with the observation that AI is trained to express certainty regardless of accuracy, and that citation failure is a structural output of that training. Articles 1 through 3 addressed the failure mode: wrong, confident, at scale.

This article addresses the failure of the defence: the assumption that ranking well in Google provides meaningful protection against AI misrepresentation or AI invisibility. It did provide meaningful protection — when 76% of citations came from the top 10. That protection has halved. The direction of travel is further weakening.

The next article in this series addresses provenance: why being cited is not the same as being cited accurately, and why the citation history AI inherits may reflect what was true, not what is true.

Forward to next article

*Sources: Ahrefs Brand Radar, AI Overview citation analysis, 863,000 SERPs / 4M URLs, January 2026 (AI Overviews powered by Gemini 3); Ahrefs AI Overview research, July 2025 baseline data; our earlier article on ChatGPT’s search dependency (88% search-channel dependency finding).*

Series: Confident and Wrong — Article 3 of 7

Intro:

In the first seven weeks of 2026, one in every 277 academic papers published contained a fabricated citation — a reference to a source that does not exist, generated by an AI system and presented as fact.

Two years earlier, the rate was one in 2,828.

That is not an improvement story. It is a sixfold increase. And it is accelerating.

Confident and Wrong — Article 3 of 7.

To read the whole series from the start click here

The Lancet Data: Scale and Trend

The finding comes from a peer-reviewed study published in The Lancet in May 2026, led by Maxim Topaz at Columbia University. The research team analysed more than 2 million papers containing 97 million citations. What they found was systematic, not incidental.

The numbers in sequence:

– 2023: 1 in 2,828 papers contained fabricated citations

– 2025: 1 in 458 papers

– Early 2026 (first seven weeks of the year): 1 in 277 papers

Across the full dataset: approximately 4,000 fabricated citations identified, spread across roughly 2,800 papers.

The trend line is steeper than the current rate implies. From 2023 to 2025 alone, the rate increased sixfold. The early 2026 figure continues that trajectory — it does not level off, and there is no mechanism in the data that would cause it to.

Misha Teplitskiy at the University of Michigan, commenting on the study, was direct: “This is one of the first papers telling us something about the quality of what’s being produced with LLMs, and it’s a signal of slop.”

The distribution of fabricated citations is not uniform. More than one-third originate from two large open-access publishers. Science journals with strong editorial oversight report no published fabricated citations. Publishers with weaker oversight — PLOS among them — acknowledge “numerous” unverifiable references. The correlation is clear: fabrication risk is inversely proportional to editorial rigour. Where the system is designed to catch errors, fewer fabrications survive. Where it is not, they accumulate.

Why This Rate Is a Floor, Not a Ceiling

The Lancet study measures academic publishing. That context matters for understanding what the numbers actually represent.

Academic papers are the most adversarial environment for AI-generated fabrication. Authors have professional and reputational incentives to check their own references. Editorial systems at the better journals are specifically designed to identify errors. Peer reviewers are domain experts who may recognise a fabricated citation by name. And the stakes — careers, institutional credibility, funding — are high enough that the detection pressure is real.

None of those conditions apply to commercial brand representation.

When an AI system describes your company’s founding date, your service offering, your partnership history, or your executive team, there is no editorial system checking the claim. There is no peer review. The person receiving the information is unlikely to verify it against primary sources. And the model generating it is the same model generating academic citations — it has no separate “commercial accuracy” layer, no distinct mechanism for verifying facts about brands that it would not also apply to academic literature.

The academic fabrication rate is therefore a lower-bound estimate for commercial entity hallucination — not an upper bound. The conditions that cause fabrication exist across all AI output. The conditions that catch fabrication are largely absent outside of high-oversight publishing contexts.

If one in 277 papers contains a fabricated citation in a context designed to catch fabrication, the implied rate in contexts that are not designed to catch fabrication is not lower. The data does not support optimism in that direction.

The Corroborating Evidence From a Different Domain

The Lancet study is not alone. It confirms a pattern that the CJR/Tow Center study at Columbia Journalism Review identified from a different angle in 2025.

That study tested eight generative AI tools against 1,600 prompts drawn from news publishers — evaluating whether AI correctly identified articles, publishers, and URLs. The headline finding: more than 60% of responses across all eight tools contained incorrect information. Error rates ranged from 37% at best to 94% at worst.

Two studies, different domains, different methodologies, different research teams, different publication years. Both arrive at the same structural conclusion: AI citation and reference failure is not an edge case, and it is not confined to low-quality tools. It is a systemic property of how current AI systems generate text.

The Deloitte finding adds a third dimension. In 2024, 38% of business executives reported making incorrect business decisions based on hallucinated AI outputs. That number does not come from a lab — it comes from operational deployment, at scale, by sophisticated organisations that were presumably paying attention. One in three executives making a wrong call because of fabricated AI information is not a user-education problem. It is a system-level failure rate.

The Mechanism That Makes It Worse

The acceleration from 2023 to 2026 is not explained by a single cause. But the primary driver is structural, and it is self-reinforcing.

AI systems are trained on text corpora assembled from the open web, academic publishing databases, and other large-scale text sources. As AI-generated content proliferates — in academic papers, on websites, in news articles — it is absorbed into the corpora that future models train on. Fabricated citations in published papers become part of the training data. The model that trained on clean data in 2022 was not exposed to AI-generated academic slop. The model training in 2026 is.

This is not a theoretical risk. The Lancet data shows the rate accelerating at the pace you would expect if fabricated content were being absorbed as legitimate training signal. Each generation of models learns, in part, from the outputs of previous models. Each generation produces a slightly higher baseline fabrication rate. The loop compounds.

The academic publishing world is beginning to respond — stronger editorial checks, citation verification tooling, mandatory AI disclosure requirements. But those responses operate on the current margin, not the accumulated training signal. The models already trained on pre-2026 academic literature carry whatever was in that corpus. They cannot be retroactively cleaned.

Article 2 in this series established that AI is trained to sound certain. This article establishes the rate at which that confident output is factually wrong. The combination is the problem. Fabrication alone would be detectable — a system that generated obvious nonsense with visible hesitation would be identifiable as unreliable. Fabrication delivered with the linguistic confidence of an accurate statement is something different. It passes human scrutiny at the first reading. It propagates.

What This Means for How AI Represents Your Brand

AI systems that fabricate citations in peer-reviewed science are the same systems writing about your products, your executives, your founding date, and your competitive positioning. The failure mode is not domain-specific.

The underlying cause — confident text generation without evidential grounding — is a property of how these systems work, not of the subject matter they are applied to. A model that invents a citation because it does not have the source but needs to produce fluent, confident output will apply exactly the same process when it does not have accurate information about your brand but is asked to describe it. The mechanism is identical.

This means that “AI is often wrong about things” is not the frame. The frame is: AI fabrication is a measurable, accelerating, systemic problem with an identified rate, a documented trend, and a structural mechanism that is making it worse. The one-in-277 figure is not a worst-case scenario. In the context of academic publishing — with all its oversight and incentives — it is the measured current rate.

For brands, the question this data raises is not “could AI get our information wrong?” The data has answered that. The question is: what does the AI have to draw on when it is generating text about us? A model working from rich, consistent, independently corroborated entity data will still sometimes fabricate. But it is less likely to fabricate about the things it has strong evidence for — because the fabrication mechanism kicks in when evidence is absent or thin.

The structural response to a system that confidently generates wrong information is not to hope the system improves. The Lancet trend runs directly against that hope. The structural response is to ensure that the system has enough accurate evidence about your entity that there is less gap for fabrication to fill.

The Direction of Travel Is Not Reassuring

The “AI is improving” response to fabrication data is persistent, but the Lancet study directly contradicts it on the specific question that matters here.

Overall AI capability is increasing. Benchmarks are improving. Some hallucination rates on standardised evaluations have declined for some models in some contexts. None of that reverses the finding that fabricated citations in academic publishing increased sixfold from 2023 to 2025 and continued accelerating into 2026.

Capability and accuracy are not the same thing. The Premium Model Paradox — described in Article 1 of this series — established that higher-tier models can produce more confidently wrong outputs, not fewer. The Lancet data confirms the trajectory at the publishing level: as AI use grows, as AI-generated text enters training corpora, as the feedback loop compounds, the fabrication rate in the corpus goes up, not down.

The question for brands is whether they want to wait for the trend to reverse — with no evidence it will — or whether they want to do something about their own entity evidence now, while the gap between well-evidenced and poorly-evidenced brands is still a gap rather than a cliff.

The rate is one in 277. It was one in 2,828 two years ago. The direction is unambiguous.

The Next Article in This Series

If fabrication is accelerating and confidence is not a reliable signal of accuracy, the next question is whether the traditional defence — ranking well in Google — still offers any protection. Article 4 shows that it does not, and quantifies exactly how much ground has been lost.

Forward to next article

*Sources: Topaz et al., “Fabricated Citations in Scientific Literature,” The Lancet (May 2026); CJR / Tow Center for Digital Journalism, “We Compared Eight AI Search Engines. They’re All Bad at Citing News.” (2025); Deloitte, AI in the Enterprise (2024).*

Intro:

AI overconfidence is not a bug in how models think. It is a feature of how they were trained. The confidence you hear in an AI answer is not evidence that the answer is accurate. It is evidence that the model learned to produce confident-sounding outputs because human raters reward them.

Confident and Wrong — Article 2 of 7.

To read the whole series from the start click here

The Problem Is Not Lying — It Is That AI Was Never Trained to Know the Difference

The dominant mental model for AI error is hallucination: the model invents something that does not exist. That frame is accurate as far as it goes, but it misses the deeper structural problem. Even when an AI produces accurate content, the confidence it expresses in that content is not calibrated to its accuracy. The two are independent. A model can be right and overstate its certainty. It can be wrong and do the same. The confidence is not a signal. It is a style.

Ghafouri et al. (arXiv:2411.06528, November 2024) formalised this as epistemic miscalibration: “a model’s linguistic assertiveness fails to reflect its true internal certainty.” The word “fails” is doing work here. This is not a random fluctuation around the correct confidence level. The direction is consistent — models systematically err toward overconfidence rather than appropriate hedging. The authors describe it as “an urgent risk of overstated certainty which may mislead users on a massive scale.”

The CJR/Tow Center empirical study (2025), which tested eight generative AI search tools against 1,600 prompts, confirmed this pattern at the output layer: chatbots “presented inaccurate answers with alarming confidence, rarely using qualifying phrases.” More than 60% of responses contained incorrect information. The confidence was constant. The accuracy was not.

This is the premise Article 1 in this series established empirically — the premium model paradox. This article explains why it happens.

Layer One: The Training Incentive

Large language models are trained using Reinforcement Learning from Human Feedback (RLHF). Human raters evaluate model outputs and score them. Those scores become the training signal: outputs rated highly are reinforced; outputs rated poorly are suppressed.

The problem is in what human raters consistently score more highly. Confident, fluent, authoritative-sounding responses outperform hedged, qualified, appropriately uncertain ones. A response that says “Based on available information, it appears that…” scores lower than one that simply states the answer with conviction. Not because the confident version is more accurate — but because it reads better. It sounds like it knows.

This creates a systematic training bias. The objective the model is optimised for is not accuracy. It is the appearance of quality to a human evaluator. Fluency and confidence are the visible proxies for quality in the training loop. Accuracy is not directly observed or rewarded — it would require evaluators to independently verify every factual claim, at scale, which is not how RLHF pipelines operate in practice.

The consequence: models emerge from training overconfident not because they are designed to deceive, but because confidence was the signal that survived selection. Every generation of training that optimised for human preference also optimised, as a side effect, for confident-sounding outputs. The model learned that expressing certainty is what gets rewarded. It learned this correctly.

Layer Two: Ownership Bias

Base miscalibration from training incentives is the first layer. A June 2026 paper (arXiv:2606.03437) identifies a second, compounding mechanism that makes things worse: ownership bias.

The finding: when models respond in their own voice — the standard conversational chat template — they assign up to 26% higher confidence to their outputs than to word-for-word identical content attributed to a user.

The experimental design makes this result precise. Take a model response. Ask the model to assess confidence in it, framed as its own output. Then ask the model to assess confidence in the same text, framed as something the user wrote. The model scores the same content more confidently when it believes it produced it. The content is identical. The attributed source changes. The confidence score moves by up to 26 percentage points.

The mechanism is the chat template. When a model generates a response through the standard instruction-tuning and chat formatting process, it creates an ownership attribution — “this is what I said” — that compounds the already-present base overconfidence. Post-training processes (RLHF and instruction fine-tuning) reinforce outputs that human raters find authoritative, and the chat template then layered additional certainty on top by framing everything as first-person assertion.

This is tested across 6 open-weight LLMs, 3 benchmarks, and 3 different confidence elicitation approaches. The ownership bias is not a quirk of one model or one measurement method. It is a structural feature of how instruction-tuned models operate.

The Fix Exists. It Is Not Deployed.

The same research that identifies ownership bias also demonstrates how to correct it. The solution is inference-time prompt engineering: reframe the model’s generated output as user-provided content during the confidence assessment step. Instead of “how confident are you in what you just said?”, the system asks “how confident are you in what the user has said?” — using the same text.

This approach reduces overconfidence by up to 26%. It requires no model retraining. It can be applied to any existing deployment as a prompt-engineering intervention.

It is not applied in any standard commercial deployment.

Every default LLM interaction — every AI Overview, every ChatGPT response, every Perplexity summary — carries the full ownership bias uncorrected. The models that serve hundreds of millions of queries daily are expressing inflated certainty about every output they produce. The fix is known. The fix is cheap. The fix is not implemented.

This matters for the question of whether overconfidence is a temporary flaw that better models will solve. The ownership bias fix does not require a better model. It requires a different system design. The overconfidence is therefore not primarily a function of model capability — it is a function of how AI systems are deployed. It will persist as long as standard deployments do not apply the correction, regardless of how capable the underlying model becomes.

Ownership Bias Is Not Hallucination

These two failure modes are frequently conflated. They are independent.

Hallucination is a content failure: the model produces something that is false or fabricated. Epistemic miscalibration — including ownership bias — is a confidence failure: the model expresses more certainty than accuracy warrants. A model can produce entirely accurate text and still be miscalibrated. It can hallucinate and be appropriately hedged. In practice they often appear together — miscalibrated hallucination, wrong and expressed with false certainty — but the combination should not be confused with either element alone.

For brands, this distinction is consequential. The AI statement about your brand may be accurate. The confidence expressed in that statement is still inflated by ownership bias on top of base miscalibration. The accuracy of what the model says does not mean the confidence was earned. The confidence is a structural product of training objectives and conversational framing. It is not proportional to evidential quality.

Conversely, if the AI statement about your brand is inaccurate, the miscalibration problem compounds the harm: users receive no linguistic signal that the confident-sounding statement is unreliable. Tone is a training artefact, not an epistemic indicator.

Why More Capable Models Are More Convincingly Wrong

This is the mechanism behind the Premium Model Paradox documented in Article 1.

A more capable language model is, in the most direct sense, a model that is better at producing fluent, confident-sounding text. That is what capability looks like at the output layer. A more capable model can sustain coherent argument across longer passages, avoid obvious contradictions, produce text that reads as authoritative. All of these properties are what “fluency” means — and fluency is the proxy that training incentives reward.

If the training signal optimises for confidence, and a more capable model is better at producing confident outputs, then a more capable model is better at producing confident outputs regardless of whether they are accurate. Capability and calibration are trained on different objectives. Improving capability does not improve calibration.

OpenAI’s system card for its o3 and o4-mini models confirmed this from the inside: hallucination rates worsened as measured capability improved. The premium model paradox is not an anomaly in the CJR/Tow Center data. It is a predictable output of training dynamics that nobody publicly advertises.

A premium model represents a higher-quality version of confident-sounding text. It does not represent a higher-quality relationship between expressed confidence and actual accuracy.

What Two Layers of Overconfidence Mean for Brand Representation

Every confident AI statement about a brand is subject to both mechanisms simultaneously.

The base layer: the training incentive that selected for confident-sounding outputs, producing systematic miscalibration in the direction of overconfidence. This applies to every factual claim a model makes, including brand claims.

The ownership layer: the additional 26% confidence inflation that arises from the chat template ownership attribution. When the model says something about your brand in its own voice — which is all standard conversational AI interactions — it expresses that claim with inflated certainty built on top of the already-inflated base.

The result is a confident statement about your brand that is the product of two compounding structural biases, neither of which has anything to do with the accuracy of the underlying information. The model is not expressing what it knows with appropriate certainty. It is expressing what it knows — or thinks it knows, or has pattern-matched from training data — through a structural lens that inflates certainty at two levels, by design, without correction.

A confident AI statement about your brand is not evidence of accurate representation. It is evidence that the model produced fluent text.

This is why provenance infrastructure matters. A brand with strong, consistent, independently corroborated entity signals does not fix the model’s calibration — that is a training and deployment problem that no brand can address from the outside. But it changes the content that the model’s confidence is attached to. When the underlying evidence is richer and more consistent, the confident-sounding output is more likely to be expressing something accurate. The confidence mechanism stays broken. What it is expressing improves.

That is not a complete solution. It is the only solution that is currently available.

The Next Article in This Series

If two layers of structural overconfidence produce inflated certainty, and that inflated certainty is expressed in confident statements at scale, the next question is: how often do those statements turn out to be wrong?

Article 3 answers that with data from peer-reviewed research: one fabricated citation in every 277. And the rate is accelerating.

Forward to next article

*Sources: Ghafouri, Yao et al., “Epistemic Integrity in Large Language Models,” arXiv:2411.06528 (November 2024); arXiv:2606.03437, “Large Language Models Are Overconfident in Their Own Responses” (June 2026); CJR / Tow Center for Digital Journalism, “We Compared Eight AI Search Engines. They’re All Bad at Citing News.” (2025); OpenAI o3/o4-mini system card.*

Series: Confident and Wrong — Article 1 of 7

Intro:

The intuition seems sound: pay for a better AI tool, get better outputs. More capable model, more accurate results. It is the same logic that applies to most software — higher tier, higher quality. But the data says otherwise.

*Confident and Wrong — Article 1 of 7.

     Forward to next article

What the Study Found

The Tow Center for Digital Journalism (Columbia Journalism Review, 2025) tested eight generative AI search tools against 1,600 prompts — 20 publishers, 10 articles, 8 chatbots. For each response, researchers evaluated whether the AI correctly identified the article, the publisher, and the URL.

The headline finding: over 60% of responses across all eight tools contained incorrect information. Error rates ranged from 37% at best (Perplexity) to 94% at worst (Grok 3). Three-quarters of Grok 3’s citations led to error pages. DeepSeek misattributed sources in more than half its responses.

These are not edge cases. They are the median experience.

But buried inside the dataset is a finding that deserves more attention than the headline numbers: the premium model paradox. When the researchers compared model tiers within the same platforms, the higher-priced, more capable versions showed *higher* error rates than their free counterparts. Spending more produced less accurate results.

This is not a Grok-specific problem. It appeared across the platforms tested.

Why More Capable Means More Confidently Wrong

The mechanism is not mysterious once you understand what “more capable” actually means.

Language model training is optimised for human preference. Outputs that human raters score as helpful, fluent, and confident are reinforced. Outputs that hedge, express uncertainty, or refuse to answer are rated lower. The training objective is not accuracy — it is the appearance of quality to a human evaluator.

Ghafouri et al. (arXiv:2411.06528, 2024) formalised this as epistemic miscalibration: a model’s linguistic assertiveness does not reliably reflect its actual accuracy. The direction of miscalibration is consistent — models systematically err toward overconfidence rather than randomly fluctuating around the correct confidence level. The researchers described this as “an urgent risk of overstated certainty which may mislead users on a massive scale.”

A more capable model is, in large part, a model that is better at producing fluent, confident-sounding text. That is what capability looks like in the output layer. It is not a model that is better at knowing what it does not know.

OpenAI’s own system card for its o3 and o4-mini models confirms the same pattern from a different angle: hallucination rates worsened even as measured model capability improved. The models got better. They also got more wrong.

The CJR/Tow Center study observed the same phenomenon empirically: chatbots “presented inaccurate answers with alarming confidence, rarely using qualifying phrases.” Premium tiers produced more fluent versions of the same confident incorrectness.

What This Means for Brands

The practical implication is significant: AI tier is not a reliable proxy for AI accuracy when it comes to brand representation.

A brand that assumes enterprise-tier tools will represent them more faithfully than free versions is making an unsupported assumption. The CJR/Tow Center data suggests the premium tier may in fact produce more convincingly wrong outputs — the same errors, expressed with greater fluency and fewer hedges.

This is not an argument against investing in AI tools. It is an argument against treating model capability as a substitute for provenance infrastructure. The brands that AI systems represent accurately are not the brands that use the most expensive tools. They are the brands that have given AI systems the most and best evidence to draw on.

Epistemic miscalibration is a model-level property. It cannot be fixed from the outside by purchasing a higher subscription. But its effects on brand representation can be partially addressed — not by changing the model, but by changing what the model has to work with.

When a brand has strong provenance infrastructure — consistent entity signals, independent editorial coverage, verifiable primary sources — the model’s confident outputs are more likely to be grounded in adequate evidence. The confidence is still a product of training objectives, not genuine calibration. But the content that fluency is built on is better. A model expressing certainty about a brand with rich provenance is more likely to be expressing certainty about something accurate than a model expressing the same certainty about a brand with none.

The Upside-Down Assumption

The premium model paradox should recalibrate expectations in both directions.

Brands should not panic because AI tools are generating inaccurate outputs at scale — the error rates in the CJR/Tow Center study are high enough that panic would be proportionate, but not productive. The problem is structural, not accidental, and the solution is correspondingly structural.

And brands should not assume that the enterprise AI subscription their competitors are running means those competitors have solved the accuracy problem. The evidence suggests the opposite.

What grounds AI accuracy is not the model tier. It is the quality of the underlying evidence. That is a provenance problem, not a procurement problem — and it is one that brands can actually do something about.

The Next Article in This Series

If premium models produce more confidently wrong outputs rather than fewer, the obvious question is why. The next article examines the mechanism behind this: why AI is structurally trained to sound certain, and why more capable models make the problem worse, not better.

Forward to next article

*Sources: CJR / Tow Center for Digital Journalism, “We Compared Eight AI Search Engines. They’re All Bad at Citing News.” (2025); Ghafouri, Yao et al., “Epistemic Integrity in Large Language Models,” arXiv:2411.06528 (Nov 2024); OpenAI o3/o4-mini system card.*