Is your website locking AI out?

Most conversations about AI visibility assume the problem is that you don’t have enough of the right content. But sometimes the reason is that your website is quietly turning away the very software that needs to read it. Your business isn’t invisible for want of something to say. But perhaps it has been set up in a way that has locked the AI visibility entrance.

This is a better problem to have, because unlike most of AI visibility, it is fixable in an afternoon.

What we found

In a dry run of our AI Visibility Check across 45 businesses in one local-service sector, 7 of the 45 websites (about one in six) could not be read by the check at all. [1] Instead of any result, the owner would just see a message saying we couldn’t complete the assessment.

Two of those seven were down to out-of-date web addresses in our own records — a real business with a working site we simply had the wrong link for. Set those aside, and roughly one in nine legitimate businesses had a website our check genuinely could not read. [2] The reasons:

  • Most were the site actively refusing the automated request — a “403 Forbidden,” which is a website’s way of saying “no robots allowed.” Three of the five. [3]
  • One had a security-certificate problem that stopped the page loading cleanly. [3]
  • One was built entirely in JavaScript with no plain text for an automated reader to pick up. [3]

Different causes, same outcome: software arrives at the page and comes away with nothing.

Why a locked door makes you invisible

When an AI assistant answers a question like “best plumber in [locality]” or “who should I use for X near me,” it relies on what it can actually read from the live web. If your site returns “no robots allowed” to an automated visitor, the assistant can’t read your own words about who you are, what you do, or where you are — so it has nothing of yours to work with, and reaches for competitors it can read instead. You can have the best website in your market and still be absent from the answer, because the answer is written from competitor pages the machine was allowed to open. [4]

It is worth being precise here: our check is not the same software as any given AI engine’s, and a site that blocks one automated visitor doesn’t necessarily block every one. But a blanket “no robots” rule is blunt — it tends to turn away the legitimate readers along with the nuisance ones. If your site is blocking automated traffic wholesale, the crawlers behind AI answers are very likely on the wrong side of that wall too. [4]

How to find out, and what to do

You don’t need to fix any of this yourself. There are three questions to put to whoever looks after your website — an agency, a developer, or the person who set it up:

  • “Does our site block bots or automated traffic?” Many sites switch this on — through a security plugin, a firewall setting, or an aggressive bot-blocker — without anyone deciding the AI assistants should be shut out too. It is often a setting, not a rebuild.
  • “Is our security certificate valid and not mismatched?” A certificate warning doesn’t just unnerve human visitors; it can stop an automated reader loading the page at all.
  • “Is our content readable without running scripts?” A site built entirely in JavaScript with little plain text can look perfect to a person yet be blank to a reader that doesn’t execute code.

Then re-check. Once the door is open, the same content that was invisible can start being read — and used.

The reason this matters more than it first appears: a business with thin content has a long job ahead of it. A business whose website is simply shutting AI out has a short one. It is not that AI has judged you and found you wanting. It is that AI never got to see you at all — and that is among the most fixable problems in this whole field.

—

These figures come from EC’s own AI Visibility research: a single grounded AI engine, a small sample, and a live snapshot of the web that changes over time. They illustrate a pattern, not a fixed market rate. Method: entityconfidence.ai/research-methodology/ · Corrections and right of reply: entityconfidence.ai/corrections-and-right-of-reply/

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *