There is a text file on your website, one you have probably never opened, that may be telling ChatGPT, Claude, and Perplexity to go away. It is called robots.txt, it was set up by whoever built your site, and on a surprising share of Canadian business websites it quietly blocks the exact crawlers that would otherwise read your pages and cite your company in AI answers. Nobody decided this. A default setting, a security checkbox, or a 2023-era "block the AI scrapers" fad decided it for you. The check takes five minutes.
Quick answer: robots.txt is the file that tells web crawlers which parts of your site they may read, and it controls whether AI engines can see you at all. To check yours, load yourdomain.com/robots.txt and look for lines mentioning GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended under "User-agent" followed by "Disallow: /". If AI crawlers are blocked, the engines cannot read your pages, and they will recommend competitors instead. For most businesses the right move is to allow them.
What is robots.txt and why does it control your AI visibility?
robots.txt is a plain-text file sitting at the root of your website that declares, crawler by crawler, what may be read. It is the oldest etiquette system on the web: reputable crawlers request the file first and honour what it says. One line can welcome a crawler everywhere; two lines can ban it from the entire site.
For twenty years this file mattered mostly for classic search. Now it is the front door for AI too. When ChatGPT browses to answer a buyer's question, when Perplexity assembles its cited answer, when Google gathers material for AI Overviews, each sends a named crawler to your site first. Blocked crawler, no content. No content, nothing to cite. Nothing to cite, and the engine builds its answer from whoever it can read, which is to say, your competitors. The cruel part is the silence: no error, no warning, no dip in any report you look at. Just answers, every day, assembled without you.
Which AI crawlers matter in 2026?
A handful of named user-agents cover the engines your buyers actually use. Worth knowing by name, because these are the names that appear, or should appear, in your robots.txt:
- GPTBot and OAI-SearchBot. OpenAI's crawlers: the first gathers training material, the second feeds ChatGPT's live search answers. Blocking the second is what erases you from ChatGPT's browsing-mode recommendations.
- ClaudeBot. Anthropic's crawler, feeding Claude, which a growing share of professionals use for research and vendor shortlists.
- PerplexityBot. Feeds Perplexity, the engine that shows its sources, and a disproportionate share of high-intent B2B research.
- Google-Extended. Google's control for AI model training. Note that AI Overviews visibility rides primarily on the regular Googlebot, so ordinary Google indexing still matters most there.
- Bingbot. The classic crawler that also feeds Microsoft Copilot's answers; almost nobody blocks it, but it belongs on the checklist.
The list evolves, engines add crawlers and rename them, which is one reason a robots.txt written in 2022 and never revisited is a liability.
How do you check your robots.txt in five minutes?
The whole diagnostic runs in a browser, no tools and no login:
- Load the file. Type yourdomain.com/robots.txt into a browser. It is public by design; you are allowed to look at anyone's, including every competitor's.
- Scan for the AI names. Search the page for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. For each, see whether the lines beneath say "Disallow: /" (blocked everywhere) or "Disallow:" with nothing after it, or "Allow: /" (welcome).
- Check the catch-all. A section starting "User-agent: *" applies to every crawler not named separately. If it says "Disallow: /", your site is closed to everything by default, which sometimes survives from a pre-launch staging setup.
- Compare with a competitor who wins AI answers. Load their file the same way. If they welcome the crawlers you block, you have found one concrete reason they get named and you do not.
While you are in diagnostic mode, run two or three buyer questions through the engines and see who gets named; that pairing of file check plus answer check is the core of the query panel method for measuring AI visibility.
Why do so many sites block AI crawlers by accident?
Because the blocking usually happened as a side effect, not a decision. The common paths: some website platforms and security services shipped defaults or one-click options that turned AI crawlers away wholesale, and busy site admins clicked them; a wave of 2023-era advice urged everyone to block "AI scrapers" to protect content, before it was clear that the same crawlers feed the answers buyers now rely on; a developer copied a robots.txt template from a forum without reading it; or a staging-site block that made sense before launch simply never got removed.
Each of these made local sense at the time. The world changed around them. In 2023, blocking AI crawlers read as protecting your content. In 2026, with buyers asking engines for supplier recommendations daily, the same lines read as declining to be considered. Here is the trade, side by side:
| Dimension | AI crawlers blocked | AI crawlers allowed |
|---|---|---|
| ChatGPT browsing answers | Your pages unreadable; competitors cited | Your guides and specs quotable in answers |
| Perplexity citations | Absent from the sources panel | Eligible to be a named, linked source |
| Claude and Copilot research | Invisible to professional shortlists | Readable when buyers ask for vendor options |
| What you protect | Content from reuse in training, in theory | Nothing withheld; expertise works as marketing |
| Who benefits | Competitors who stayed readable | You, compounding as engines re-encounter you |
Not sure what your robots.txt is telling the machines?
We'll check your crawler access, schema, and llms.txt, run the buyer queries in your category through the major AI engines, and show you exactly what is blocking your visibility, in plain English.
Book Free AuditShould you ever block AI crawlers?
Sometimes, and it is worth stating the honest case. A publisher whose product is the content itself, paywalled journalism, proprietary research, licensed data, has a real interest in controlling reuse, and blocking training crawlers can be rational there. Some companies also distinguish between crawler types: blocking training-focused crawlers like GPTBot or Google-Extended while allowing search-focused ones like OAI-SearchBot, keeping their content out of future model training but present in live answers.
But for a business that publishes content so buyers can find it, buyer guides, specs, case studies, service pages, blocking the crawlers defeats the purpose of the content. Your marketing pages are not the product; they are the pitch. A pitch delivered where buyers cannot hear it is not protected, it is wasted. For nearly every Canadian B2B and industrial company we look at, the right posture is open to all the major AI crawlers, with sensitive areas like customer portals excluded by path, not by banning engines site-wide.
What should an open, well-configured setup look like?
Three layers, all cheap, all mostly one-time. First, a robots.txt that explicitly allows the major AI crawlers, keeps ordinary search access intact, and disallows only genuinely private paths. Second, an llms.txt file, the plain-text summary of who you are, what you make, and where your key pages live, written for AI systems; our llms.txt explainer has a sample to adapt. Third, the readability layer: content in real HTML rather than trapped in scripts and PDFs, plus schema markup, the technical SEO basics that decide what a crawler finds once the door is open.
Then verify the outcome, not just the config: after opening access, give the engines a few weeks to re-crawl, and re-run your buyer queries. The door being open is necessary; the measurement proves it mattered. This whole stack, access, structure, and proof, is the foundation stage of our AEO services, and it is the least glamorous, highest-leverage work we do.
Frequently asked questions about AI crawlers and robots.txt
What is robots.txt in plain terms?
It is a public text file at the root of your website that tells web crawlers which parts of the site they may read. Reputable crawlers, including the ones behind ChatGPT, Claude, Perplexity, and Google, check it before reading anything and honour what it says. Two lines in this file can make your entire site invisible to an AI engine, which is why it deserves five minutes of attention.
How do I check if my website blocks AI crawlers?
Type yourdomain.com/robots.txt into a browser and search the page for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. A crawler followed by Disallow with a slash is blocked from the whole site. Also check the User-agent asterisk section, which covers every crawler not named individually; a site-wide Disallow there closes the door to everything.
Will allowing AI crawlers hurt my Google rankings?
No. AI crawler permissions and classic search indexing are separate controls; allowing GPTBot or PerplexityBot has no bearing on how Googlebot ranks your pages. The two concerns run on different lines in the same file. The practical risk runs the other way: an overly aggressive robots.txt written for one purpose can accidentally block crawlers you need, including search itself.
Is there any good reason to block AI crawlers?
For content businesses, sometimes: publishers, research firms, and data companies whose product is the content itself may rationally block training crawlers to control reuse. Some split the difference, blocking training-focused crawlers while allowing search-focused ones so they stay present in live answers. For businesses whose content exists to attract buyers, blocking defeats the content's whole purpose.
We blocked AI crawlers in 2023 on our developer's advice. Should we reverse it?
Almost certainly, unless your content is itself the product. The 2023 advice predates buyers using AI engines for supplier recommendations at today's scale; what protected content then suppresses demand now. Reverse the block, keep private paths excluded by path rather than site-wide, then re-run your buyer queries a few weeks later to confirm the engines can see you again.
How long after unblocking do AI engines start seeing our site?
Live-search engines typically re-crawl active sites within days to a few weeks, so answers that rely on browsing can reflect your pages fairly quickly. Signals that ride on accumulated data, like being a default recommendation without live search, move on the slower clock of months. Unblocking is instant; visibility follows on the crawl and trust timelines.
Does llms.txt replace robots.txt?
No, they do opposite jobs. robots.txt controls access: which crawlers may read which paths. llms.txt provides orientation: a plain-text summary of who you are, what you offer, and where your key pages live, written for AI systems that already have access. A well-set-up site has both: the door open, and a clear sign inside explaining the building.
Related reading
- How to Measure Your AI Visibility as a Canadian Business
- Zero-Click Search in Canadian B2B: Winning When Nobody Visits Your Site
- Answer Engine Optimization for Canadian B2B Companies: The 2026 Guide
The bottom line on AI crawlers and robots.txt
Every marketing dollar you spend downstream of your website assumes machines can read the site. One inherited text file can void that assumption silently, and in our audits it does, far more often than anyone expects. Five minutes to check, an hour to fix, a few weeks for the engines to notice, and every guide, review, and citation you build afterward finally counts. Check the file before you spend another dollar on what sits behind it.
If you would rather have the whole access-and-visibility stack checked at once, start with our free AI visibility audit, or book a free fit call with AlphaPixels and we will tell you in plain English what the machines can and cannot see.