A technical SEO audit for AI search checks three things: whether AI crawlers can reach your pages, whether they can read the content once they arrive, and whether anything on your site tells AI search not to use it. Four hidden blockers are worth checking first: a bot setting in your CDN or firewall, a robots.txt rule written too broadly, content that only appears after JavaScript runs, and snippet controls that stop Google using a page in AI answers. Most can be checked with free tools, and the eight checks below show how.
Start with one question, because it catches the blocker most audits miss: when did anyone last look at your CDN or firewall bot settings? Cloudflare alone says more than one million customers had switched on its one-click AI crawler block by July 2025. If your site was one of them, that choice may still be in place.
Technox offers AI SEO services and technical audits, so we have an interest in this topic. For why technical SEO decides AI visibility in the first place, read our guide to technical SEO, topical authority and AI search visibility. This post is the hands-on companion: what to check, where to click and what a pass looks like.
The audit at a glance
Check 1: Is your CDN or firewall blocking AI crawlers?
This is the check most audits miss, because the setting lives outside the website. Cloudflare's history shows why it matters:
• In September 2024, Cloudflare added a one-click option to block AI crawlers, and more than one million customers had turned it on by July 2025 (Cloudflare, July 2025).
• On 1 July 2025, Cloudflare announced it would block AI crawlers by default unless site owners chose otherwise, asking every new domain whether to allow them (same source).
• In September 2026, Cloudflare added a "Disallow AI Training" setting that lets a site stay indexed for search while refusing training. It reported that fewer than 1% of Cloudflare sites block search bots, while 17% use some mechanism to block training (Cloudflare, September 2026).
• Cloudflare's current documentation says that for new domains, Search, Agents and Training all default to "Allow (do not block)" (Cloudflare Docs, updated November 2026).
So whether your site blocks AI crawlers depends on when it was set up and what someone clicked since. To check on Cloudflare, go to Security > Settings, filter by Bot traffic and open AI Crawler and Agent access policies. You will see separate actions for Search, Agents and Training. For individual crawlers, AI Crawl Control's Crawlers tab shows each one with an Allow or Block action (Cloudflare Docs).
A sensible default for most businesses: allow Search, so AI search tools can find and cite you, and decide on Training as a separate business choice. If you use another CDN, security plugin or host firewall, look for a similar AI bot setting. We could not find a primary statement from other major providers on whether they block AI bots by default, so check yours directly.
Check 2: What does robots.txt actually say?
Open yourdomain.com/robots.txt and read it, rather than relying on what your SEO plugin claims. Two things to watch for:
• Rules added by your CDN. If Cloudflare's Bot Preference Sync is on and any policy is set to Block or Disallow, Cloudflare generates managed robots.txt directives from those policies (Cloudflare Docs). Your robots.txt may contain rules nobody on your team wrote.
• Rules that are too broad. A rule meant to stop a training crawler can also stop a search crawler if it names the wrong user agent or blocks everything. Our engine-by-engine guide to optimising for AI search includes a table of which crawler does what.
Remember that robots.txt is a request, not a lock. Cloudflare's own documentation says "robots.txt compliance is voluntary" and that the file does not prevent crawlers from accessing content at a technical level. Blocking at the CDN or server is what actually refuses requests, and even that only works against crawlers that identify themselves honestly.
Check 3: Can bots see your content without JavaScript?
Google renders JavaScript. To see what Google sees, Google recommends the Rich Results Test or the URL Inspection Tool in Search Console, both of which show the rendered page, loaded resources and JavaScript errors (Google Search Central).
Many AI crawlers do not. A 2024 Vercel analysis, reported by Search Engine Journal, found that major AI crawlers did not render JavaScript, though they did fetch JavaScript files (Search Engine Journal, December 2024). Glenn Gabe's tests, first published in August 2025, found ChatGPT, Claude and Perplexity could not read content rendered by client-side JavaScript, while Google's AI Overviews and AI Mode, and Bing's Copilot, could; his October 2026 update says ChatGPT and Claude still cannot render JavaScript-based content (GSQi, updated October 2026).
How to check:
1. Turn off JavaScript in your browser's settings and reload your key pages. Anything that disappears, such as product details, prices, reviews or FAQs, may be invisible to many AI crawlers.
2. View the page source (not the inspected elements) and search for a sentence from your main content. If it is not in the source, it is loaded by JavaScript.
3. Run the page through Google's URL Inspection live test to confirm Google, at least, sees the rendered version.
If important content depends on JavaScript, ask your developer about server-side rendering or pre-rendering for those pages.
Check 4: Are the bots in your logs real?
Anyone can put "GPTBot" or "Googlebot" in a user agent string. To know which visits are genuine, compare the visiting IP address with the IP ranges each company publishes:
This matters for two reasons. Fake bots can scrape or attack your site while pretending to be crawlers you allow. And the reverse also happens: in August 2025, Cloudflare reported that when Perplexity's declared crawlers were blocked, it saw undeclared requests using a generic Chrome browser user agent and IPs outside Perplexity's published ranges, at 3 to 6 million requests a day (Cloudflare, August 2025). Perplexity disputed this, saying Cloudflare had confused it with traffic from BrowserBase, a third-party browser service it uses for fewer than 45,000 requests a day (Perplexity, August 2025). Either way, user agents alone do not tell you who is visiting.
One practical note: if you test access by sending a request with a bot's user agent from your own computer, you are only testing rules based on user agent. Cloudflare's free plan, for example, identifies AI crawlers by user agent string (Cloudflare Docs), while paid bot management uses more signals.
Check 5: Which AI bots visit, and how much?
Your server or CDN logs show which AI bots fetch which pages. Look for three patterns: search crawlers reaching your important pages, user-triggered fetchers (such as ChatGPT-User) appearing on pages people are asking about, and training crawlers using a large share of your bandwidth.
For context, AI crawling is large and mostly not sending visitors back:
• Cloudflare's 2025 review found Anthropic had the highest crawl-to-refer ratio of the leading platforms, reaching as much as 500,000 crawls per referral, while OpenAI's peaked at 3,700:1 in March 2025 (Cloudflare Radar 2025 Year in Review).
• Training made up 22% of the crawler requests Cloudflare saw in spring 2025 and 52% by June 2026, and daily requests from AI agents grew by more than 1,700% in a year (Cloudflare, September 2026).
• Fastly found AI crawlers made up almost 80% of AI bot traffic between April and July 2025, with Meta generating 52% of AI crawler traffic, and 98% of real-time fetcher requests coming from OpenAI's bots (Fastly, August 2025).
If a training crawler is consuming heavy bandwidth with no benefit to you, that is a reason to limit it. Search crawlers and user-triggered fetchers are usually worth keeping, because they are how AI answers find and cite you.
Check 6: Google's snippet and AI controls
Two settings can quietly remove your pages from Google's AI Overviews and AI Mode: a nosnippet tag (or a very low max-snippet value) on a page, and the Search Console setting that opts a site out of Google's generative AI features. Search your page templates for "nosnippet" and check Search Console's settings. Our guide to optimising for AI search engines explains exactly what each control does, with Google's wording.
Check 7: Does structured data validate?
Run key page types, such as your home page, a service or product page and an article, through Google's Rich Results Test, which reads the rendered source code (Google Search Console Help), and through the Schema Markup Validator for markup Google's tool does not cover. Check that the markup matches what is visible on the page: business name, address, prices and reviews. Structured data will not get you into AI answers on its own, but wrong or broken markup gives search systems conflicting facts.
Check 8: Are key pages indexed in Google and Bing?
Google's AI features draw on Google's index, and several AI tools draw on Bing's. Confirm your most important pages are indexed in Google Search Console and Bing Webmaster Tools, submit an accurate sitemap to both, and use IndexNow if your platform supports it. If pages are missing, our post on why Google is not indexing your pages covers the usual causes.
What about llms.txt?
You do not need one to pass this audit. Google's Lighthouse tool now includes an llms.txt check, but its documentation says a missing file is marked "Not Applicable" because "providing the file is optional at the moment" (Chrome for Developers). Google has said its search features treat llms.txt like any other text file. If you add one, treat it as a convenience, not a fix for any of the problems above.
How often to run this audit
Run checks 1, 2 and 6 after any change to your CDN, security plugin, theme or SEO plugin, because those are the changes that silently add blocks. Run the full audit every quarter, and check logs (check 5) monthly if AI visibility matters to your business. To see whether the fixes improve your visibility, use our guide on how to measure AI visibility.
Where Technox fits
Checks 1, 2, 6 and 8 need no technical skill, only access to the right dashboards. Checks 3, 4 and 5 involve page source, server logs and IP verification, which is where most business owners hand over to a developer or agency.
If you want us to run the audit, ask for a free audit or contact us. Our AI SEO services in Coimbatore page explains how we work.
Frequently asked questions
Does Cloudflare block AI crawlers by default?
Not for new domains today. Cloudflare announced default blocking in July 2025, but its current documentation says new domains default to allowing Search, Agents and Training crawlers. Sites set up earlier, or where someone switched on the AI block, may still block them. Check Security > Settings > AI Crawler and Agent access policies in your Cloudflare dashboard.
Do AI crawlers render JavaScript?
Google does, for its own search and AI features. Tests and analyses found that many standalone AI crawlers, including ChatGPT's and Claude's, do not render JavaScript. Content that only appears after JavaScript runs may be invisible to them, so key text, prices and FAQs should be in the page's HTML.
How do I check if AI bots can crawl my site?
Check your CDN or firewall bot settings, read your robots.txt, view your key pages with JavaScript turned off, and look at your server logs for visits from crawlers such as OAI-SearchBot, PerplexityBot and Bingbot, verified against their published IP ranges.
How do I know a visit is really from GPTBot or Googlebot?
Compare the visiting IP address with the IP lists the companies publish, such as OpenAI's gptbot.json and searchbot.json, or run a reverse DNS lookup for Googlebot. A user agent string alone can be faked.
Do I need an llms.txt file for AI search?
No. Google's Lighthouse treats a missing llms.txt file as not applicable because the file is optional, and Google has said its search features treat llms.txt like any other text file. Fixing crawler access, rendering and indexing matters far more.
Should I block AI training crawlers?
It is a business decision, separate from search. Blocking training crawlers keeps your content out of future model training, while allowing search crawlers keeps you eligible to be cited in AI answers. Cloudflare now offers a "Disallow AI Training" setting that keeps search indexing while refusing training.




