Crawlability vs Indexability: The Real Difference

Crawlability and indexability get used as if they mean the same thing. They don't, and treating them as interchangeable is why some SEO fixes go nowhere: you spend weeks improving something Google could already do fine, while the actual blocker sits one step further down the pipeline.

This article draws the line between the two clearly. What each one controls, where they depend on each other, how to tell which is actually broken on a given page, and what changes now that AI crawlers are part of the equation too.

What Crawlability Actually Means
Crawlability is whether a search engine's bot, such as Googlebot, can reach a page and read what's on it. Crawlers move through the web the same way a person clicking through a site would, following <a href> links from one page to the next. Ahrefs' glossary entry on the topic puts it plainly: crawlability is the ability of a crawler to access website pages and resources, and it's distinct from what happens to that page afterward. For readers who want the broader picture before this technical breakdown, our guide to what SEO actually involves covers where crawlability fits within the discipline as a whole.
A page needs to be found before it can be crawled. That sounds obvious, but it's where a lot of sites quietly fail. A page that exists only in an XML sitemap, with zero internal links pointing to it, is technically "known" to Google but poorly positioned to be crawled promptly. Pages with no links pointing to them at all, often called orphan pages, may never get discovered.
Google's own documentation on crawlable links is specific about the mechanics: it can only reliably crawl a link if it's a proper <a> element with an href attribute. Links built entirely through JavaScript click handlers, buttons styled to look like links, or <a> tags missing an href, often go unseen. This is a common gap on sites built with heavy client-side frameworks, where navigation renders correctly in a browser but isn't present in the raw HTML Googlebot first fetches.
What Indexability Actually Means
Indexability picks up after crawling ends. Once Googlebot has fetched a page, Google still has to decide whether that page earns a place in its index, the database it actually searches when someone types a query. A page can be perfectly crawlable and still fail this second test.
That decision weighs several things at once: whether a noindex directive is present, whether the page duplicates something already indexed, whether a canonical tag points somewhere else entirely, and increasingly, whether the content clears a basic usefulness bar. None of these are crawling problems. Googlebot saw the page fine. It just decided not to store it.
The Difference, in One Table

Crawlability
Indexability
What it governs
Whether a crawler can reach and read the page
Whether Google stores the page in its index
Happens
Before content is evaluated
After the page has been crawled
Controlled by
Internal links, sitemaps, robots.txt, server responses, JavaScript rendering
Noindex tags, canonical tags, duplicate or thin content, quality signals
Typical failure
Page is orphaned, blocked, or never rendered properly
Page loads fine but gets excluded or consolidated into another URL
Where to check it
Screaming Frog crawl, server logs, robots.txt tester
Search Console's Page Indexing report, URL Inspection tool
A page has to clear both bars to show up in search results. Clearing only one gets you nowhere, and knowing which one it failed changes everything about how you fix it.
A Page Can Be One Without Being the Other
Four combinations exist here, and three of them are worth understanding on their own.
Crawlable and indexable. This is the normal, healthy state. Googlebot reaches the page and adds it to the index. Most pages on a functioning site fall here.
Crawlable but not indexable. Google can see the page just fine and chooses not to keep it. A noindex tag, a canonical tag pointing to a different URL, or content that overlaps too closely with something already indexed are the usual reasons. This is a content or configuration decision, not an access problem.
Not crawlable but technically indexed. This one surprises people. Google's own robots.txt documentation states directly that a page blocked in robots.txt can still show up in search results if other sites link to it, since Google can find and index the URL from those external links even without ever fetching the page's content. The result is a bare, degraded listing: the URL itself, maybe some anchor text pulled from the linking page, but no real title or description, because Google never read the page to generate one.
Not crawlable and not indexed. Fully invisible. No path in, nothing to show. This is usually the result of an overly broad robots.txt rule, a server that returns errors on every request, or a page that simply has no links pointing to it anywhere on the web.
That third category is the one most guides skip, and it's exactly why "blocked in robots.txt" is not the same guarantee of invisibility that people assume it is.
What Actually Controls Crawlability
A handful of technical factors decide whether Googlebot ever reaches a page in a usable form.
Discoverability. A page needs a path in, ideally more than one. Internal links from already-crawled pages do more here than a sitemap entry alone, because a sitemap simply lists a URL exists; it doesn't tell Google the page matters enough to prioritize.
Nofollow links. Googlebot does not follow links carrying a rel="nofollow" attribute. If the only internal link to a page carries that attribute, the page is effectively invisible to that path of discovery, even though the URL might still appear elsewhere.
robots.txt rules. Broad disallow rules written to block a handful of URLs sometimes catch an entire folder by accident. This is one of the most common self-inflicted crawlability problems on larger sites.
JavaScript rendering. Content injected client-side after the initial HTML load requires Google to render the page in a second pass before it can be evaluated. That second pass isn't guaranteed to happen quickly, and on large sites with limited crawl budget, it sometimes doesn't happen at all for lower-priority pages.
Server responses. Slow response times, intermittent 5xx errors, or redirect chains all reduce how much of a site Googlebot is willing to fetch in a given crawl session. Google's crawl budget documentation frames this directly: crawl capacity, how much load a server can absorb, and crawl demand, how much Google wants to visit a given URL, together decide how far a crawl actually goes.
What Actually Controls Indexability
Once a page clears crawling, a different set of signals decides whether it stays.
A no index directive, whether placed as an HTML meta tag or as an X-Robots-Tag HTTP header, is the most direct override. Google's documentation on blocking search indexing is explicit that this only works once Google has actually crawled the page and seen the tag; a no index rule sitting behind a robots.txt block that prevents crawling in the first place never gets read at all, which is a contradiction that trips up a surprising number of migrations.
Canonical tags decide which version of near-duplicate content gets the index slot when several URLs return substantially the same thing, such as a product page reachable through three different filter combinations. Content depth and originality matter too, particularly for pages generated at scale, like bulk location pages or templated service variants, where each individual URL needs enough unique substance to justify existing as its own indexed entry rather than being folded into a stronger, more complete page.
How to Tell Which One Is Actually Broken
Start with the URL Inspection tool in Search Console rather than guessing. If it reports the page as "Discovered, currently not indexed" or shows no crawl history at all, the problem sits upstream, in crawlability. If it shows the page was crawled and then lists a specific exclusion reason, such as a no index tag or a canonical pointing elsewhere, the crawl succeeded and the problem is indexability.
A second, faster check: run a crawler like Screaming Frog against the site and compare what it finds by following links against what's listed in the XML sitemap. Any URL sitting in the sitemap that the crawler never reaches through a link is an orphan, a crawlability issue by definition, regardless of how good the content on that page is.
For sites carrying real technical debt, this is usually where a proper technical audit earns its cost. A technical SEO audit walks through exactly this sequence, page by page, rather than fixing one symptom and hoping the rest follows, and our own breakdown of what a technical SEO audit process actually covers goes into that sequence in more detail.
Why the Distinction Actually Matters to a Business
Confusing the two wastes real time. A team that spends a sprint rewriting product descriptions because pages "aren't showing up" gains nothing if the actual cause was a robots.txt rule blocking the entire /products/ folder after a platform migration. The content was never the problem; nobody ever got to see whether it was good. The broader case for treating this kind of technical groundwork as a business priority, not just a developer task, is covered in why SEO matters for business success.
The reverse mistake costs just as much. A site that keeps requesting re-crawls and resubmitting sitemaps for pages that are being crawled just fine, but rejected on quality or duplication grounds, is treating an indexability problem as if it were an access problem. No amount of resubmission fixes a canonical tag pointing somewhere else.
AI Crawlers Have Added a Third Layer
Crawlability used to mean one bot: Googlebot. That's no longer accurate. A growing set of AI crawlers now fetch pages for entirely different reasons, and they don't all behave the same way robots.txt has trained site owners to expect for the last three decades.
Training crawlers like GPTBot, ClaudeBot, and Google-Extended collect content to build model training datasets and generally respect robots.txt closely. Search and answer crawlers, including OAI-SearchBot, PerplexityBot, and Claude-SearchBot, fetch pages in real time specifically to power AI-generated answers and citations, which makes blocking them a direct visibility decision rather than a training-data preference.
Data from a Cloudflare-network analysis published in September 2026 put concrete numbers on how publishers are actually treating these bots: GPTBot was named in more robots.txt disallow rules than any other AI crawler, ahead of ClaudeBot, Google-Extended, and CCBot, and the naming volume for all of them has been climbing steadily through the year. Separate research tracking the top 1,000 sites found GPTBot blocking has plateaued at roughly a quarter of those sites since 2024, while a "middle path," blocking training bots but explicitly allowing search-time bots like PerplexityBot and OAI-SearchBot, has become the single most common configuration. The same research noted that close to 90% of AI crawler traffic overall is training-related rather than search-related, which is part of why publishers are comfortable blocking the bulk of it while leaving the smaller, citation-driving slice open.
For a business weighing this, the practical takeaway isn't "block everything" or "allow everything." It's that robots.txt now needs separate, deliberate rules for training bots versus search bots, something covered in more depth in our guide to Google AI Mode optimization, since getting cited in an AI-generated answer depends on the search bot reaching the page in the first place, the same crawlability question this article started with, just with a different crawler.
The Data Behind This

Data point
Source
What it means for your strategy
A robots.txt-disallowed page can still be indexed, without being crawled, if other sites link to it
Google Search Central
Blocking a URL in robots.txt is not the same as guaranteeing it stays out of search results entirely. Use no index, not robots.txt alone, when the goal is full removal.
Google can only reliably crawl links built as proper <a href> elements
Google Search Central
JavaScript-only navigation risks leaving whole sections of a site undiscovered. Test with rendered and raw HTML both.
Crawl budget is set by crawl capacity limit combined with crawl demand
Google Search Central
Slow servers get crawled less thoroughly per visit regardless of how good the content is, delaying indexing for new pages.
GPTBot appears in more robots.txt disallow rules than any other AI crawler as of September 2026
Cloudflare Radar data, via Technology Checker
If AI-search visibility matters to your business, verify GPTBot and search-specific bots aren't accidentally caught by the same blanket rule.
GPTBot blocking has held at roughly 25% of top sites since 2024, and blocking training bots while allowing search bots is now the most common configuration
Presence AI, 2026 research
A blanket "block all AI bots" robots.txt rule is now a minority position, and it forecloses citation in AI Overviews and AI-powered search answers.
Roughly 89% of AI crawler traffic is training-related, not search-related
Presence AI, 2026 research
Most of the AI crawl volume hitting a server produces no referral value, which explains why selective blocking of training bots specifically has become standard practice.
Common Mistakes
Blocking a folder in robots.txt to "remove it from search," not realizing the page can still surface as a bare, titleless listing if anything else on the web links to it.
Adding a no index tag to pages that are also disallowed in robots.txt, which means Google never crawls far enough to see the no index rule at all.
Treating a slow site purely as a user experience problem, missing that it also throttles how much of the site Googlebot bothers to crawl per visit.
Assuming a JavaScript-heavy site is fully crawlable because it looks complete in a browser, without checking what the raw HTML actually contains.
Applying one blanket robots.txt rule to every AI bot, which either blocks legitimate AI-search citation opportunities or does nothing to stop training crawlers, depending on which direction the rule leans.
Frequently Asked Questions
What is the main difference between crawlability and indexability?

Crawlability is whether a search engine's bot can access and read a page. Indexability is the separate decision, made after crawling, about whether that page gets stored in the search engine's index and made eligible to appear in results.
Can a page be crawlable but not indexable?

Yes, and it's common. Google can crawl a page without issue and still exclude it from the index because of a no index tag, a canonical tag pointing elsewhere, or content that's too thin or duplicative compared to what's already indexed.
Can a page be indexed without being crawled at all?

In a limited sense, yes. If a page is blocked in robots.txt but other sites link to it, Google can still list the bare URL in search results based on those external signals, without ever having read the page's actual content.
Does blocking a page in robots.txt guarantee it won't appear in Google?

No. Google's own documentation states that a disallowed page can still surface in search results if it's linked from elsewhere on the web. Use a no index directive, password protection, or full removal if the goal is to keep a page out of results entirely.
How do I check whether a page's problem is crawlability or indexability?

Use the URL Inspection tool in Search Console. If it shows no crawl history or a "discovered, not indexed" status, the issue is upstream in crawlability. If it shows the page was crawled and lists a specific exclusion reason, the issue is indexability.
Does JavaScript affect crawlability?

It can. Content and links that only appear after JavaScript executes require Google to render the page in a separate pass before it can evaluate them, and that pass is not guaranteed to happen quickly, particularly on large sites.
What is an orphan page, and why does it matter for crawlability?

An orphan page has no internal links pointing to it from anywhere else on the site. Without a link path, crawlers may never find it, regardless of whether it's listed in the sitemap.
Do AI crawlers follow the same rules as Googlebot?

Not entirely. Training-focused AI crawlers like GPTBot and ClaudeBot generally respect robots.txt, but some bots have been documented ignoring it, and real-time, user-triggered fetchers sometimes behave differently from automated crawlers entirely. Each bot's behavior needs to be checked individually rather than assumed.
Should I block all AI crawlers to protect my content?

That depends on the goal. Blocking training crawlers keeps content out of model training data. Blocking search and answer crawlers as well removes any chance of being cited in AI-generated search answers, which functions as a visibility channel in its own right for many businesses.
What tools can I use to check crawlability and indexability separately?

Search Console's URL Inspection tool and Page Indexing report cover indexability directly. For crawlability, a crawler like Screaming Frog, Semrush's Site Audit, or Ahrefs Site Audit can map which pages are actually reachable through links versus which only exist in a sitemap.

SEO

The SEO & AI Visibility Checklist for 2026

Visibility used to mean one thing: ranking in Google. It now means two. You need to be found in classic search and cited in AI answers and the two share a foundation but diverge at the edges. This checklist covers both, grouped so you can audit one area at a time and hand the fixes to whoever owns them.  Work through it top to bottom, because the order matters: a page that engines cannot read cannot rank or be cited, so the technical checks come first and the AI-specific ones build on them. Each item says what to check and why it earns its place. Where a topic needs real depth, it is linked out rather than re-explained here, so this stays a checklist and not an essay. Tick what is done, flag what is not and treat every unticked box as a task with an owner and a date. A checklist you only read changes nothing. Search and AI visibility share a spine, then split at the top It helps to picture the two as one trunk with two branches. The trunk is everything a machine needs to reach and understand your content: a crawlable site, clear structure, consistent facts and genuine authority. Classic search and AI answers both grow from that trunk, which is why most of this checklist serves both at once. The branches are where they part.  Classic search still rewards the page as a whole and the links between pages, so architecture and internal linking carry weight. AI answers reward the individual passage and the wider agreement about your brand, so answer formatting and off-site mentions carry more. Audit the trunk first, because a weakness there starves both branches, then tune each branch for the surface it feeds. 1.  Make search engines able to read you before anything else Nothing below this line works if this section fails, so start here. Check Why it earns its place Every important page is indexable If an engine cannot index a page, nothing else on this list matters for it. An XML sitemap lists live URLs and is submitted It helps engines discover pages faster and reveals what is missing. robots.txt allows crawling of what should rank A single stray disallow can hide whole sections from search. Core Web Vitals pass on mobile Slow, unstable pages shed rankings and visitors before a word is read. The site is served entirely over HTTPS Security is a baseline trust signal for engines and buyers alike. No important page is orphaned A page with no internal links is invisible to crawlers that travel by links. Canonical tags resolve duplicate versions They point engines to the version to rank and consolidate its signals. Broken links and redirect chains are cleared They waste crawl budget and leak authority you have already earned. These checks follow Google's guidance on crawling and indexing and on Core Web Vitals . 2.  Give every page one clear intent to answer Each page should do one job and make that job obvious to a machine. Check Why it earns its place One primary search intent per page Two pages chasing the same intent split your own ranking signal. A descriptive title tag under about 60 characters It is your headline in results and a strong relevance cue. A meta description written to earn the click It does not rank the page, but it decides whether people choose it. A single, specific H1 naming the topic It anchors what the page is about for both readers and machines. Question-shaped H2s that name each section Specific headings let engines match a section to a query. The answer stated first, the detail second Front-loaded answers are easier to read and easier to quote. Descriptive alt text on meaningful images It aids accessibility and opens up image search visibility. A visible, accurate last-updated date Freshness signals that the page is maintained, not abandoned. 3.  Tell machines who you are with entities and schema Structured data is how you stop being a guess and start being a known entity. Check Why it earns its place Organization or LocalBusiness schema on the site It defines your brand as an entity engines can recognise and trust. Name, address and phone identical everywhere Conflicting details make engines hesitate to surface or cite you. FAQ schema on pages with genuine questions It labels answers so machines can lift them without guessing. Product and review schema where relevant It surfaces rich results and feeds comparison signals. Author and reviewer markup on expert content It supports the experience and authority signals engines weigh. sameAs links to your verified profiles They connect your site to your wider entity footprint. A presence in Wikidata and key platforms It gives AI independent confirmation of who you are. Mark it up to the schema.org vocabulary and Google's structured data guidelines . Context: what AI SEO actually is , and why entities sit at its centre. 4.  Become the answer AI can extract Answer engine optimization is mostly the discipline of writing passages that lift out whole. Check Why it earns its place Each key question has a self-contained answer A passage that stands alone can be quoted alone. Comparisons are written as tables Structured formats are lifted more readily than buried prose. Steps and sequences are numbered lists The format matches how an assistant assembles an answer. Definitions are single, quotable sentences A clean definition is the easiest thing for a machine to cite. Answers are concise rather than padded Tight answers extract better than long build-ups. Headings phrase the question the section answers They route the engine straight to the relevant passage. Deep-dive: building answer-ready pages at the passage level. Understanding how AI systems retrieve and assemble these answer-ready passages can make this process easier. Our guide on how AI search works explains how crawling, indexing, retrieval, and generation work together to determine what information can appear in an AI-generated answer. 5.  Earn the trust that makes AI cite you Citations are earned across the web, not just on your own pages. Check Why it earns its place Your site is verified and indexed in Bing AI answers lean on Bing, so Bing visibility feeds them directly. The brand is mentioned across trusted third-party sites Consensus across independent sources is what AI leans on to cite. You appear in relevant best-of and comparison lists These are the sources assistants quote when recommending. Reviews are genuine, recent and responded to Ratings are among the strongest trust signals for AI recommendations. An llms.txt file points AI crawlers to key content It guides engines to what matters, at almost no cost. Major AI crawlers are allowed to reach your content Blocking them removes you from the answers they generate. These off-site signals are part of a broader brand authority picture. To understand how third-party mentions, reviews, citations, and consistent brand signals influence AI visibility, see our guide on Does Brand Authority Matter for AI Visibility? Building these signals consistently requires more than optimizing individual pages. AI SEO services in Coimbatore can bring together technical foundations, entity optimization, content structure, and authority signals into a coordinated AI visibility strategy. Start by verifying your site in Bing Webmaster Tools , since AI answers lean on Bing. Deep-dive: how to get cited by ChatGPT , the off-page mechanics in full. 6.  Win the map and the near-me answer For a local business, this section often outranks everything above it in value. Check Why it earns its place Your Google Business Profile is complete and verified It is the entry point for map and local-pack visibility. Categories are accurate and specific They decide which local searches you surface for. Reviews are growing and you reply to them Volume and responses lift local trust and ranking. Local citations carry consistent details They reinforce your location entity across the web. Location or service-area pages exist where relevant They match nearby, high-intent searches to a real page. Photos and posts are kept current An active profile signals a live, trustworthy business. 7.  Measure search and AI visibility together You cannot improve what you do not watch, and AI visibility needs its own watching. Check Why it earns its place Google Search Console is connected and reviewed It shows impressions and positions, the earliest signs of progress. Google Analytics 4 tracks organic conversions It ties visibility back to enquiries and revenue. Keyword and local rankings are tracked over time Trends reveal direction that any single day hides. Key questions are tested in ChatGPT, Perplexity and AI Overviews It shows whether AI cites you, a competitor, or no one. Brand mentions are monitored across the web They act as a leading indicator of future AI citations. Connect Google Search Console first; it is the earliest window on whether the rest is working. The ten checks that move the needle first If the full list is daunting, start here. These ten give the most visibility per hour of work across both search and AI, so clear them before anything else. 1.  Confirm your key pages are indexable and actually indexed. 2.  Fix Core Web Vitals on mobile. 3.  Add Organization or LocalBusiness schema with consistent contact details. 4.  Give each page one clear intent, a specific title and a single H1. 5.  Front-load a direct answer under each question-shaped heading. 6.  Add FAQ schema to pages with genuine questions. 7.  Verify and index your site in Bing. 8.  Complete and verify your Google Business Profile. 9.  Earn and respond to genuine reviews. 10.   Test your top questions in ChatGPT, Perplexity and AI Overviews to see who gets cited. Run it as a cycle, not a one-off Visibility is not a project you finish; it is a state you maintain. Audit the full list once to establish a baseline, fix the failures in priority order, and then re-run the AI-specific sections every quarter, because the engines and their preferences change faster than the technical basics do.  A page that was cited in an assistant answer last quarter can quietly drop out when a model updates, so the test-your-questions check belongs on a repeating calendar, not a to-do list you clear once and forget. Questions this checklist tends to raise What is the difference between SEO visibility and AI visibility? SEO visibility is being found in classic search results; AI visibility is being cited inside AI answers such as Google's AI Overviews or a ChatGPT reply. They share a foundation, a crawlable site, clear structure and real authority, but AI visibility adds passage-level answer formatting, entity clarity and off-site consensus. This checklist audits both because a business now needs the two together. Do I have to finish the whole list before I see results? No. The list is ordered by dependency, so clearing the technical and on-page sections often produces visible movement on its own, while the AI-specific work compounds on top of it. Start with the ten priority checks, measure, and then work outward. Partial progress on the right items beats a perfect plan you never begin. How often should I re-run this audit? Audit everything once for a baseline, then set a lighter quarterly review focused on the entity, AEO and AI-citation sections, since those shift most as engines change. The technical and local basics need checking after any major site change or a few times a year. Measurement runs continuously in the background. Can I do this myself or do I need help? Much of the on-page, local and measurement work is doable in-house with care and time. The technical fixes, schema and off-page authority building tend to move faster with specialist help, especially where a mistake, like blocking a crawler or breaking canonicals, can cost visibility quietly. Use the checklist to decide which boxes you can tick yourself and which are worth handing over. Turn the boxes into a plan A checklist earns its keep only when it becomes a schedule. Assign each unticked box an owner and a date, put the AI-visibility checks on a quarterly repeat, and review progress against the measurement section every month. Visibility in 2026 is won by the businesses that treat it as maintenance, not a launch. Want the whole audit run for you, with the fixes prioritised against your market? Talk to our SEO team in Coimbatore , and start with a baseline audit of where you stand across search and AI.

8 minsRead Article

SEO

How to Optimize Title Tags & Meta Descriptions for Better SEO in 2026

Most businesses treat the title tag and meta description as a five-minute afterthought. That habit is expensive. These two elements decide whether a page that ranks on page one ever gets clicked at all. In 2026, they also decide whether an AI search engine can confidently quote your page in an answer. This guide covers what happens to your title tags and meta descriptions once Google gets hold of them, why Google rewrites most of them anyway, and how to write ones that survive. If you're still getting up to speed on the basics, our what is SEO primer is a good starting point before diving into this one. What Title Tags and Meta Descriptions Actually Do A title tag is the HTML element that names a page. It shows up as the blue clickable headline in Google results and in your browser tab. A meta description is a short HTML summary of the page. It shows up as the grey text underneath the title. Here's the key difference: The title tag is a real relevance signal. Google uses it to understand the page, and it's your biggest lever over click-through rate. The meta description carries no ranking weight on its own. But it strongly shapes whether someone clicks your result over the next one. Together, they do one job: earn the click. Write them separately and you waste space repeating yourself instead of making the case. Pixel Width Matters More Than Character Count Most checklists say "keep your title under 60 characters." That's a rough proxy, not the real rule. Google truncates titles and descriptions by pixel width , not character count. A capital "W" takes up roughly three times the space of a lowercase "i". So two titles with the same character count can display completely differently. Element Desktop Cutoff Character Equivalent Mobile Cutoff Character Equivalent Title Tag ~600 pixels 50-60 characters ~480 pixels 40-50 characters Meta Description ~920 pixels 150-158 characters ~680 pixels 110-120 characters Treat these character counts as a safety margin, not a target to max out. Practical fix: Use a pixel-based preview tool before publishing. Put your most important words in the first 40-50 characters so the meaning survives even if the rest gets cut. Why Google Rewrites Most Titles Anyway Here's the part that surprises most business owners: a perfectly sized, well-written title tag still isn't guaranteed to show up as written. The data: Zyppy SEO studied over 80,000 title tags across 2,370 websites. Google rewrote 61.6% of them, at least partially. A newer 2025 study found the rewrite rate had climbed to 76.04% . Only 24% of titles survived untouched, and Google removed an average of 2.71 words when it did rewrite. What this means for you: title tag optimization isn't a "set once, forget it" task. It's an ongoing discipline of writing titles specifically to reduce the odds of a rewrite. Length is the single biggest trigger. Here's the breakdown: Title Length Rewrite Rate What It Means Under 20 characters Rewritten almost every time Too little context for Google to trust it 51-60 characters Lowest rewrite rate (39-42%) The safest range to write in 60+ characters, wide letters High risk Wastes the words that do survive Titles with brackets [ ] Rewritten 77.6% of the time Google often strips the bracketed part entirely A quick example: a title like "Best CRM Software [2026 Guide]" will most likely lose the bracketed part. Google reshapes what's left around your H1 or page content instead. Parentheses fare better than square brackets. Dashes survive more often than pipes. So "Best CRM Software for Small Teams - Complete 2026 Guide" holds up better than the bracketed version. Meta Descriptions Get Rewritten Even More If title tags feel unreliable, meta descriptions are worse. This is why many businesses give up on writing them at all. The data: Ahrefs studied 20,000 keywords and found Google rewrites meta descriptions 62.78% of the time. Portent's independent study found an even higher rate: 71% on mobile , 68% on desktop . That doesn't mean stop writing them. It means aim your meta description at one job: winning the click for your single most important target query. Google often pulls a passage from your body content instead, when it thinks that passage answers a searcher's specific query better than your fixed description. A page ranking for fifteen keyword variations might show fifteen different snippets, only one of which is the description you wrote. The real fix: write a strong description for your main query, and structure your body content with clear, quotable sentences that can stand in as a snippet for everything else. The Click Math That Makes This Worth Doing Ranking well isn't the same as being seen. Being seen isn't the same as being clicked. Backlinko analyzed roughly 4 million search results and found: The #1 organic position earns a 27.6% average click-through rate . The top 3 results together capture 54.4% of all clicks on the page. By position 10 , click-through rate drops to roughly 1.7-2.8% . What this means in practice: if two competing pages rank #2 and #3 for a valuable query, a stronger title and description on the lower-ranked page can genuinely out-click the weaker pairing above it. Ranking gets you in the room. The title and description decide who gets clicked. How to Write Titles That Hold Up Work through these in order. Lead with the topic, not the brand. Put the words a searcher actually typed in the first 40 characters. "Real Estate CRM for Small Agencies" beats "Technox Technologies - Software Solutions." Add one differentiator, not five. A location, a use case, or the current year is usually enough. Stacking three pushes you past the safe range. Match the title to your H1. Google increasingly ignores a title tag and pulls the H1 instead when the two don't align. Keep them close. Use dashes, not pipes. Skip brackets. This one formatting choice measurably changes your odds of a clean display. Write as if the title is all a searcher will ever see. For a meaningful share of visitors, it is. How to Write Descriptions That Earn Clicks The description has a different job than the title. The title matches intent and survives truncation. The description persuades the gap between "this looks relevant" and "this is worth my time." State the specific outcome first , in the first 110-120 characters, so it survives on mobile. Add one concrete supporting detail (a number, a timeframe, a deliverable) that a generic snippet couldn't guess. Don't repeat the title tag. Searchers scan both lines together, so repeating wastes the second line. Skip generic calls to action like "learn more." Google is statistically more likely to override these with its own text. How AI Search Changes the Game Search Everywhere Optimization means optimizing for Google Search, AI Mode, AI Overviews, ChatGPT Search, Perplexity, and Copilot all at once. These systems use your title as a compact summary when deciding whether your content deserves a citation in a generated answer. A vague, marketing-heavy title gives an AI system nothing concrete to lift. A clear, specific, entity-rich title gives it an easy, low-risk sentence to quote. Structured data helps here too. Adding Article, Product, FAQPage, or LocalBusiness schema doesn't fix a weak title. But it gives search engines and AI crawlers a second, machine-readable confirmation of what the page covers, which increases the odds your intended title gets used instead of a generated substitute. Title Tags vs. H1 Tags: Why the Confusion Costs You Business owners often assume the title tag and the H1 heading are the same thing. They're not, and mixing them up creates real problems. The title tag lives in the page's <head> section. It's invisible on the page itself. It only shows up in the browser tab and in search results. The H1 is the main visible heading on the page itself, the one a human reader actually sees first. They can say similar things, but they don't have to be identical. A title tag can be written tighter and more keyword-focused for the search results. The H1 can read more naturally for a person already on the page. Where this goes wrong: when the two are wildly different, Google increasingly treats that as a trust signal problem. If your title promises one thing and the H1 delivers something else, Google may just pull the H1 into search results instead of your carefully written title. Keep the core topic consistent across both, even if the phrasing differs slightly. Real Examples: Before and After Seeing a weak title rewritten well makes the framework easier to apply. Here are three examples across different industries. Real estate listing page Before: "Homes | GreenView Realty | Properties [Best Deals]" Problem: brackets, vague structure, no location, no property type. After: "3 BHK Apartments in Coimbatore - GreenView Realty" Wellness and healing center Before: "Welcome to Serenity Wellness Center" Problem: boilerplate "welcome" language tells Google nothing about the service. After: "Holistic Healing & Wellness Therapy in Coimbatore" Fitness and gym membership page Before: "PowerFit Gym | Home | Fitness | Workout | Gym" Problem: keyword stuffing, repeated terms, no differentiator. After: "Personal Training & Group Fitness Classes - PowerFit Gym" In each case, the fix follows the same pattern: drop the boilerplate or the keyword stack, lead with what the page actually offers, and add one differentiator (location, service type, or specialty) instead of five. Should Every Page Have the Brand Name in the Title? Not always. This is a judgment call, not a rule. For your homepage and top commercial pages, yes, adding the brand name at the end usually helps, especially once people recognize it. It builds trust and reinforces the association between the brand and the service. For long-tail blog posts and deep informational pages, it's often a waste of space. A brand suffix eats characters that a useful keyword or qualifier could occupy instead, and most readers landing on a blog post care more about the answer than the publisher. A reasonable default: keep the brand name on service pages and the homepage, drop it on individual blog articles unless there are characters to spare after the core topic is covered. How This Fits Into a Broader SEO Strategy Title tags and meta descriptions don't work in isolation. They sit inside a wider set of on-page and technical decisions that all reinforce each other. Strong internal linking helps Google understand which pages on a site are most important, which in turn affects how much attention Google gives to getting that page's title right. Clean site architecture makes crawling more efficient, so updates to titles and descriptions get picked up faster. Structured data adds a second layer of confirmation for what a page is about, on top of the title itself. Treat title and meta description work as one part of a larger technical and content SEO effort, not a standalone task that gets fixed once and forgotten. Google's own guidance on title links and meta descriptions is worth bookmarking as a reference. Beyond that, use Google Search Console's Performance report. Look at impressions and click-through rate together, at the page and query level. High impressions + low CTR = the title or description is the problem, not the ranking. Give it four to six weeks after any edit before judging results. Google doesn't always adopt a new title or description right away. Compare your live SERP snippet against your actual HTML to see whether Google is using your text or substituting its own. Where to Go From Here Getting titles and descriptions right across dozens or hundreds of pages, while fixing the underlying causes of rewrites like weak site structure or misaligned H1s, is ongoing work. It's exactly what a dedicated SEO company in Coimbatore handles as standard practice. Frequently Asked Questions What is a title tag in SEO?  The HTML element that names a webpage. It appears as the clickable blue headline in Google search results. What is a meta description and does it affect rankings?  A short HTML summary shown beneath the title in search results. It's not a direct ranking factor, but it strongly affects click-through rate. How long should a title tag be in 2026?  Aim for 50-60 characters. That usually keeps you under Google's ~600-pixel desktop cutoff, though the real limit is pixel width, not character count. How long should a meta description be?  Around 150-158 characters on desktop, 110-120 on mobile. Put the key information in the first 110-120 characters so it survives on mobile. Why does Google rewrite my title tags?  Usually because the title is too long, keyword-stuffed, doesn't match the H1, or doesn't match the searcher's specific query closely enough. Does using brackets or pipes in a title hurt SEO?  Brackets get rewritten far more than parentheses or dashes. Pipes get replaced or removed more than dashes. Plain, dash-separated titles hold up best. How much does professional title tag and meta description optimization cost?  It depends on site size and scope. A small business audit and rewrite can be a short fixed-fee job. Larger sites usually handle this as part of an ongoing monthly SEO retainer. How long does it take to see results after updating title tags?  Expect four to six weeks for Google to re-crawl and display your updates, and longer for click-through and ranking changes to show clearly in Search Console. Do title tags and meta descriptions matter for AI search results like Google AI Overviews or ChatGPT Search?  Yes. Clear, specific, entity-rich titles and descriptions make it easier for these systems to summarize and cite your page accurately. That's a core part of Generative Engine Optimization (GEO). What tools help check if my title tag will be truncated?  Pixel-based SERP preview tools. They show exactly where a title or description gets cut off before you publish, which is more reliable than counting characters. Should my title tag and H1 be exactly the same?  No, but they should stay closely aligned. The title tag can be tighter and more search-focused, while the H1 can read more naturally, as long as both communicate the same core topic. Do I need a different title tag for every page?  Yes. Duplicate or near-identical titles across multiple pages tell Google those pages aren't meaningfully different, which can hurt how each one ranks individually.

11 minsRead Article

Digital Marketing

How to Run ChatGPT Ads: Complete Step-by-Step Guide

ChatGPT is no longer just a place people go to ask questions. It is now a place businesses can pay to be found in. OpenAI began showing ads to Free and ChatGPT Go tier users in the US on February 9, 2026, then opened a self-serve Ads Manager to all US businesses on May 5, 2026, after briefly lowering the pilot's minimum spend from 200,000 dollars to 50,000 dollars in April. Within roughly six months, ChatGPT Ads crossed 1 billion dollars in annualized revenue run rate and expanded to more than 40 countries, including India. That speed of growth means most businesses evaluating this channel are working from outdated blog posts, half-finished pilot documentation, or a straight copy-paste of their Google Ads playbook. None of those approaches hold up well here. This guide walks through what ChatGPT Ads actually are, how the platform works, and exactly how to set up, target, and optimize a campaign, using OpenAI's own documentation and real setup data from early advertisers. What ChatGPT Ads Are ChatGPT Ads are sponsored placements that appear beneath an AI-generated answer inside ChatGPT, visible to logged-in adult users on the Free and ChatGPT Go tiers. They are clearly labeled "Sponsored" and visually separated from the organic response itself, and only one ad appears per conversation. Ads Manager is still a beta product, though it is fully self-serve, has no minimum spend, and is live in more than 40 countries. Features, eligible categories, and available markets are still being added on a rolling basis, so it is worth treating as a maturing platform rather than a finished one. Paid ChatGPT subscribers on Plus, Pro, Business, Enterprise, and Education tiers do not see ads at all, and OpenAI also excludes accounts identified as belonging to users under 18. That exclusion has held constant since the earliest pilot and is one of the few fixed points in an otherwise fast-changing rollout. How ChatGPT Ads Work The system reads the current conversation for commercial intent and matches it against the ad's landing page, title, copy, and advertiser-provided context hints, natural-language descriptions of the situations an ad is relevant to, rather than exact keyword matches. Past conversations only enter targeting if a user has personalization turned on. When enabled, past chats and saved memory add a secondary relevance layer on top of the current conversation. When turned off, targeting relies solely on the current session. How to Run ChatGPT Ads Step by Step Step 1: Create Your Ads Manager Account Go to ads.openai.com and register your business. You will need your legal company name exactly as it appears on your website, a website URL, an industry category, and a square logo of at least 256 by 256 pixels. Registration covers three parts: business details, account details (country, currency, timezone), and identity verification through a third-party provider. Verification can clear the same day or take longer depending on industry, so it is worth starting early. Agencies can manage ChatGPT Ads on behalf of a client, but the account must be created under the client's own business identity, not the agency's. The agency is added afterward as a user with role-based permissions. There is currently no agency-level account structure the way Google and Meta have built. Step 2: Set Up Billing and Payment Add a payment method (major cards are supported) and a billing address. Optionally set a monthly spend cap as a safety net; it pauses all campaigns automatically once reached. There is no minimum budget required. Self-serve access removed the original 200,000 dollar pilot commitment entirely as of May 5, 2026. OpenAI recommends a starting CPC bid of 3 to 5 dollars, and daily budgets can go well below that in newly opened markets such as India. Step 3: Create Your Campaign Give the campaign a clear, descriptive name (a format like Brand-Objective-Audience-Date works well) and set your schedule and country targeting. Targeting currently stops at the country level; there is no city or regional layer yet. Step 4: Choose the Right Campaign Objective You'll choose between Reach (billed on CPM, best for awareness), Clicks (billed on CPC, best for traffic and lead generation), and Conversions (optimized CPC against a defined action, once a measurement pixel is connected). Campaign objective is locked once launched, one of the few permanently fixed decisions in the setup flow. A different objective later means creating a new campaign rather than editing the existing one. Step 5: Create Ad Groups and Set Context Hints Context hints are free-text descriptions that tell the system which conversations your ad belongs to. OpenAI describes them as hints, not exact-match keywords, and they do not guarantee delivery in any specific conversation. This is still the single most important input in the entire campaign. Keyword-list hints don't work well. A hint written as "CRM, best CRM, CRM software" gets interpreted as a literal sentence, which produces poor matches. The strongest hints follow a four-part structure: who is having the conversation, what they're trying to accomplish, what narrows the match, and what should be excluded. For example, "A founder or team lead at a company under 20 people comparing CRM tools that integrate with Slack and offer a free tier. Not enterprise procurement" consistently outperforms a keyword-style equivalent. Most well-performing ad groups use somewhere between 5 and 15 hints, each covering a distinct angle on the same theme rather than repeating one idea in different words. Step 6: Create Your Ads The ad unit is a single card with six fields: advertiser name, favicon, title, copy, landing page, and one image. There is no second headline, no carousel, and no video in self-serve. The table below summarizes the working creative limits. Field Recommended limit Title (headline) 16 to 24 characters (visually truncates near 24) Copy (description) 32 to 48 characters (visually truncates near 48) Image Square, 256×256 px minimum, 512×512 px recommended, PNG or JPG Favicon Small square logo, roughly 128×128 px Ad variations per group 3 to 5, minimum A note on the character limits: many older guides still publish looser figures such as 30/60 or 50/100 characters, left over from the pilot phase. Writing to those looser limits risks a rejection at review, so it's safer to write tight, then confirm the exact figures against the live Ads Manager before publishing. A single ad per group gives no way to isolate whether underperformance is a targeting problem or a creative problem, which is why the 3 to 5 variation minimum matters. Step 7: Submit for Review Most ads clear review within a few hours; ads in regulated categories like finance or healthcare can take longer. The most common causes of rejection are a mismatch between the account's legal business name and the website, a landing page that blocks OpenAI's crawlers (OAI-AdsBot and, ideally, OAI-SearchBot), or copy that trips OpenAI's advertising policy around misleading claims. The status field inside Ads Manager names the specific reason on hover. Step 8: Launch and Monitor Performance Once approved, status moves from "In Review" to "Active" and the dashboard begins reporting impressions, clicks, spend, CTR, average CPC, average CPM, and conversions in real time. CTR benchmarks break down as follows: CTR range What it signals Below 0.3% Context hint or creative mismatch 0.3% to 0.5% Acceptable, but unproven 0.5% to 1.0% Targeting and creative are aligned Above 1.0% Top-quartile performance Step 9: Optimize Based on Early Data When to pause, adjust, or scale a campaign Pause an ad group once it has spent more than twice your target cost per acquisition with zero conversions, or once CTR sits below 0.1 percent after 10,000-plus impressions. Adjust, rather than pause, when CTR sits in the 0.2 to 0.5 percent range. Change the fastest-to-test elements first: title and copy before the image, and the context hint last. Scale only once an ad group holds 0.5 percent CTR or better with positive return for at least seven consecutive days, and increase budget in 20 to 30 percent steps rather than doubling, since large jumps can destabilize the platform's own optimization. How Targeting Works in ChatGPT Ads Targeting stops at the country level, so comparing performance across regions currently means running separate campaigns rather than targeting a specific city or region directly. Negative context hints are supported, capped at 25 per campaign, useful for excluding conversations that are technically related but commercially irrelevant. Product Feed and Catalog Targeting for E-commerce Businesses selling physical products can skip individual ad creation and run a product-feed campaign instead, reusing the same catalog fields (title, price, availability, image, product URL) already maintained for Google Shopping or Meta catalog ads, mapped into OpenAI's own feed schema. Product titles and images pull directly from the feed, so it's worth checking generated previews before submission. Who You Can Target with ChatGPT Ads Ads are shown only to logged-in adults on the Free and ChatGPT Go tiers. Plus, Pro, Business, Enterprise, and Education subscribers, and any account identified as belonging to a user under 18, are excluded entirely. Custom audience lists can be uploaded to include or exclude specific groups, though they currently require a minimum of roughly 25,000 matched users, which puts this option out of reach for most small and mid-size advertisers at launch. Certain business categories also cannot advertise at all right now, including alcohol, gambling, weapons, adult content, and parts of financial services and healthcare that require additional documentation. OpenAI has also said sensitive conversation topics, such as mental health or politics, are excluded from ad placement entirely, regardless of category. How to Create ChatGPT Ads Creative can't be reused directly from other platforms. Google gives 30 characters across up to 15 rotating headlines; Meta allows a 125-character primary text block on top of longer supporting copy. ChatGPT's card gives a title of roughly 16 to 24 characters and copy of roughly 32 to 48 characters, with nothing else to lean on. Copy written to depend on that extra space reads as fragmented here. The landing page must be accessible to OpenAI's crawlers and not blocked in robots.txt. A technically inaccessible page can quietly suppress ad delivery with no clear error message in the dashboard, since aggressive bot management tools such as Cloudflare or Akamai can block the crawler at the edge before it ever reaches the page. The advertisers seeing the best early results write for the format the ad actually sits in: directly beneath an AI-generated answer, not in a scrolling feed. A specific, concrete claim ("Cut reporting from three days to three hours") consistently outperforms a generic slogan in that context. How Much ChatGPT Ads Cost Pricing runs through a relevance-weighted, second-price auction, so a stronger, more relevant ad can win placement at a lower bid than a generic one. OpenAI's own guidance recommends starting CPC bids of 3 to 5 dollars, with CPM typically running in the 25 to 60 dollar range depending on category. ChatGPT Ads Google Ads Entry CPC Roughly $3 to $5 Varies widely by keyword competitiveness Targeting model Semantic context hints Exact and broad match keywords Minimum spend None None Auction maturity Early, fewer advertisers competing Mature, highly competitive in most categories Direct cost comparisons are still noisy this early in the platform's life, since fewer advertisers are bidding on ChatGPT than on Google, which naturally keeps prices lower for now. That gap is expected to close as more advertisers move budget into the channel. Minimum daily budgets are as low as 25 dollars in the US and, following India's rollout in late August 2026, as low as 725 rupees there. How to Measure ChatGPT Ads Performance The OpenAI Pixel is a lightweight JavaScript tag placed on your website to track post-click events like purchases, form submissions, and signups. It is not mandatory. Campaigns run fine without it, but you'll only see impressions, click, and spend data, not what happened after the click. A server-side Conversions API is also available for advertisers who prefer not to rely on browser-based tracking. Tracking conversions requires installing the pixel or configuring the Conversions API before launch, and appending UTM parameters to every landing page URL (utm_source=chatgpt, utm_medium=cpc or cpm, utm_campaign=your-campaign-name) so traffic is attributable independently of OpenAI's own reporting. ChatGPT Ads works with Google Analytics through UTM-tagged URLs, though there is no native GA4 integration, so consistent UTM structure across every ad is what makes cross-platform comparison possible. Seeing which conversations triggered a given ad isn't currently possible. This is a black box: you get aggregate performance metrics, not the underlying conversation content, which is exactly why independent UTM tracking matters so much here. ChatGPT Ads in India: What's Different Right Now Ads began appearing to eligible Free and Go tier users in India on August 27 to 29, 2026, launching with roughly 50 brands and agency partners WPP and Omnicom. OpenAI has said it expects about one in every five ChatGPT queries in India to show an ad, while roughly 80 percent of queries with no commercial intent will show none at all, even on the free tier. Self-serve access through Ads Manager for Indian advertisers opened on September 4, 2026, with campaigns starting from a minimum daily budget of 725 rupees, about 8 dollars. Before that date, Indian businesses could only buy ChatGPT ads through OpenAI's sales team or an agency partner, so it is worth confirming current self-serve availability on OpenAI's Ads Manager page before assuming a new account can launch immediately. The same category restrictions apply globally: consumer goods, lifestyle and household products, local services, travel, digital products, and education are among the current approved categories for Indian businesses, with regulated sectors like finance and healthcare requiring additional review. Given how recently India opened, expect eligible categories and self-serve depth to keep expanding over the next few months. Global Rollout Timeline: Where ChatGPT Ads Are Live ChatGPT Ads has moved fast enough that a country list from even six weeks ago is already out of date. Here's the dated sequence, drawn from OpenAI's own posts and confirmed press coverage: Date Milestone February 2026 ChatGPT Ads launches as an invite-only US pilot, gated behind a $200,000 minimum spend April 7, 2026 Self-serve Ads Manager enters closed testing May 5, 2026 Self-serve Ads Manager goes live for all US businesses — CPC bidding, no minimum spend May 7, 2026 OpenAI announces upcoming expansion to the UK, Mexico, Brazil, Japan, and South Korea June 5, 2026 Conversion-optimized (oCPC) bidding rolls out June 6, 2026 UK goes live — first European market, via partners rather than self-serve June 22, 2026 Japan and South Korea go live; self-serve access expands to the UK August 18, 2026 OpenAI announces expansion to 31 European countries August 24, 2026 Ads begin serving across the 31 European markets — initially through OpenAI's Ads Solutions team and agency/technology partners only August 27, 2026 Ads begin appearing to users in India on the Free and Go tiers, launching with 50 brands through agency partners WPP and Omnicom August 31, 2026 OpenAI announces a $1 billion annualized revenue run rate and opens self-serve Ads Manager access across Europe, India, the Middle East, and North Africa September 4, 2026 Self-serve Ads Manager access is specifically confirmed live for Indian advertisers, with a ₹725 (~$7.60–8.30) daily minimum Two things stand out in that table. First, "ads are live" and "self-serve buying is live" are two different milestones in every new market. India is the clearest recent example — ads started showing to Indian users nearly a week before Indian businesses could open their own Ads Manager account and run a self-service campaign. Second, OpenAI's own count puts the platform at "more than 40 countries" as of the August 31 announcement. Some third-party ad-industry trackers cite higher, more granular figures as additional markets activate, but OpenAI's official language is the number worth quoting until they update it. Mistakes to Avoid When Running ChatGPT Ads The most common reasons an ad fails to serve: A bid set below the recommended floor, often below roughly 3 dollars per click An unverified or incomplete billing setup A landing page blocked to OpenAI's crawlers, whether through robots.txt, a WAF, or aggressive bot protection The status field inside Ads Manager names the specific reason on hover, so check there before adjusting the bid. Reusing the same ad copy across multiple ad groups is possible, but it defeats the purpose of ad group segmentation. Each ad group should represent a distinct buyer situation with copy written to that situation specifically; reusing one generic ad across groups makes it impossible to tell which targeting is actually working. Other recurring mistakes worth naming directly: writing context hints as keyword lists instead of natural-language descriptions, skipping UTM setup before launch and trying to reconstruct attribution afterward, and judging a campaign's performance in the first few days before it has cleared a meaningful learning period. Conclusion ChatGPT Ads vs. AI Search Optimization: Run One or Both? Paid placement and organic AI visibility solve different problems. ChatGPT Ads earn a labeled placement beneath an answer. AI Search Optimization work, sometimes called Generative Engine Optimization, earns your brand an actual mention inside the answer itself, the kind of structured, entity-clear content covered under our AI SEO services . A brand appearing in both places within the same conversation builds a level of trust neither channel produces alone, the same reinforcing relationship organic SEO and paid search have had for two decades. For most businesses testing this channel for the first time, the sequence that works is starting with a contained budget, a genuinely differentiated set of context hints, and a landing page built specifically for the offer, then judging results against the benchmarks in this guide rather than against assumptions carried over from Google or Meta. If your website or landing pages need work before they're ready to take ChatGPT Ads traffic, that's worth fixing first; see our web design and development services or browse recent case studies for examples of what a conversion-ready setup looks like. For help planning or running a ChatGPT Ads pilot alongside your existing SEO work, get in touch with our team .

13 minsRead Article

Crawlability vs Indexability: What Is the Difference in SEO?

What Crawlability Actually Means

What Indexability Actually Means

The Difference, in One Table

A Page Can Be One Without Being the Other

What Actually Controls Crawlability

What Actually Controls Indexability

How to Tell Which One Is Actually Broken

Why the Distinction Actually Matters to a Business

AI Crawlers Have Added a Third Layer

The Data Behind This

Common Mistakes

Frequently Asked Questions

What is the main difference between crawlability and indexability?

Can a page be crawlable but not indexable?

Can a page be indexed without being crawled at all?

Does blocking a page in robots.txt guarantee it won't appear in Google?

How do I check whether a page's problem is crawlability or indexability?

Does JavaScript affect crawlability?

What is an orphan page, and why does it matter for crawlability?

Do AI crawlers follow the same rules as Googlebot?

Should I block all AI crawlers to protect my content?

What tools can I use to check crawlability and indexability separately?

Technox Technologies Team

Related Blogs

The SEO & AI Visibility Checklist for 2026

How to Optimize Title Tags & Meta Descriptions for Better SEO in 2026

How to Run ChatGPT Ads: Complete Step-by-Step Guide

	Crawlability	Indexability
What it governs	Whether a crawler can reach and read the page	Whether Google stores the page in its index
Happens	Before content is evaluated	After the page has been crawled
Controlled by	Internal links, sitemaps, robots.txt, server responses, JavaScript rendering	Noindex tags, canonical tags, duplicate or thin content, quality signals
Typical failure	Page is orphaned, blocked, or never rendered properly	Page loads fine but gets excluded or consolidated into another URL
Where to check it	Screaming Frog crawl, server logs, robots.txt tester	Search Console's Page Indexing report, URL Inspection tool

Data point	Source	What it means for your strategy
A robots.txt-disallowed page can still be indexed, without being crawled, if other sites link to it	Google Search Central	Blocking a URL in robots.txt is not the same as guaranteeing it stays out of search results entirely. Use no index, not robots.txt alone, when the goal is full removal.
Google can only reliably crawl links built as proper <a href> elements	Google Search Central	JavaScript-only navigation risks leaving whole sections of a site undiscovered. Test with rendered and raw HTML both.
Crawl budget is set by crawl capacity limit combined with crawl demand	Google Search Central	Slow servers get crawled less thoroughly per visit regardless of how good the content is, delaying indexing for new pages.
GPTBot appears in more robots.txt disallow rules than any other AI crawler as of September 2026	Cloudflare Radar data, via Technology Checker	If AI-search visibility matters to your business, verify GPTBot and search-specific bots aren't accidentally caught by the same blanket rule.
GPTBot blocking has held at roughly 25% of top sites since 2024, and blocking training bots while allowing search bots is now the most common configuration	Presence AI, 2026 research	A blanket "block all AI bots" robots.txt rule is now a minority position, and it forecloses citation in AI Overviews and AI-powered search answers.
Roughly 89% of AI crawler traffic is training-related, not search-related	Presence AI, 2026 research	Most of the AI crawl volume hitting a server produces no referral value, which explains why selective blocking of training bots specifically has become standard practice.