AI crawlers and robots.txt: what to allow, what to block and what to test

Why are AI crawlers a decision site owners now have to make?

More products now fetch web pages to answer questions, summarize content or train models, and each one arrives as a crawler with its own name, purpose and rules. Site owners need to decide which of them to allow, which to restrict and how that affects visibility, server load and content reuse.

What changed in how content is fetched?

For years the main question was how to help Googlebot and Bingbot crawl a site. Today there are search crawlers, crawlers that collect data for model training, crawlers that fetch a page on demand when a user asks an assistant about it and crawlers run by publishers' tools and SEO platforms. Their purposes differ, and so should your response. The principle behind how Googlebot crawls and renders the DOM still applies: a crawler can only use what it can fetch and read.

Is every AI crawler the same?

No. Some fetch pages to build an index that powers search answers, some collect data to train models, and some request a single URL at the moment a person asks a question. Blocking one purpose does not necessarily block the others, and each operator documents its own names and behavior.

What is at stake?

Three things: visibility, control and cost. Allowing a crawler can make your content eligible to be quoted or linked in answers. Restricting it can protect content you license or sell. Heavy crawling can raise server load, which is a practical cost for smaller hosting plans. How slow servers reduce crawling is explained under server response time and crawl frequency.

Who should make the decision?

It is a business decision, not only a technical one. Marketing cares about visibility, legal cares about content rights, and development cares about load. Write the decision down so that the next audit does not reopen it from scratch.

What does Google itself require?

Google's AI features page, last updated December 10, 2025, says there is no special markup for AI Overviews or AI Mode, and that pages need to be indexed and eligible to be shown with a snippet. This is the most important sentence in the debate, because it moves the question back to fundamentals: crawlable, renderable, indexable and useful pages. What site owners can do is covered under AI Overviews and AI Mode.

Does blocking Googlebot remove you from AI features?

Blocking Googlebot from a page can prevent it from being crawled and indexed, and a page that is not indexed cannot be used as a supporting link. That is a blunt tool, and it also removes the page from classic results, so use it only when you truly want that.

What can robots.txt do, and what can it not do?

Robots.txt is a plain text file at the root of a host that tells crawlers which paths they may request. Google's introduction, updated December 10, 2025, says its main use is to manage crawler traffic and avoid overloading a site, and that it is not a mechanism for keeping a page out of Google.

What does it control?

It controls crawling: whether a compliant crawler requests a URL. It can keep crawlers away from low-value or near-duplicate sections, reduce server load and keep some media files out of search results. It works per host and per protocol, so the file on www and the file on a subdomain are separate documents.

Where must the file live?

At the root of the host, for example the address of the site followed by /robots.txt. A file in a subfolder is not read as the host's robots.txt.

What does it not control?

It does not guarantee that a URL stays out of results. A blocked URL can still appear if other sites link to it, usually without a description. To keep a page out of the index, allow it to be crawled and use a noindex rule, or protect it with a password. Combining a robots.txt block with a noindex tag on the same URL defeats the tag, because the crawler never sees it. This conflict for filtered pages is worked through under noindex versus disallow.

Is robots.txt a security tool?

No. Compliance is voluntary, and the file is public. Listing a private path in it advertises the path. Use authentication for anything that must stay private.

Why do crawlers interpret rules differently?

Google's documentation notes that different crawlers can interpret the same syntax differently, and that mixing crawling and indexing instructions can create conflicts. Treat each operator's documentation as the authority for its own crawler and test the effect on the crawlers you care about.

Which syntax mistakes are common?

Blocking the whole site with a stray slash, forgetting that rules are case sensitive in paths, using wildcards a crawler does not support, serving different files to different user agents by accident and publishing a staging robots.txt on the live site.

How do Google's AI features relate to robots.txt, snippets and indexing?

Google's AI features draw on the same index and the same eligibility rules as classic Search. Controls over crawling, indexing and previews therefore decide whether and how your content can appear, and there is no separate switch for AI Overviews or AI Mode.

Which controls does Google point to?

Goal	Control	What it does
Stop Googlebot fetching a path	robots.txt disallow	Prevents crawling, does not guarantee exclusion from the index
Keep a page out of the index	noindex or password protection	Removes the page from results once recrawled
Show no text snippet from a page	nosnippet	Prevents text previews, including use in AI features
Hide part of a page from previews	data-nosnippet	Excludes marked sections from snippets
Limit snippet length	max-snippet	Caps how much text may be shown
Limit use for AI training and grounding in some other Google systems	Google-Extended	Robots.txt token that restricts that use

What is the main trade-off?

Preview limits can reduce how much of your content appears in features that quote or summarize it, which may reduce visibility as well as reuse. Apply them to content you truly want to protect and not site-wide by default.

What does Google say about measurement and recrawling?

Google says that traffic from AI Overviews and AI Mode is included in Search Console's Performance report under the Web search type, and that if content still appears after you add preview controls, you should check the markup with the URL Inspection tool and allow time for recrawling, which can take from several days to several months, or request a recrawl. What you can and cannot measure is explained under measuring SEO in the AI era.

Why might a change not take effect immediately?

Google has to recrawl and reprocess the page. Changes to robots.txt, meta tags or headers are applied only after that happens, so record the date of each change.

What about Google-Extended?

Google-Extended is a robots.txt product token. Google says it can be used to limit AI training and grounding in some of its other systems, and it is separate from Googlebot's crawling for Search. Blocking it does not remove you from Search, and it is not a way to opt out of AI Overviews.

Should you block it?

That depends on your content strategy. Publishers who license content may choose to. Most small business sites have little to protect and may gain more from broad visibility. Make the choice deliberately, and write down why.

How should you treat AI crawlers from other companies?

Treat each crawler according to its operator's published documentation. Names, purposes, IP ranges and opt-out methods differ, change over time and are not governed by Google's rules. A short, reviewed policy per crawler is better than copying a list from a blog post.

What information should you gather for each crawler?

The user agent and robots.txt token the operator documents, what the crawler is used for (search index, model training or on-demand retrieval), whether the operator publishes IP ranges so that you can verify requests, how quickly changes to your robots.txt are honored and whether the operator offers other controls, such as a meta tag or contact route. Where this information is not documented, treat the crawler cautiously.

Where do you find the documentation?

On the operator's own site, usually in a crawler or bot information page. Do not rely on third-party lists as the final word, because they go out of date.

How do you decide to allow or restrict?

Ask three questions. Does the crawler's purpose benefit you, for example by sending referral traffic or citations? Does the content you expose carry value that you license, sell or consider proprietary? Is the crawling rate affecting your server? Allow where benefit is clear and cost is low, restrict where the content is proprietary or the load is a problem, and monitor the result.

Can you allow some sections and block others?

Yes. Robots.txt rules are path-based, so you can allow public guides and block gated areas, internal search results or large archives. Sections that waste crawl effort for any crawler are covered under crawl traps.

How do you verify who is really crawling?

User agent strings can be faked. Google documents how to verify Googlebot with reverse DNS or its published IP ranges, and other operators describe their own methods or publish ranges. Use those methods before drawing conclusions from logs, and before blocking an address range. How to read crawler activity at scale is shown by log file analysis.

What if an unknown bot is heavy?

Rate limit or block it at the server or firewall level. Robots.txt will not stop a crawler that ignores it.

Can AI crawlers see your content at all?

Many crawlers fetch raw HTML and do not run JavaScript, so content that appears only after scripts execute may be invisible to them. Google renders pages, but you should not assume that every crawler does. Delivering key content in the initial HTML is the safest approach.

What should be in the initial HTML?

The main text, headings, internal links, titles, canonical tags, structured data and any facts you want quoted. The risk when content depends on scripts is explained under client-side versus server-side rendering, and this site's static HTML delivers all of it without needing JavaScript.

How do you test what a crawler without JavaScript sees?

Fetch the page with a plain HTTP client such as curl and read the response, or view the source in your browser. If the text you want quoted is missing, that crawler will not see it either.

What else can block access?

Firewalls and bot protection that challenge or block unknown user agents, cookie walls and consent screens that hide content until a click, login walls, geographic blocks, rate limits and long redirect chains. Test as a visitor without cookies and from outside your own network.

Should you allow crawlers through your bot protection?

Only the ones you have decided to allow, and only after verifying them. Blanket allow rules for any user agent that claims to be a known crawler invite abuse.

Does page size matter?

Yes. Googlebot's documentation says it crawls the first 2 MB of a supported file, and other crawlers may have their own limits. Keep HTML lean and place important content early, as Googlebot and the DOM explains.

How do you make your content easy for AI systems to use correctly?

Write pages that state facts plainly, attribute them, keep them current and mark them up consistently. These practices help classic search, AI summaries and human readers alike, and they are within your control, unlike the internal workings of any model.

What does a quotable page look like?

A clear answer near the top, headings phrased as the questions people ask, short paragraphs that stand on their own, tables for comparisons, named authors with credentials, dates and sources, and consistent entity information such as the business name, address and services. Why the same fundamentals apply is explained in GEO versus SEO.

Do you need a special file for AI?

Google says no special files or markup are needed for its AI features. Some site owners publish summary files for AI tools, but treat them as experimental, because no operator has committed to relying on them in the way that crawlers rely on robots.txt.

How does structured data help?

Structured data helps search systems understand entities, products and relationships, but it must match the visible page and does not guarantee any feature. How to keep markup and visible content consistent is explained in feeds and structured data.

Can markup hurt you?

Markup that misrepresents the page can lose rich result eligibility or trigger manual actions. Keep it accurate.

What about originality?

Original data, first-hand experience and clear expertise are what make a page worth citing. Pages that repeat what everyone else says give a system no reason to prefer yours. The policy side of low-value pages is covered under scaled content abuse.

What is a practical decision framework for allowing or blocking?

Sort your content into three groups, public and useful, public but proprietary and private, then match a policy to each. The framework turns a vague worry into concrete rules in robots.txt, headers and server settings.

How do the three groups work?

Public and useful content, such as guides, service pages and product information, is what you want found, quoted and linked, so allow it for crawlers whose purpose benefits you. Public but proprietary content, such as paid research, licensed material or databases, deserves restrictions through robots.txt, preview controls, terms of use and, where necessary, authentication. Private content, such as accounts, carts and internal tools, should sit behind a login and should never rely on robots.txt for protection.

Where does a typical small business land?

Almost everything is public and useful. The main work is to avoid blocking by accident, keep pages fast and make content easy to read, not to build a wall.

Which scenarios come up most?

• A local service business that wants to be recommended: allow broadly, make the facts clear and keep the Business Profile accurate.
• An ecommerce store worried about price scraping: rate limit, monitor logs and consider bot management, accepting that robots.txt will not stop a determined scraper.
• A publisher with licensed content: restrict training use where the operator offers a control, and use contracts for the rest.
• A site with a heavy crawl load: block low-value sections, improve caching and consider crawl rate controls where offered.

How often should you review the policy?

At least twice a year and whenever an operator announces a change. Keep a dated record of every robots.txt edit.

How do you roll out changes safely?

Test the new file with a robots.txt tester or a parser, deploy it, fetch it from the live host to confirm it is the file you expect, then watch Search Console's crawl stats and your logs. If crawling collapses or important pages drop out, roll back at once. A technical SEO audit checklist includes a robots.txt review.

What does a sensible robots.txt look like?

A sensible file is short, specific and commented. It blocks low-value paths, points to the sitemap and treats particular crawlers separately only where there is a reason. The example below is illustrative and must be adapted to your site and to each operator's current documentation.

What does an illustrative file contain?

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Allow: /

# Example: restrict one named AI crawler from a licensed section
User-agent: ExampleAIBot
Disallow: /research/

Sitemap: https://www.example.com/sitemap.xml

The first group applies to all crawlers that do not match a more specific group, and the second applies only to the named bot. The path names and bot name are placeholders. Check the exact token that each operator documents before you write a rule.

What are the traps?

Crawlers follow the most specific matching group, so a named group replaces the general one for that crawler instead of adding to it. If you add a group for one bot, repeat any general rules you still want it to follow.

How do you test the file?

Use Search Console's robots.txt report to see how Google fetched and parsed the file, test individual URLs with the URL Inspection tool and use a parser library to check other crawlers' interpretation. Keep the file under version control so that changes are reviewed.

Can you use the file to hide pages?

No. Do not list private paths, because the file is public. Use authentication and noindex as appropriate.

How do you monitor crawler activity and the effect of your choices?

Use server logs, Search Console crawl stats and referral reports to see who is crawling, how often and whether any of it sends visitors. Monitoring turns an opinion about AI crawlers into evidence you can act on.

What should you look at?

Requests per user agent and per day, response codes, bytes served, the most requested paths, the share of requests from verified addresses and referral visits from AI products where they show up in analytics. Compare periods before and after a change.

Can you see AI referrals?

Some assistants pass a referrer, and some do not, so analytics will undercount. Treat the numbers as a floor and not a total. This limit is a recurring theme in measuring SEO in the AI era.

What thresholds should trigger action?

A sustained rise in requests that slows pages, many requests to paths that exist only in a crawl trap, requests from addresses that fail verification and heavy crawling that sends no benefit. Each is a reason to adjust rules, caching or server protections.

Who owns the follow-up?

Assign one person, usually in development or SEO, to review the logs monthly and report anything unusual.

How do you keep records?

Keep a simple change log: date, file or setting changed, reason, person responsible and what you observed afterwards. The log answers the question that always comes up later, which is why a particular crawler was blocked.

What about assistants that fetch a page on demand?

Some assistants fetch a URL at the moment a person asks about it, acting on the person's behalf. Operators treat these requests differently from background crawling, and their documentation says whether robots.txt is honored for them. Read it before you assume that a block applies.

Why does the difference matter?

A background crawler builds a corpus over time, while an on-demand fetcher retrieves a single page for a single conversation. If you block the second kind, a user who pastes your link into an assistant may get an error instead of a summary, which can look like a broken site to that person. If you allow it, the assistant can quote the page accurately and may link to it.

How do you find out how an operator treats these requests?

Look for the operator's crawler page and check whether it lists separate user agents for training, search indexing and user-initiated fetches. Where the operator lists them separately, you can set a different rule for each.

What should you do about paywalled and gated pages?

If content is behind a login or paywall, make sure the gate is enforced by the server and not by a script or a robots.txt line. Content that is visible in the raw HTML to any fetcher is effectively public, whatever a front-end overlay does. Google's guidance on paywalled content describes structured data that tells it which parts are gated, which helps avoid being treated as cloaking.

Does a login wall protect against crawlers?

Yes, for compliant and non-compliant crawlers alike, because they cannot pass the login. It is the only dependable protection for private material.

What does a decision look like in practice?

This is a fictional example that illustrates the method. The outcomes are invented.

What was the situation?

A fictional regional tile retailer noticed heavy bot traffic slowing its category pages and asked whether to block all AI crawlers. The site is a mix of public guides, product pages and a paid trade price list.

What did the review find?

Log analysis showed that most of the load came from one unverified bot hitting filtered URLs, that Googlebot and two documented AI crawlers behaved normally, that the paid price list sat in a public folder with no login and that the robots.txt file was a copy from an old project that blocked the CSS folder.

What did they change?

They removed the CSS block so that rendering worked, disallowed filtered URLs and internal search for all crawlers, rate limited the unverified bot at the firewall, put the trade price list behind a login, kept guides and products open to documented crawlers and wrote down the policy and review date. They monitored crawl stats and server load for a month, noting that visibility effects could not be guaranteed and would be hard to isolate.

What was the lesson?

The problem was never "AI crawlers" as a category. It was an old file, an exposed price list and one aggressive bot, all of which were fixed with specific rules.

Which mistakes should you avoid?

The most common mistakes are blocking everything out of fear, copying someone else's file, relying on robots.txt for privacy and forgetting to test. Each can cost visibility or expose content.

What do these mistakes look like?

Blocking all unknown bots, including search crawlers, with a blanket rule. Disallowing CSS and JavaScript so that pages cannot be rendered. Adding noindex to a URL that robots.txt blocks. Believing that a block removes a page from results. Leaving a staging file in production. Ignoring server load until a crawler causes an outage.

Which one is most expensive?

An accidental site-wide disallow on the live site. Add a check to your release process that fetches robots.txt and compares it with the expected file.

Why do copied files cause trouble?

A file copied from a template or another site carries rules written for a different structure, and nobody on your team knows why each line exists. Rules that blocked a login area on someone else's platform may block your product images or your whole blog on yours. Read every line, delete what you cannot justify and keep a comment next to each rule that explains its purpose.

How do you avoid them?

Keep robots.txt in version control, review changes, test before and after deployment, monitor crawl stats and write down decisions. Our technical SEO service includes these checks.

How does AH SEO Agency help with crawler policy?

We review your robots.txt, headers, rendering and server logs, classify your content, recommend a policy per crawler and document it, and we test changes before and after release. We follow the operators' own documentation and say plainly where something is unknown.

Where do you start?

Explore SEO audit support, learn about AI Overviews and AI Mode or request a free SEO audit.

Summary: how should you handle AI crawlers?

Keep your pages crawlable, renderable and indexable, use the right control for each goal, decide per crawler with evidence and review the policy regularly.

• Robots.txt manages crawling and is not a privacy or indexing tool.
• Google's AI features need no special markup, only indexed, snippet-eligible pages.
• nosnippet, data-nosnippet, max-snippet and noindex limit previews, and Google-Extended limits some AI uses.
• Other operators set their own rules, so read their documentation.
• Test, monitor and record every change.