Identifying crawl traps: how infinite filters waste search engine resources

What a crawl trap is

A crawl trap is a part of a website that generates an endless, or practically endless, supply of URLs, so that a crawler that enters it can keep fetching pages without ever reaching the end. Google's own documentation calls the underlying problem infinite spaces, and lists it among the main sources of low-value URLs that can waste a site's crawling, together with faceted navigation, session identifiers, duplicate content and soft error pages.

The trap does not have to be deliberate or even visible to visitors. A shopper never meets a million filter combinations. They meet the three they click. But a crawler that follows every link it finds meets all of them, because each page it fetches contains links to more combinations. For ecommerce stores, filters are the most common cause, though calendars, search boxes, relative links and tracking parameters can all do the same thing.

This article explains how crawl traps arise on online stores, how to find them with tools you probably already have, how to close them and how to stop new ones from forming. It is part of our coverage of faceted navigation SEO and works with the articles on noindex versus disallow and clean versus parameterized URLs.

Why crawl traps matter

The effect depends on the size of your site. Google has said that crawl budget, the amount of crawling it will devote to a site, is mainly a concern for very large sites or for sites with many frequently changing pages. A small store may never feel it. But traps cause trouble beyond the budget, and several harms apply to stores of any size.

• Delayed discovery and refresh. When crawlers spend their attention on endless filter pages, new products and updated prices are found and refreshed more slowly.
• Server load. Each request costs your server something. A crawler that requests tens of thousands of expensive filtered pages can slow the site for real shoppers.
• Index clutter. Some trap pages get indexed, creating thousands of thin, duplicate results, as covered in our article on Shopify tags and indexation bloat.
• Diluted internal links. Links to trap pages take a share of the signals that should go to your key categories.
• Noisy data. Reports fill with errors and exclusions that hide real problems.

Traps rarely announce themselves. The usual first symptom is that someone looks at a crawl report or the Pages report and sees numbers that make no sense for the size of the catalog.

The common causes on ecommerce sites

Filter combinations

The most common cause. Each filter adds a parameter or a path segment, each page links to the other options and the combinations multiply. Add sort orders, view modes and pagination, and the space is effectively infinite.

Filters that can be added repeatedly

Some systems allow the same filter to be appended again and again, producing URLs such as ?color=red&color=red&color=red. Others let a path segment repeat, such as /sofas/leather/leather/leather/. These loops can produce unlimited URLs from a single page.

Relative link errors

A link written as a relative path without a leading slash can append itself to the current path. On /shop/sofas/, a link to shop/chairs/ becomes /shop/sofas/shop/chairs/, and each of those pages produces another nested variant. A badly written template can create an endless tree of nested addresses.

Calendars and date-based pages

Event, booking and archive pages that link to "next month" can be followed forever into the future. Stores with delivery date pickers or event calendars sometimes expose these to crawlers.

Internal search

Search results pages accept any query, and links such as "related searches" can lead to more queries. Without a block, they offer an infinite supply of pages.

Session and tracking parameters

If a platform adds a session ID to every link for visitors without cookies, each crawl visit sees a different set of addresses for the same pages. Tracking parameters from campaigns can create a copy of the site for every campaign code.

Infinite scroll without an end

A listing that loads more products as the user scrolls, backed by URLs that increment a page number with no upper limit, can lead crawlers to request page 5,000 of a category that only has 40 pages, each returning an empty page with a normal status.

Faulty redirects

Redirect loops and chains, for example a rule that adds a trailing slash and another that removes it, send a crawler in circles. They waste effort and may stop the crawl of that path altogether.

Empty results with a normal status

If a combination with no matching products returns a normal page, every impossible combination becomes a valid-looking URL, which expands the space of crawlable addresses.

Warning signs that you have one

• A crawl of your site finds many times more URLs than you have products and categories, and the count keeps rising the longer the crawl runs.
• Search Console's Pages report shows tens or hundreds of thousands of URLs in categories such as "Discovered, currently not indexed", "Crawled, currently not indexed" or "Duplicate" for patterns that include parameters.
• The Crawl stats report shows a large share of requests going to parameterized or unusual URLs, and few to products.
• Server logs show Googlebot requesting very deep or very long URLs, or the same page with endless variations.
• URLs in your crawl or logs are very long, with repeating segments or parameters.
• Server response times worsen during crawl spikes.
• New products take a long time to be indexed, even though the site seems healthy otherwise.

None of these proves a trap by itself, but together they point to where to look.

How to find crawl traps, step by step

1. Configure a crawler carefully

Run a site crawler such as Screaming Frog, Sitebulb or a similar tool, but set limits first: a maximum number of URLs and a maximum crawl depth, so that the crawl itself does not become a trap. Respect robots.txt for the first run, to see what a normal crawler sees, and run a second pass ignoring it to see what it hides.

2. Group URLs by pattern

Export the list and group URLs by their path structure and by parameter names. A pivot table in a spreadsheet, or a short script, will show the number of URLs for each pattern. The trap is usually the pattern with the highest count that does not correspond to real pages: for example, 80,000 URLs with a ?filter parameter against 3,000 product pages.

3. Look at depth and length

Sort by crawl depth and by URL length. A healthy store has most pages within a few clicks of the home page. A cluster of pages 12 or 20 clicks deep, or URLs several hundred characters long, usually results from combinations stacking on top of each other.

4. Look for repeating segments

Search the URL list for repeated path segments or repeated parameters. A simple pattern search for the same word appearing twice in a path often exposes the relative link errors and looping filters quickly.

5. Check Search Console

The Pages report shows how many URLs of each kind Google knows about. The Crawl stats report, under Settings, shows crawl requests by response, by file type, by purpose and by Googlebot type, along with host status. Rising requests for a pattern, or a heavy share of "discovery" requests for unfamiliar URLs, is informative.

6. Analyze server logs

Logs are the ground truth. Filter for verified Googlebot requests, because many bots pretend to be Google, then group by URL pattern and count. Compare the share of requests going to patterns that matter with those going to noise. Look at the status codes: a large number of 200 responses for impossible combinations is a strong sign, as is a large number of 404 responses for nested paths.

7. Test by hand

Pick a category page and follow the links yourself. Click filters, then click the filters on the resulting page. See whether you can reach a combination that returns nothing, or a page that offers links to yet more combinations. Add a parameter twice. Change the order. Add an impossible value. If the site happily returns a page each time, the space is open.

Reading the Crawl stats report

Search Console's Crawl stats report, found under Settings, is the quickest way to see how Google crawls your site without touching a log file. It is available for Domain properties and for URL-prefix properties at the root of a host, and it covers roughly the last ninety days.

• Total crawl requests. A line that climbs sharply without any change to your catalog suggests that Google has found a new source of URLs, such as a new filter or a broken template.
• Total download size and average response time. Rising response times during heavy crawling mean the server is struggling, which is itself a reason to act.
• By response. A high share of 404 responses points to nested relative-link errors or removed pages that are still linked. A high share of 301 responses points to redirect problems or links that use an outdated form.
• By file type. A large share of requests for HTML is normal, and a surprising share for other types may show that crawlers are spending effort on scripts, images or feeds.
• By purpose. The split between discovery, finding new URLs, and refresh, rechecking known ones, is informative. A very high share of discovery in a site that is not growing is a classic trap signature: Google keeps finding new URLs because the site keeps making them.
• By Googlebot type. Smartphone and desktop crawlers behave differently, and a problem that appears for only one may come from a device-specific template.
• Host status. This shows whether Google has had trouble reaching your server or reading your robots.txt file, which can reduce crawling.

Click into the examples under each breakdown. The sample URLs they list show the patterns that the crawler is seeing, which is often enough to identify a trap in a few minutes. The report does not show every request, so treat it as a guide and confirm with logs where precision matters.

A simple log analysis recipe

You do not need an expensive tool to learn something useful from logs. If your server records requests in a common format, a few commands will give you a ranked list of the URL patterns that Googlebot requests most. The example below is for a Unix-style shell and a standard access log, and it should be adapted to your format.

# requests from Googlebot, grouped by the first path segment and by whether a query string is present
grep "Googlebot" access.log \
  | awk '{print $7}' \
  | sed -E 's/\?.*/?PARAMS/' \
  | awk -F/ '{print "/" $2 "/" $3}' \
  | sort | uniq -c | sort -rn | head -30

The output lists the thirty busiest patterns. Compare the counts with the number of real pages in each area. If a filter path is taking 70 percent of requests and brings in no revenue, you have found your trap.

Three cautions apply. First, anyone can claim to be Googlebot in a request, so verify important findings against Google's published IP ranges or with a reverse DNS lookup. Second, logs from a content delivery network may record requests that never reached your origin, and the numbers can differ between layers. Third, handle logs responsibly, as they contain visitor IP addresses and other data that may be subject to privacy rules.

If you cannot get logs, your hosting provider or developer may be able to export a sample for you, and some CDNs offer their own analytics. Even a day of data is enough to identify a dominant pattern.

A cheat sheet of trap patterns to search for

When you have a list of URLs, a few simple searches turn up the usual suspects. These are descriptions of patterns to look for, which you can express in your crawler's filters or in a spreadsheet.

• A parameter name that appears twice in the same query string, such as color= appearing two times.
• A path segment that repeats two or more times in a row, or that repeats a section name.
• URLs with a very large number of parameters, for example more than four.
• URLs longer than a few hundred characters, which often come from stacked filters or nested paths.
• Parameters with names like sessionid, sid, token, ref or utm, which should not appear in crawlable links.
• Paths containing words such as calendar, archive, search, compare or print that are followed by long chains of dates or identifiers.
• Page numbers far beyond the real last page, such as page=900 on a category with 20 pages.
• The same page title on thousands of URLs, which indicates many addresses for the same content.
• Pages that say "no products found" or that show a very small number of products, returned with a 200 status.

Combine these checks with the pattern grouping from the previous step. A family that matches several of them at once is almost certainly a trap.

Explaining the problem and the fix to the business

Crawl traps are technical, and the people who must approve the work are often not. A short explanation, built around outcomes, helps.

• The problem in one sentence: "Our filters create millions of pages that nobody searches for, and Google is spending its time on them instead of on our products."
• The risk: slower discovery of new products, a slower and noisier site and wasted engineering and analytics effort.
• The evidence: a chart of crawl requests by pattern, and a count of known URLs against the number of real pages.
• The fix and its cost: a small set of template and logic changes, with a clear estimate of developer time.
• The result to expect: a shorter list of URLs, a shift of crawling toward revenue pages and a clean set of reports, over weeks to a few months. Avoid promising a ranking jump, because the effect on rankings is indirect and hard to isolate.

Offering a way to measure success also helps: the share of crawl requests that go to products and categories, the number of known URLs, the count of indexed filter pages and the time it takes for a new product to appear in the index. These are concrete, and they make the progress visible.

Classify what you find

Not every large URL family is a trap, and not every trap needs the same cure. Classify each family with a short set of questions.

Closing the trap: fixes by cause

The best fixes remove the source. Stopping a template from generating links to endless combinations is cleaner and more durable than trying to hide the combinations afterwards, because it deals with the cause instead of the symptoms. Where the source cannot be changed quickly, use the controls from our noindex versus disallow article, in the correct order.

When a trap is hurting the server right now

Sometimes the problem is urgent: a crawler is hitting an endless space so hard that the site is slowing down. In that case, speed matters.

1. Confirm the pattern from the logs, and make sure the traffic is Googlebot and not another bot or an attack.
2. Block the pattern in robots.txt as an immediate measure, if the URLs are not indexed. This stops compliant crawlers quickly. If they are indexed, accept that the block will freeze their state in the index for the moment, and plan the longer clean-up afterwards.
3. Consider temporary responses that tell crawlers to slow down. Google's documentation says that returning 500, 503 or 429 responses for a short period can make it reduce crawling. This is a blunt instrument that should not be left in place for long, since extended errors can lead to pages being dropped.
4. Fix the source in the templates or application logic.
5. Review after a few days in the logs and the Crawl stats report to confirm that the pattern has stopped.

Afterwards, record what happened and add a check to prevent a repeat.

Preventing new traps

Traps often arrive with a new feature, a new filter app or a theme update. A few habits keep them out.

• Cap the combinations. Decide the maximum number of filters that can be applied at once and, in the interface, stop offering more. Two or three is common.
• Validate input. Reject unknown parameters and impossible values, return a 404 and do not echo input into links.
• Use a whitelist. Allow indexable facet pages only from an approved list, as described in our article on clean versus parameterized URLs.
• Test every release. Add a crawl to your release checklist: run a limited crawl on staging and compare the URL counts by pattern with the previous release. A sudden jump is a warning.
• Monitor continuously. Review Crawl stats and the Pages report monthly, and set an alert for unusual growth in the number of known URLs.
• Review third-party tools. Search, filtering, calendar and personalization tools can create URLs of their own. Ask vendors how they handle crawlers.
• Write it down. Keep a short document that states the rules, so that developers and agencies know them before they build.

Telling a trap from legitimate depth

A large number of URLs is not by itself a problem. A catalog with 200,000 real products should have hundreds of thousands of crawlable URLs, and crawlers should spend their time on them. Before you declare a trap, check that the URLs are not simply a big catalog doing its job.

• Does the URL count track real inventory? If products, categories and deliberate landing pages explain the number, there is no trap.
• Does each URL have a purpose? A page that a shopper might genuinely land on, with distinct content, is not noise.
• Does the number keep growing during a crawl? A trap is open-ended, so the count rises as long as you keep crawling. A real catalog levels off.
• Do new URLs come from following links on existing trap pages? That self-feeding behavior is the signature of an infinite space.

The aim is not to minimize the number of URLs. It is to make sure that the URLs crawlers meet are the ones you want them to meet, and that the rest are controlled. Being careful here avoids the opposite error of blocking pages that matter, which can be more costly than leaving a modest amount of noise alone.

A note on JavaScript filters and traps

Filters that run entirely in the browser, without changing the URL or producing crawlable links, cannot create a crawl trap, because crawlers have nothing to follow. That is one reason some stores choose them. The trade-off is that filtered views then have no address of their own, so none of them can rank, and content that loads only after interaction may not be seen at all. A balanced approach is to use script-driven filters for convenience and crawlable pages for the combinations you want indexed. We explore the details in managing JavaScript-based filters.

A worked example: an outdoor store with a repeating filter

A fictional outdoor equipment store has 6,000 products. A monthly audit shows that Search Console has discovered more than 2 million URLs. The Crawl stats report shows that most requests go to a single category path with parameters.

The team crawls the category with a limit of 50,000 URLs and groups the results. About 90 percent contain a parameter called brand, many with the value repeated, such as ?brand=acme&brand=acme&brand=zenith. The filter code appends the chosen brand to the existing query string without checking whether it is already present, and each page links to every brand, so the space has no end. Logs confirm that Googlebot is following these links, and that the server returns a 200 status for every variant.

The fix has three parts. The developers change the filter logic so that a parameter can appear only once, and an attempt to repeat it redirects to the normalized form. The category template stops linking to combinations with more than two brands. And invalid combinations, such as brands that have no products in the category, now return a 404. In parallel, the SEO team adds noindex to the crawlable duplicates, waits for the index to settle and then blocks the remaining multi-brand patterns in robots.txt.

Over the following weeks, the number of known URLs falls steadily, the crawl stats show requests shifting toward product and category pages, new products are discovered faster and the audit that once reported millions of URLs reports tens of thousands. The team adds a crawl check to its release process, and the next theme update is caught in staging when it tries to reintroduce a similar link pattern.

Common mistakes

1. Blocking in robots.txt before the index is clean. Indexed URLs then linger without descriptions.
2. Treating a symptom. Hiding the URLs without fixing the code that generates them lets the problem return.
3. Running an unlimited crawl. Your own crawler can fall into the trap and never finish.
4. Ignoring the logs. Reports sample, while logs show what really happens.
5. Returning 200 for impossible pages. Invalid URLs should return a real error.
6. Leaving "next" links without an end. Pagination and calendars need limits.
7. Using relative links carelessly. Use root-relative or absolute paths.
8. Assuming small stores are immune. Traps cause duplication and clutter at any size.
9. Forgetting third-party tools. Filter and search apps generate URLs too.
10. No monitoring. New traps appear with new features, so keep watching.

How we help

At AH SEO Agency we find crawl traps with a combination of controlled crawls, Search Console data and log analysis, then trace each one to the template or application logic that produces it. We give your developers a specific fix for each cause, advise on the right order for controls, and set up the checks that stop traps from returning. See our technical SEO service, the faceted navigation SEO page or request a free SEO audit. The next article explains when to turn a filtered attribute into a permanent landing page.

Summary

• A crawl trap is an endless supply of URLs that lets a crawler wander without reaching new content, and filters are the most common cause on ecommerce sites.
• Traps waste crawling, load the server, clutter the index and dilute internal links, and they are not confined to huge sites.
• Find them with a limited crawl grouped by pattern, Search Console's reports, server logs and manual tests.
• Fix the source where you can: validate input, cap combinations, return real 404 pages and stop generating endless links.
• Use noindex and robots.txt in the right order for what is already indexed, and use emergency crawl blocks only briefly.
• Prevent recurrence with release-time crawl tests, monitoring and a written URL policy.