Log file analysis for ecommerce: measuring Googlebot crawl frequency on SKU sets

What is log file analysis and why does it matter for ecommerce SEO?

Log file analysis is the review of the records a web server writes every time something requests a page, image or script. For an online store, those records are the only complete, first-hand account of how search engine crawlers move through thousands or millions of product and category URLs.

What is a crawl, and what does a log have to do with it?

A crawl is a visit by an automated program, called a crawler or bot, that fetches pages so a search engine can understand them. Google's main crawler is Googlebot. Each fetch is a request, and the server records it in an access log, which is why the log is a direct measurement of crawling.

How do logs differ from a crawler tool such as Screaming Frog?

A site crawler tool imitates a bot and shows what a bot could find. Logs show what bots actually did. The two views complement each other: the tool lists the pages that exist, and the logs show which of them Google visits.

How do logs differ from Search Console?

Google Search Console reports summarize and sample crawl activity over a limited period. Logs record every request your server handled, with the exact URL, time and response, so they can answer questions that the reports cannot.

Why do large SKU sets make logs more valuable?

A store with a handful of products can be crawled completely in a short time, so crawl effort rarely limits it. A catalog with tens or hundreds of thousands of products, each with variants, filters and parameters, forces crawlers to choose. Logs reveal what they chose.

What does Google say about crawl budget?

Google's documentation says crawl budget is mainly a concern for very large sites or for sites with many pages that change quickly. Even smaller stores still benefit from logs, because they expose errors, redirect chains and wasted requests.

What does an access log contain?

An access log is a text file in which each line describes one request: who asked, when, for what, and what the server answered. The exact format depends on the server software, but the core fields are the same everywhere.

Which fields appear in a typical log line?

Most logs follow the common or combined format and include the fields in the table below.

What does a real log line look like?

A single line usually reads like a sentence: an address, a date, a request for a product URL, a 200 status, a size and the Googlebot user agent. Once you can read one line, you can read millions, because tools only repeat the same structure.

Where do log files live?

Logs sit wherever requests are handled. On a traditional server that is the web server's log folder. On a content delivery network (CDN) or hosted platform, the provider keeps them and may offer an export. Several layers can each hold a different version of the truth.

What if the store runs on a hosted platform?

Some hosted ecommerce platforms do not provide raw access logs. In that case, ask the platform whether an export or a CDN log feed is available, and rely on Search Console Crawl stats plus site crawls for the parts you cannot see directly.

What about CDN logs?

A CDN answers many requests without contacting your origin server, so origin logs undercount crawler activity. CDN logs are more complete, but they may use a different format and need to be combined with origin data for response times.

How do you collect and prepare log files?

You collect logs by exporting a representative period, usually at least two to four weeks, from every layer that handles requests, then clean them so that only the requests you care about remain. Preparation matters because raw logs are large and noisy.

How much log data is enough?

Aim for a period long enough to cover several crawl cycles for your important pages. Many analysts start with about a month, and extend the window for sites where Googlebot visits deep pages rarely. A shorter window is still useful for diagnosing a sudden problem.

Should you include seasonal peaks?

Compare a normal period with a promotional or seasonal one. Crawl behavior often changes when prices, stock or content change quickly, and that comparison shows how well your store handles busy periods.

How do you filter and clean the data?

Filter to requests for pages, not for static files, unless you are studying images or scripts. Remove health checks, internal monitoring and your own tools. Normalize URLs by lowercasing the host and removing known tracking parameters, so that the same page is not counted twice.

How do you respect privacy?

IP addresses can count as personal data under some privacy laws. Limit access to raw logs, keep only what you need and delete raw files when the analysis is finished, in line with your own privacy policy and legal advice.

What tools can process large logs?

Small files open in a spreadsheet. Larger ones need a dedicated log analyzer, a database or a script. Common choices are Screaming Frog Log File Analyser, a data warehouse such as BigQuery, an ELK stack, or Python with the pandas library.

How do you verify that a request really came from Googlebot?

You verify Googlebot by checking the IP address, not the user agent text, because anyone can claim to be Googlebot. Google documents two methods: a reverse and forward DNS lookup, or a comparison against its published lists of crawler IP ranges.

How does the DNS check work?

Run a reverse DNS lookup on the IP address. A genuine Google crawler resolves to a host name ending in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that host name and confirm it returns the same IP address.

Why are both lookups needed?

A reverse lookup alone can be faked by whoever controls the address. The forward lookup closes the loop, because only Google controls the records for its own host names.

Why compare against Google's published IP ranges?

Google publishes machine-readable lists of the address ranges used by its crawlers. Matching your log addresses against those lists is faster than DNS lookups for millions of lines, and it can be automated in a script or a log tool.

Which Google user agents matter for SEO?

Googlebot Smartphone and Googlebot Desktop crawl pages for Search. Other Google crawlers exist for images, video, ads and tools, such as the one behind the URL Inspection tool, and they should be separated so they do not distort the picture of search crawling.

What do fake Googlebot requests tell you?

A large share of fake requests usually means scrapers or other bots are using up server capacity. That is a performance and security matter, and it also explains why the totals in raw logs often exceed the numbers Google reports.

How do you group SKU sets and templates for analysis?

You group URLs by what they are, not by what they are called, using patterns in the path and parameters. A grouped view turns millions of lines into a dozen rows that a team can act on.

What is a URL template?

A template is a family of pages built the same way, such as product pages, category pages, filtered pages, search results, blog articles and cart pages. Each template has a recognizable URL pattern, for example /products/ followed by a slug.

How do you label templates?

Write a short list of rules that match the beginning of the path or a parameter name, and apply them in order to every URL. Keep a final group called "other" and review it regularly, because new URL types appear without announcement.

What is a SKU set?

A SKU set is a group of products that share a business meaning, such as a brand, a category, a price band, a supplier, a stock status or a season. Grouping by SKU set links crawl data to commercial importance.

Which SKU sets are worth separating?

Start with best sellers, new arrivals, discontinued items, out-of-stock products and seasonal lines. These sets show whether crawlers spend effort on the products that earn revenue, or on those that no longer matter.

How do you join logs with other data?

Attach each URL to extra columns from a crawl, from your product database and from Search Console: template, click depth, internal links, indexability, canonical status, revenue and impressions. The joined table lets you compare crawl frequency with importance.

Which crawl metrics should you track?

Track a small set of measures that connect crawling to outcomes: crawl share by template, crawl frequency for key pages, status code mix, response time and discovery lag for new products.

What is crawl share by template?

Crawl share is the percentage of Googlebot requests that go to each template. A healthy store sends most requests to products and categories. A store with a problem often shows a large share going to filters, parameters or redirects.

What is a good distribution?

There is no universal figure, so compare against your own business priorities. If filters earn no revenue and attract a major share of crawling, that is a gap between effort and value, whatever the exact percentage.

How often should important pages be crawled?

Measure the interval between visits for your best products and categories. Pages that change often, such as price or stock, benefit from frequent visits. Long gaps for key pages, with frequent visits to low-value pages, suggest weak internal linking or wasted crawl.

What does crawl frequency by depth show?

Plot visits against click depth. Googlebot usually visits shallow pages more often than deep ones. A steep drop-off after a few clicks points to structure problems that internal links can fix.

What do status codes reveal?

A high share of 200 responses on useful pages is good. Many 301 and 302 responses suggest outdated internal links or redirect chains. Many 404 responses suggest broken links or removed pages that are still linked. Any 5xx responses suggest server trouble and can reduce crawling.

What about 304 responses?

A 304 status means the page has not changed since the last visit, and the server avoided sending it again. Many 304 responses are normal and efficient, and they show that conditional requests work.

What patterns reveal crawl waste and crawl gaps?

Crawl waste is effort spent on URLs that should not matter, and a crawl gap is a page that matters but is rarely or never visited. Logs show both, and comparing them with a site crawl makes them obvious.

How do you spot crawl waste?

Look for large groups of URLs with parameters, session identifiers, sorting options, internal search results and endless filter combinations. If crawlers visit them in volume, they are consuming attention that products and categories could use. Our guide to identifying crawl traps explains the typical causes.

What does waste on redirects look like?

Googlebot repeatedly requests old URLs that redirect. The fix is to update internal links and sitemaps to point straight at the final address, so crawlers stop walking through the redirect.

How do you spot crawl gaps?

List every important URL from your sitemap, your database and a site crawl, then check which of them appear in the logs. Important pages with no requests in a month are crawl gaps and need stronger internal links or fixes to blocking.

How do logs reveal orphan pages?

An orphan page is a page that no internal link points to. If logs show a URL that your crawl never found, something outside the site, such as an old sitemap or an external link, is leading crawlers there. Our guide to orphan page identification covers this in detail.

What does a sudden change in crawling mean?

A sharp fall in requests can follow server errors, slow responses, a blocking rule or a large change in the site. A sharp rise can follow a new feature that generates URLs. Compare the date of the change with your release log.

How do you turn log findings into fixes?

You turn findings into fixes by pairing each pattern with a specific action, an owner and a way to measure the result. A report that only lists numbers will not change crawling.

Which fixes follow which findings?

The table below maps common findings to usual responses.

How do you prioritize?

Rank findings by the share of crawl affected and the value of the pages involved. A change that frees a large share of requests, or that rescues pages that earn revenue, comes first.

How do you brief developers?

Give each fix a short ticket: the problem in one sentence, the evidence from the logs, the URL patterns, the required behavior and a test. Our guide to noindex versus disallow includes a ticket template that works well for filter problems.

How do you confirm that a fix worked?

Take logs from the weeks after release and repeat the same grouped analysis. The share of requests to the affected template should fall, and the share to products and categories should rise. Record the result in a change log.

How often should you analyze logs, and which tools help?

Review logs monthly for a growing store and quarterly for a stable one, and always after a migration, a redesign or a major technical change. Automated dashboards make a regular review quick.

What is a sensible routine?

Export logs on a schedule, verify Googlebot, label templates, produce the same dashboard each time and compare it with the previous period. Consistency matters more than sophistication, because trends are easier to read than single snapshots.

What should the monthly dashboard show?

Crawl share by template, status code mix, response time, crawl frequency for key pages and the count of important URLs not crawled. Five or six charts are enough.

Which tools suit which teams?

Small teams can use a desktop log analyzer. Teams with large volumes often load logs into a data warehouse and build dashboards in a business intelligence tool. Developers may prefer scripts that run automatically after each export.

How do logs fit with the rest of a technical audit?

Logs are one input to a full audit alongside a site crawl, Search Console and page speed data. Our technical SEO audit checklist shows where log analysis belongs, and our technical SEO service uses it for large catalogs.

What does a log analysis look like in practice?

A worked example shows how the steps fit together. The numbers below are illustrative, not data from a real store, and they demonstrate the method rather than a benchmark.

What does the starting position look like?

Imagine a furniture retailer with 40,000 products. Its team exports four weeks of CDN logs, verifies Googlebot by IP range and labels templates. The grouped table shows that product pages receive a modest share of Googlebot requests, category pages a small share and parameterized filter URLs the largest share of all. Redirected URLs take another noticeable slice, and the new-arrivals collection is crawled only once a week.

What does the comparison with a site crawl add?

A crawl of the live site finds 40,000 product URLs, but the logs show that thousands of them were never requested during the four weeks. Many of those sit more than five clicks from the home page and are linked only from deep pagination. The crawl also reveals that the internal links in the category template point to the old versions of several hundred URLs that now redirect.

What fixes does the team choose?

The team ranks the findings by the value of the pages involved. First, they update category template links to point at final URLs, which removes the redirect waste. Second, they limit crawlable filter links to a short whitelist and apply canonical and noindex controls to the rest, following the safe order described in our noindex versus disallow guide. Third, they add a "new arrivals" block and a "best sellers" block to the home page and top categories, so that fresh and valuable products are reachable in two clicks.

How do they measure the result?

Four weeks after release, they export logs again and rebuild the same table. The share of requests to filter parameters has fallen, the share to products and categories has risen, the redirect share has almost disappeared and the new-arrivals collection is crawled several times a week. They also check Search Console for the indexing status of the previously uncrawled products and note that more of them move into the index over the following weeks.

What should you not conclude from an example like this?

Do not treat the pattern as a promise. Changing crawling does not guarantee higher rankings, because rankings depend on relevance, quality and many other signals. The reliable benefits are cleaner crawling, faster discovery of new products and fewer wasted requests, which create better conditions for the rest of your SEO work.

How does log analysis connect to Google Search Console Crawl stats?

The Crawl stats report in Google Search Console and your server logs describe the same activity from two angles, and using both gives a more reliable picture than either alone. The report is quick and free, while logs are exact and customizable.

What does the Crawl stats report show?

The report summarizes total crawl requests, download size and average response time, and breaks requests down by response code, file type, purpose and crawler type. It also shows host status, such as problems reaching your server or reading robots.txt. It is the fastest way to notice that something has changed.

When is the report enough?

For a small or medium store with no obvious problems, a monthly look at the report is often enough. It flags rising error rates, slower response times and shifts between discovery and refresh requests, which are the early signs that deeper analysis is needed.

When do you need the logs as well?

Use logs when you need exact URLs, patterns by template or evidence for a developer. The report shows sample URLs, but logs let you count every request for a pattern, join the data with revenue or stock and prove that a specific fix changed behavior. They also expose requests from crawlers that the report does not cover, and they cover periods and hosts outside what the report retains.

How do you reconcile the two sources?

Expect the totals to differ. The report covers Google crawlers only, uses its own counting and may group hosts differently, while raw logs include every bot and every layer. Compare trends, not absolute numbers: if both sources show a rise in errors at the same time, the signal is real.

How should you handle privacy and governance for log data?

Treat logs as sensitive operational data, because they contain visitor IP addresses and sometimes query strings with personal details. A short policy covering access, retention and sharing keeps the analysis compliant.

Who should have access to raw logs?

Limit raw access to the people who need it, usually a developer and the SEO analyst. Share the grouped, anonymized results with the wider team, because charts of crawl share by template contain no personal information and answer most business questions.

What should be removed before sharing?

Strip IP addresses of normal visitors, remove query strings that may contain emails or tokens, and keep only verified crawler rows when you share examples.

How long should you keep logs?

Keep raw logs only as long as the analysis needs them and your legal advice allows, then delete them. Keep the aggregated monthly tables for trend comparison, since they hold no personal data and cost almost nothing to store.

What are the common mistakes in log file analysis?

The most common mistakes are trusting unverified bot traffic, analyzing too short a period, ignoring templates and measuring without acting. Each one leads to confident but wrong conclusions.

Which mistakes distort the data?

Counting fake Googlebot requests, mixing search crawlers with other Google crawlers, forgetting that CDN logs differ from origin logs and comparing periods of different length all produce misleading numbers.

Which mistakes waste the effort?

Producing a long report with no owners, fixing symptoms instead of causes, and failing to repeat the analysis after changes are the usual reasons that log work does not improve crawling.

How do you avoid them?

Verify bots, keep a fixed method, keep a change log and finish every analysis with a short list of actions that someone has agreed to do.

How does AH SEO Agency help with log file analysis?

We collect and verify your logs, label every URL template and SKU set, compare the results with site crawls and Search Console, and give your developers a ranked list of fixes with expected effects. We then repeat the analysis to confirm that crawling has moved toward the pages that earn revenue.

Where can you learn more?

See our ecommerce SEO audit work, read about server response time and crawl frequency or request a free SEO audit.

Summary: what should you remember about log file analysis?

Server logs are the ground truth of crawling. Use them to verify Googlebot, group URLs by template and SKU set, measure crawl share and status codes, find waste and gaps, and convert every finding into a tested fix.

• Logs record every request, so they show real crawler behavior.
• Verify Googlebot by IP address using DNS checks or Google's published ranges.
• Group by template and SKU set to connect crawling with business value.
• Compare logs with a site crawl to find orphan pages and crawl gaps.
• Repeat the analysis after each fix to prove the effect.