Googlebot, crawling and the DOM: what site owners should check

How does Googlebot crawl, render and index a page?

Googlebot requests a URL, receives the HTML, discovers the resources and links it references, renders the page with a recent version of Chromium and then passes the content to indexing. Google's documentation describes crawling, rendering and indexing as separate phases.

What happens in the crawling phase?

Googlebot picks a URL from its queue, checks robots.txt and fetches the page. It reads the HTML and extracts links from anchor elements with an href. It also notes resources such as stylesheets, scripts and images. The server response code decides what happens next: a 200 can be processed, a redirect is followed and errors are handled according to Google's status code rules.

Why does robots.txt matter at this stage?

Googlebot checks robots.txt before fetching. If a resource that you need for rendering is blocked, Google may be unable to see the page as users do. Our crawl trap guide covers related controls.

What happens in the rendering phase?

Pages that rely on JavaScript are queued for rendering, where a headless Chromium runs scripts and produces the DOM, the live structure of the page after scripts have changed it. Google's JavaScript SEO basics explain that rendering can happen after the first crawl and that the rendered DOM is what Google uses to find content and links added by scripts.

Is the DOM the same as the HTML source?

No. The source is what the server sends. The DOM is the browser's in-memory version after parsing and scripting. They can differ greatly on JavaScript-driven sites.

What happens in the indexing phase?

Google analyzes the rendered content, chooses canonical versions, evaluates signals and decides whether to store the page in its index. Indexing is never guaranteed, and a page can be crawled and rendered without being indexed.

Where do most problems occur?

Typically at the boundaries: blocked resources, content that appears only after user actions, slow or failing scripts, and signals that change between the raw HTML and the rendered DOM.

What is Googlebot's 2 MB file size limit?

Googlebot's documentation says it crawls the first 2 MB of a supported file type, and the first 64 MB of a PDF. When a file exceeds the limit, Googlebot stops the fetch and sends only the downloaded part for indexing consideration. The page was last updated February 3, 2026.

How does the limit apply to your pages?

The limit applies per fetched file and to uncompressed data, so a compressed response that expands beyond 2 MB still counts as larger. Each resource referenced by the page, such as a script or stylesheet, is fetched separately and has its own limit. This means that the HTML document itself must be lean enough that your main content, links and tags all appear within the first 2 MB.

Does this mean most pages are at risk?

No. Typical pages are far smaller. Pages with huge inline JSON, embedded base64 images, giant inline CSS or enormous server-rendered lists are the ones to check.

What can push content past the cutoff?

Large inline state blobs from JavaScript frameworks, thousands of product links rendered before the main content, inline SVG sprites, embedded fonts or images as data URIs, and long navigation menus repeated on every page. Content placed after such bulk can fall beyond the limit.

Which elements should appear early?

The title, meta robots, canonical link, main heading, primary content and key internal links. Anything critical should not sit behind a large block.

How do you check your page size?

Fetch the URL with a tool that shows the uncompressed HTML size, such as a crawler, curl combined with a byte count or browser developer tools. For large templates, measure a sample of the heaviest pages. Our TTFB and crawl frequency guide covers the speed side of crawling.

What if a page is close to the limit?

Move inline data to external files, trim unneeded markup, paginate long lists sensibly and place important content high in the HTML.

How do you compare raw HTML and the rendered DOM?

Fetch the raw HTML, render the page, and compare content, links and tags. Anything that exists only in the rendered version depends on JavaScript execution, and anything that differs in key signals can cause problems.

Which tools show the difference?

View source shows the raw HTML. The browser's Elements panel shows the live DOM. The URL Inspection tool in Search Console shows Google's rendered HTML and a screenshot for a specific URL, including resources it could not load. Crawlers with a JavaScript rendering mode can compare both versions at scale.

What does the URL Inspection tool tell you?

Whether the URL is indexed, which canonical Google chose, how the page rendered, and which resources failed to load.

What should you compare?

Element	Check in raw HTML	Check in rendered DOM	Risk if different
Title and meta description	Present	Same value	Wrong snippet or title
Canonical link	Present, correct	Same value	Conflicting signals
Meta robots	No unintended noindex	Same value	Pages dropped from index
Main content	Present or clearly delivered	Visible text	Thin or empty content
Internal links	Anchor elements with href	Same links	Pages not discovered
Structured data	Present, valid	Same data	Lost rich result eligibility

What if your framework changes tags after load?

Google advises being careful when JavaScript changes robots or canonical signals. Setting them in the initial HTML is the safer choice.

How do you spot content that depends on user actions?

If text appears only after a click, scroll or hover, Googlebot may not trigger it. Content loaded on interaction should be reachable through crawlable links or present in the DOM without needing an action. Our guide to JavaScript filters and content visibility explains how this affects ecommerce pages.

Does lazy loading harm indexing?

It can if implemented in ways that depend on user interaction. Use standard lazy loading that works when content enters the viewport without special events.

How do links and navigation affect discovery?

Googlebot discovers pages mainly by following links. Google's documentation says to use anchor elements with an href attribute, because other constructs may not be followed. Clear links make discovery reliable.

Which link types are crawlable?

Standard anchor elements with real URLs in the href are crawlable. Links that exist only as click handlers on buttons or divs, links using fragment-only hrefs for navigation or links created without any URL can be missed. If your menu relies on scripts, test it with JavaScript rendering in a crawler.

Can a button be a link?

Only if it behaves as an anchor with a destination URL. Styling a link as a button is fine, but a button with a script handler and no URL is not a link to Googlebot.

How should you structure internal linking?

Link important pages from navigation and from related content using descriptive anchor text. Avoid placing them behind search boxes, forms or infinite scroll without crawlable alternatives. Our guide on orphan pages shows how to find pages that no link reaches.

Does the order of links in the HTML matter?

It can if the HTML is very large, because early links are more certain to fall within the crawl limit. Place primary navigation and key links near the top.

What about infinite scroll and pagination?

Provide paginated URLs that can be reached through links, so that content beyond the initial load is discoverable. Google's ecommerce guidance on pagination and incremental loading describes the approach.

What should you know about JavaScript rendering?

Google can render JavaScript, but rendering adds steps and dependencies. Server-side rendering, static generation or hydration of already-present HTML reduce the risk that content is delayed or missed.

Which rendering approaches exist?

Client-side rendering sends a minimal HTML shell and builds content in the browser. Server-side rendering sends complete HTML for each request. Static generation produces complete HTML ahead of time. Hybrid approaches pre-render and then hydrate. For content that must be indexed, delivering it in the initial HTML is the most dependable.

Which approach does this site use?

This site is generated as static HTML, so content, links and tags are present in the initial response without needing scripts to appear.

What commonly breaks rendering?

Blocked JavaScript or CSS files in robots.txt, scripts that error or time out, content waiting for a user event, dependence on APIs that return errors for crawlers, and fragment-based routing. Test with the URL Inspection tool and check the console for errors.

Should you use dynamic rendering?

Google has described dynamic rendering as a workaround and not a long-term solution. Prefer server-side rendering or static generation when you can.

How do you handle status codes in single-page apps?

Return real status codes for missing pages. A page that shows "not found" but returns 200 is a soft 404. Google's guidance on HTTP status codes explains how each code is handled.

Which signals should be in the initial HTML?

Canonical links, meta robots directives, hreflang, structured data and the main content are safest in the initial HTML, because they are available to Googlebot without waiting for rendering and cannot be altered by script failures.

Why is consistency important?

If the raw HTML says one thing and JavaScript changes it, Google may see conflicting signals. A raw canonical that differs from a rendered canonical, or a noindex that scripts remove, creates uncertainty. Google's documentation advises against relying on JavaScript to change these signals where possible.

What is the safest pattern?

Output the intended value in the HTML and do not modify it with scripts.

How should structured data be added?

JSON-LD in the HTML is the simplest approach and Google's recommended format for most types. If you use a tag manager or script to inject it, test that Google sees it in the rendered output and that it matches visible content. Our AI shopping and structured data article shows how consistent markup supports product visibility.

How do you validate it?

Use the Rich Results Test and the URL Inspection tool, then monitor the relevant Search Console reports.

What about meta tags and titles?

Set titles and descriptions server-side. Google may rewrite titles in results, but a good source title is still the starting point.

How do you reduce HTML weight without hurting the page?

Reduce weight by moving bulk data out of the HTML, removing repeated markup and delivering only what the first view needs. The goal is a document where everything important appears early and the total stays comfortably under the documented limit.

Which changes give the biggest savings?

Move large inline JSON and framework state into external files that load separately. Replace data-URI images with ordinary image files that browsers and Google can fetch on their own. Load icon sprites from a file, not inline on every page. Strip comments and unneeded whitespace in production. Avoid printing thousands of items in one document, and paginate or load further items through crawlable links.

Is minification enough?

It helps, but it rarely solves a structural problem. If the page embeds a data set, moving it out of the document matters more than shaving bytes from tags.

How should long lists be handled?

A category with thousands of products should be divided into paginated pages with distinct URLs and links between them. Each page then holds a manageable number of items, loads faster and keeps important content near the top. Our article on category page content placement shows where text and links belong on those pages.

What about mega menus?

A menu that repeats hundreds of links on every page adds weight and dilutes focus. Keep primary navigation concise and link deeper pages from relevant category and content pages.

How do you verify the change?

Re-measure the uncompressed size of the heaviest templates, re-inspect key URLs in Search Console and crawl again to confirm that links and tags are intact. Record the before and after figures so that you can detect regressions when templates change.

Who should own page weight?

Developers and SEO together. Page weight is easy to inflate with each new feature, so set a budget for HTML size and test against it in your release process.

Which resources should Google be allowed to fetch?

Google needs access to the CSS, JavaScript and images that shape how a page looks and what it contains. Blocking them in robots.txt can leave Google with an incomplete picture of the page.

What should robots.txt allow?

Allow the scripts and stylesheets used to render content, along with the images that matter for search. Block only areas with no value to search, such as internal search results or admin paths. Google's introduction to robots.txt explains that the file controls crawling and not indexing, so use it for crawl management and use noindex for index control.

How do you find blocked resources?

The URL Inspection tool lists page resources that could not be loaded. A crawler configured to respect robots.txt can also report blocked assets at scale.

What about third-party scripts?

Analytics, chat widgets and ad tags can slow rendering and sometimes fail. Make sure your main content does not depend on them, load them after the primary content and test that a failure in one does not break the page. Consider whether each script is still needed.

Do third-party failures affect indexing?

They can if they block rendering or cause errors that stop your own scripts from running. Isolate them so that the main content does not depend on their success.

How do server errors and timeouts matter?

If your server returns errors or responds slowly to Googlebot, Google may crawl less. Keep resources fast and reliable, and monitor the Crawl stats report in Search Console for spikes in errors or response times. Our server response time guide explains the link between speed and crawl frequency.

Should you verify Googlebot in logs?

Yes, because other bots can pretend to be Googlebot. Google describes verifying requests with reverse DNS or published IP ranges, so use those methods before drawing conclusions from log data.

Which misconceptions about crawling and rendering should you drop?

Several beliefs persist that lead teams to waste effort or miss real problems. Replacing them with documented facts makes audits faster and fixes more precise.

Is "Google can render JavaScript, so nothing else matters" true?

No. Rendering is possible but not free. It adds delay, depends on resources loading correctly and can fail silently. Content you need indexed is safest when it does not depend on it.

What is the cost of relying on rendering?

Slower discovery, higher risk of missing content and harder debugging when something goes wrong.

Is "bigger pages always rank better" true?

No. Length is not a ranking goal, and very large documents can hit the crawl limit. Aim for pages that are complete for the user's need and efficient to fetch.

Does a longer article risk the 2 MB limit?

Almost never. Text is small. A 5,000-word article is typically a few dozen kilobytes. The limit is a problem for pages bloated by embedded data and repeated markup, not for text content.

Is "robots.txt removes pages from Google" true?

No. Robots.txt blocks crawling. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use noindex, or remove the page.

Which control suits which goal?

Robots.txt for crawl management, noindex for index exclusion, status codes for removal and canonical tags for duplicates.

Is "the DOM is only for developers" true?

No. The DOM is what Google sees after rendering, so SEO professionals should know how to inspect it and compare it with the source. A few minutes in browser developer tools often explains a puzzling indexing issue.

How do you monitor crawling and rendering over time?

Monitor with Search Console reports, server logs and scheduled crawls. Regular checks catch regressions after releases and show whether fixes had any effect on how Google handles your pages.

Which Search Console reports matter most?

The Page indexing report shows why URLs are not indexed, including "Crawled, currently not indexed" and "Discovered, currently not indexed". The Crawl stats report shows request counts, response times and file types. The URL Inspection tool lets you test individual URLs, and the sitemaps report shows submitted and discovered pages.

What does "Discovered, currently not indexed" suggest?

Google knows about the URL but has not crawled it yet. It can reflect crawl capacity, perceived value or site quality, so look at the whole set of affected URLs and not one page.

What can server logs add?

Logs show which URLs Googlebot actually requests, how often and with which response codes. Compare them with your list of important URLs to find pages that are never crawled and sections that consume crawl effort without value. Verify the bot first, since other agents can imitate it.

How do you act on log findings?

Strengthen internal links to ignored pages, remove or restrict low-value URLs that absorb requests and fix any recurring errors.

How often should you schedule crawls?

Run a rendered crawl of important templates after each release and a full crawl monthly or quarterly depending on site size. Automate comparisons so that new differences between raw and rendered output raise an alert.

What belongs in a release checklist?

Title and canonical present, no accidental noindex, internal links as anchors, HTML size within budget, structured data valid and status codes correct.

How do mobile-first indexing and AI features relate to rendering?

Google primarily uses the mobile version of a page for indexing, so the mobile HTML and rendered DOM are the versions that count. AI features in Search draw on the same index, which means content that Googlebot cannot see cannot support those features either.

What should be identical on mobile and desktop?

The main content, headings, structured data, titles, meta robots and key internal links. If a mobile layout hides content or drops links that the desktop page has, Google may see less than you expect. Google's mobile-first indexing guidance recommends equivalent content across versions.

How do you test the mobile version?

Use the URL Inspection tool, which fetches with the smartphone crawler, and compare it with a desktop crawl.

Why does this matter for AI features?

Google's documentation on AI features says that normal SEO fundamentals apply and that content must be indexable to be eligible. If a page cannot be crawled, rendered and indexed correctly, it cannot be used as a supporting link. Our guide to AI Overviews and AI Mode explains what site owners can do.

Is there a special tag for AI features?

Google says there are no extra technical requirements beyond being indexed and eligible to appear with a snippet, so technical basics are where to invest.

What is the takeaway?

Good crawling and rendering hygiene supports every part of search visibility, from classic results to AI-generated answers. It is the foundation, and it is within your control.

How do you decide which fix to prioritize?

Prioritize by impact on indexable, valuable pages and by effort. A simple scoring of reach, severity and effort turns a long list of findings into an order of work that developers can plan.

How does the scoring work?

Reach is the number or share of important pages affected. Severity is whether the problem blocks discovery, indexing or only degrades quality. Effort is the development time required. Fix high-reach, high-severity, low-effort items first. A noindex tag accidentally added to a template is a quick fix with severe impact, while rebuilding a rendering architecture is a larger project that needs a business case.

What if developers push back?

Bring evidence: the URL Inspection output, the size measurements and the affected page counts. Concrete examples persuade better than general advice.

When is a rebuild justified?

When a structural limitation affects a large share of valuable pages, workarounds are fragile and the cost of continued problems exceeds the cost of change. Treat it as a business decision, supported by data, and do not promise a specific ranking result.

What can be done in the meantime?

Pre-render the most important templates, simplify the worst offenders, and keep monitoring so that the case for change stays grounded in facts.

How do you communicate results?

Report changes in indexed pages, crawl stats and impressions for the affected templates, with dates of each fix. Be clear that search results depend on many factors and that improvements might take weeks to appear.

What is a practical audit process for crawling and rendering?

Audit in five steps: measure page size, compare raw and rendered output, test link discovery, check signals and verify status codes. Each step has a clear pass or fail, and the results prioritize fixes.

How do you run the five steps?

1. Measure uncompressed HTML size for your heaviest templates and flag any near 2 MB.
2. Crawl in both non-rendered and rendered modes and compare content, titles, canonicals and links.
3. Test that key pages are reachable by anchor links from navigation and from related pages.
4. Check that canonical, robots, hreflang and structured data are in the initial HTML and consistent.
5. Verify status codes, including soft 404s and redirect chains.

Which template should you test first?

Start with the template that earns most revenue, then the one that has the most pages.

How do you read the results?

Treat differences between raw and rendered output as the list of dependencies. Decide for each whether it matters for search. Content that serves users but not search can stay script-driven. Content you want indexed should be in the initial HTML or reliably rendered.

How often should you repeat the audit?

After major releases and at least twice a year. See our technical SEO audit checklist for the full process.

What do you log?

Keep a record of template sizes, differences found and fixes made, so that regressions are easy to spot later.

What does a rendering fix look like in practice?

The example is fictional. It shows a method, not a promised result.

What was the problem?

A furniture retailer built its category pages with a JavaScript framework that inlined a very large JSON state object before the product grid. Search Console showed many category URLs as crawled but not indexed, and the URL Inspection tool showed incomplete rendered HTML.

What did the audit find?

The raw HTML of the largest category pages exceeded the documented limit once uncompressed, the product links sat after the state blob and the canonical tag was injected by script.

What changed?

The team moved state data to an external file, rendered the first page of products on the server, placed the canonical link in the HTML head, reduced the markup of the menu and added paginated URLs reachable by links. They then re-inspected key URLs and monitored indexing over several weeks, recording that improvements were not instant and could not be guaranteed.

What did they learn?

That page weight and signal placement matter as much as the framework choice, and that measuring before and after is what shows progress.

How does AH SEO Agency help with crawling and rendering?

We measure template sizes, compare raw and rendered output, test discovery and signals and write a prioritized list of fixes your developers can act on. We work from Google's documentation and explain what is confirmed and what is unknown.

Where do you start?

See our technical SEO service, our GEO versus SEO guide or request a free SEO audit.

Summary: what should you check first?

Check HTML size against the 2 MB limit, compare raw and rendered output, make sure links are real anchors, and keep signals in the initial HTML.

• Crawling, rendering and indexing are separate phases with separate failure points.
• Googlebot crawls the first 2 MB of a supported file, and each resource is fetched separately.
• Use anchor links with href for discovery.
• Prefer server-side or static HTML for content that must be indexed.
• Keep canonical, robots and structured data consistent in the initial HTML.