XML sitemap best practices: what to include, how to use lastmod and how to audit sitemaps

What is an XML sitemap, and what does it do for crawling and indexing?

An XML sitemap is a machine-readable list of URLs that you want search engines to discover and crawl. It speeds discovery of new and updated pages, tells Google which URLs you consider canonical and gives you a clean list to compare against what Google actually indexed.

How does Google describe sitemaps?

Google's guidance on building and submitting a sitemap, last updated July 8, 2026, explains that sitemaps help Google find URLs, and that submitting one is a hint and not a guarantee that every URL will be crawled or indexed. Google supports several formats, with XML the most versatile because it can also describe images, video, news and localized versions of pages.

Is a sitemap required?

No. Google can discover pages through links alone, and a small, well-linked site may not need one. A sitemap helps most when a site is large, new, has many isolated pages, changes often or relies on media discovery.

What can a sitemap not do?

It cannot force indexing, fix poor quality or replace internal links. A URL in a sitemap that Google considers thin or duplicate may still be excluded, and a page that nothing links to remains weakly connected. The guide to internal linking and crawlable links explains why links carry context that a sitemap cannot.

Does listing a URL improve its rank?

No. Sitemaps help discovery and crawling, not ranking. Their value is in coverage and diagnostics.

Which sitemap formats exist?

Format	Best for	Limits
XML sitemap	Most sites, with image, video, news and hreflang extensions	50 MB uncompressed or 50,000 URLs per file
Sitemap index	Grouping many sitemaps	Lists sitemap files, not pages
RSS, mRSS or Atom feed	Blogs and media sites that already publish feeds	Usually lists only recent URLs
Text file	Very simple sites	One URL per line and nothing else

Which format should you choose?

XML for almost everyone. Feeds are a useful extra for frequently updated content, and a text file works only when you have no need for extensions.

What are the technical rules for a valid XML sitemap?

A valid sitemap is UTF-8 encoded, uses entity-escaped values, lists fully qualified absolute URLs and stays within the size limits. Errors in these basics cause Search Console to reject or partly ignore the file.

What do the size and encoding limits mean?

A single sitemap file can hold up to 50 MB uncompressed or 50,000 URLs, whichever comes first. The file must be UTF-8, and special characters such as ampersands in URLs must be escaped as entities. Compressing with gzip is allowed, but the limit applies to the uncompressed size. Larger sites split their sitemaps and group them in a sitemap index file.

How do you write URLs correctly?

Use the full address, including the protocol and host, for example https://www.example.com/page/, and use the same canonical form as on the page: the same protocol, host, trailing slash convention and capitalization. Do not list relative paths.

What does a minimal XML sitemap look like?

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/blog/xml-sitemaps/</loc>
    <lastmod>2026-10-24</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/seo-services/technical-seo/</loc>
    <lastmod>2026-09-12</lastmod>
  </url>
</urlset>

The addresses and dates are examples. Only loc is required, and lastmod is optional but useful when accurate.

Which tags does Google ignore?

Google ignores the priority and changefreq values, so you can omit them. Spending time tuning them has no effect on crawling.

How do sitemap index files work?

A sitemap index lists the locations of individual sitemap files, each with an optional lastmod. Use one when your site exceeds the limits or when you want logical groups, such as products, categories, articles and locations. Submit the index, and Google reads the files it lists. A sitemap index can be nested only one level, so keep the structure simple.

How should you name and place the files?

Use clear names such as sitemap-products.xml and sitemap-articles.xml, and place them at the site root. Google notes that a sitemap in a subdirectory generally covers only URLs below that directory, unless it is submitted through Search Console.

Which URLs belong in a sitemap and which do not?

Include canonical, indexable, 200-status URLs that you want in search results, and exclude everything else. A sitemap should describe your intent, so that differences between what you submit and what Google indexes point to real problems.

Which URLs should you include?

Canonical versions of pages with unique, useful content: home page, service and category pages, product pages, articles, location pages, important resources. Use the same canonical URL that appears in the page's rel canonical tag and internal links.

Should every page be listed?

Every page you want indexed, yes. Pages that serve visitors but not search, such as thank-you pages and internal search results, should be left out.

Which URLs should you exclude?

• URLs that redirect, because the sitemap should list the final destination.
• URLs that return 4xx or 5xx status codes.
• Pages with a noindex directive.
• Non-canonical duplicates, parameter variants and printer-friendly versions.
• Thin pages you do not want to promote, such as empty tag archives.
• Staging, test and private URLs.

Why does including redirected URLs matter?

Because it wastes crawl attention and fills the Page indexing report with exclusions that you created. The discussion of HTTP status codes, redirects and soft 404s explains how each response is treated.

How do sitemaps and canonicals interact?

A sitemap listing a URL while the page declares a different canonical sends a mixed signal. Google may follow the page's canonical instead. Keep sitemap, canonical tags, redirects and internal links in agreement. When variants must stay live, as with products, list the preferred URL only, as the guide to product variants and canonicalization explains.

Should paginated and filtered pages be included?

Include paginated URLs only if they are indexable and valuable. Include filtered pages only when you have deliberately chosen them as landing pages, following the logic in strategic indexing of filtered attributes.

How should you use lastmod, and what about priority and changefreq?

Use lastmod to state when a page's content last changed significantly, and keep it truthful. Google says it uses the value only when it is consistently and verifiably accurate, and that priority and changefreq are ignored.

What counts as a significant change?

A change to the main content, structured data or important links. Fixing a typo, updating a copyright year or changing a sidebar widget is not significant. If your CMS updates lastmod on every save or every build, Google will learn that the value is unreliable and may stop trusting it for your site.

How do you generate accurate lastmod values?

Store a real content-modified date in your CMS or database that changes only when an editor makes a meaningful edit, and output that value. Do not use the file system timestamp or the current date at generation time.

Why is honest lastmod helpful?

When it is reliable, Google can prioritize recrawling pages that actually changed, which helps updates appear sooner. It also pairs with Google's guidance on helpful content, which warns against changing dates without substantial updates. Genuine updates, documented in your content pruning and consolidation process, are the right trigger.

Should the visible date match lastmod?

Ideally they agree. A visible "updated" date and a sitemap lastmod that describe the same real change make a consistent story.

Can you ignore priority and changefreq safely?

Yes. Google ignores them, so omit them or leave them out of your generator. Some other search engines may read them, but they are not a ranking lever, and spending time on them is wasted effort.

What replaces changefreq?

Accurate lastmod, good internal links and fast, reliable servers. Those influence how often Google returns.

Where do you place a sitemap, and how do you submit it?

Place the sitemap at the root of the site where possible, reference it in robots.txt and submit it in the Search Console Sitemaps report. Google lists these as the standard ways to tell it about your sitemap.

How do you submit through Search Console?

Open the Sitemaps report for your verified property, enter the sitemap path or the sitemap index path and submit. Search Console then shows the status, the number of discovered URLs and any errors. The report also tracks when Google last read the file. Verify every property variant that matters, and submit the sitemap for the canonical host.

What does "Couldn't fetch" or "Has errors" mean?

It means Google could not read the file or found invalid entries. Check that the URL is reachable, returns 200, is not blocked by robots.txt and passes XML validation.

How do you reference the sitemap in robots.txt?

Add a line such as Sitemap: https://www.example.com/sitemap.xml to robots.txt. The line is independent of user-agent groups, and it can appear more than once. It helps any crawler find the sitemap without a separate submission. The overview of robots.txt and crawler controls shows how the file works.

Can a sitemap on another host work?

Cross-host sitemaps need verification of ownership of both sites, either in Search Console or through robots.txt references. Keep things simple by hosting each site's sitemap on its own host.

What other ways exist to announce changes?

RSS and Atom feeds can use WebSub to announce updates, which suits news and blog sites. The Search Console API allows programmatic submission for large sites. None replaces good internal linking and accurate sitemaps.

Do you need to resubmit after each update?

No. Once Google knows the sitemap, it rechecks it from time to time. Update the file, keep the URL stable and let Google fetch it.

When do image, video, news and hreflang sitemaps help?

Extensions let a sitemap describe media and alternate language versions. They help discovery when media or alternates are hard to find through normal crawling, and they are optional for most small sites.

What do image and video sitemaps add?

Image sitemaps list the image URLs that appear on each page, including those hosted on other domains such as a CDN, which helps Google find images loaded by scripts. Video sitemaps provide metadata such as title, description, thumbnail and duration. Both supplement standard HTML markup. Details on markup and formats appear in the guide to image SEO and Google Images.

Do you need both an XML sitemap and an image sitemap?

You can add image entries to the same XML sitemap rather than creating a separate file.

How do hreflang annotations work in a sitemap?

For multilingual sites, an XML sitemap can list alternate language and region versions of each URL using xhtml link elements. This is an alternative to placing hreflang in page headers, and it scales better for large sites. Each URL must list all alternates, including itself, and the annotations must be reciprocal. The risks of misconfiguration appear in the discussion of hreflang and IP redirection.

When is the sitemap method better than HTML annotations?

When pages are numerous, templates are hard to change or alternates vary by URL. It keeps annotation management in one place.

What about news sitemaps?

Publishers who qualify for Google News can use news sitemaps to list recent articles. They apply to news content only and have their own rules, such as a limited time window for listed articles.

Should a normal business blog use a news sitemap?

Typically not. A standard XML sitemap with accurate lastmod values suits most blogs.

How should large sites structure and segment sitemaps?

Split sitemaps by content type or section, keep each file well under the limits and group them with a sitemap index. Segmentation turns the Sitemaps and Page indexing reports into a diagnostic tool.

How do you segment effectively?

Create separate sitemaps for products, categories, articles, locations and static pages, and split any group that approaches the limit into numbered files. For very large catalogs, split by category or by date range. Update each file independently, so that only changed sections need regenerating.

Why does segmentation help diagnosis?

Because you can filter the Page indexing report by sitemap and see the indexed share per section. If products index at 95 per cent and locations at 30 per cent, you know where to look. The diagnostic method in the guide to Crawled and Discovered statuses uses exactly this comparison.

How do sitemaps relate to crawl budget?

Google's crawl budget guidance for large sites mentions sitemaps as a way to tell Google which pages matter, together with accurate lastmod values, efficient servers and removing low-value URLs. A sitemap cluttered with junk wastes attention. A clean, accurate one concentrates it. Server speed also matters, as the notes on server response time and crawl frequency show.

Should low-value pages be removed from the sitemap?

Yes, and from the site if they serve no purpose. The decision rules in the content pruning plan apply.

How do you automate sitemap generation?

Generate sitemaps from your database or CMS at publish time, with rules that include only canonical, indexable, published URLs and use real modification dates. Validate in the build, and fail the build on malformed files. This site's build generates its sitemap from the same page list that produces the pages, which prevents drift.

What if you use a plugin?

Check its output. Plugins sometimes include noindexed pages, tags and attachment URLs, or update lastmod on every save. Configure exclusions and verify the result.

How should ecommerce, local and publishing sites handle sitemaps?

The rules are the same, but each type of site has typical traps: product churn for stores, thin location pages for local firms and sprawling archives for publishers. Tailor the generation rules to each.

What should ecommerce sites do?

Include canonical product, category and key landing page URLs, and remove products as soon as they are permanently discontinued, redirecting or returning 404 or 410 as appropriate. Keep temporarily out-of-stock products listed only if the page remains useful and indexable. Exclude faceted and parameter URLs unless you have chosen them as landing pages, and segment by product, category and brand pages so that the Page indexing report shows where problems cluster. The product-level consistency described in the comparison of Merchant Center feeds and structured data is easier when the sitemap lists the same canonical URLs as the feed.

How do you handle seasonal products?

Keep the same URL year after year when you can, update lastmod only when content changes meaningfully and leave the page in the sitemap if it remains indexable.

What should local and service businesses do?

List the home page, each service page, each real location page, key guides and the contact page. Exclude duplicate or thin location pages, and make sure each listed location has unique content, as discussed in the analysis of scaled content abuse. Keep location URLs stable so that your Business Profile and citations never point at redirected addresses.

Do small local sites need more than a basic sitemap?

Rarely. A single accurate XML sitemap, referenced in robots.txt and submitted in Search Console, is enough for most sites with fewer than a few hundred pages.

What should publishers and blogs do?

Include evergreen articles, hubs and author pages, and use accurate modification dates that reflect real revisions. Consider an RSS or Atom feed for recent posts and, for news publishers, a news sitemap. Prune or consolidate thin tag archives and old duplicates, and keep them out of the sitemap while you decide.

Should author and category pages be listed?

If they have unique value and are indexable, yes. If they are thin lists with no original content, leave them out.

How do CMS platforms, plugins and static generators produce sitemaps?

Most platforms generate sitemaps automatically, but defaults often include pages you would not choose. Review the output of whichever system you use, and configure exclusions and modification dates deliberately.

What should you check in a CMS or plugin?

Whether noindexed pages are excluded, whether tag, author and attachment pages are listed, how lastmod is calculated, whether the sitemap splits at sensible sizes, whether the index file is referenced in robots.txt and whether custom post types or product variants appear as you expect. Test the output with a list-mode crawl rather than trusting the settings screen.

What if the plugin cannot be configured?

Switch plugins, generate a custom sitemap from your database or add a filter that removes unwanted URLs before output. A broken sitemap is worse than a simple one.

How do static site generators and headless sites handle it?

Generate the sitemap at build time from the same list of pages used to render the site, so the two cannot disagree. Pull real modification dates from content metadata, validate the file in the build and fail the build on errors. This approach, used on this site, also keeps staging URLs and unpublished drafts out of the file automatically.

How do JavaScript-heavy sites fit in?

Sitemaps are especially helpful there, because they list URLs that crawlers might not discover through scripts. Still make sure the listed pages render correctly and contain real content, as the guide to client-side versus server-side rendering explains.

How do multi-site and subdomain setups work?

Give each host its own sitemap, referenced in that host's robots.txt, and verify each in Search Console. Avoid listing URLs from other hosts unless you have verified both and understand the cross-submission rules.

What about international sites?

Create sitemaps per language or region, or use hreflang annotations in the sitemap, and keep each URL's canonical and alternates reciprocal.

How do you audit a sitemap step by step?

Fetch the sitemap, crawl every listed URL, compare status codes, canonicals and robots directives with the list and review the Sitemaps and Page indexing reports. The audit exposes every contradiction between your intent and your implementation.

What are the audit steps?

1. Confirm the sitemap URL returns 200, is valid XML, UTF-8 encoded and within the size limits.
2. Extract all URLs and crawl them in list mode, recording status code, canonical, meta robots and X-Robots-Tag.
3. Flag redirected, 4xx, 5xx, noindexed and non-canonical URLs, and remove or fix them.
4. Compare the list with a full-site crawl to find indexable pages missing from the sitemap.
5. Check lastmod values for accuracy against real content changes.
6. Verify that robots.txt references the sitemap and does not block listed URLs.
7. Review Search Console's Sitemaps report for errors and the Page indexing report, filtered by sitemap, for exclusions.

How often should you audit?

Quarterly for most sites, monthly for large or fast-changing ones, and after any migration. The checks fit within a wider technical SEO audit.

Which findings are the most common?

Redirected URLs left over after a migration, noindexed pages listed by a plugin, parameter variants, http and https mixtures, stale lastmod values and URLs blocked in robots.txt. Each is a quick win once identified, and the process in the website migration checklist shows how to regenerate sitemaps after moves.

What do you do with orphan pages found through the sitemap comparison?

Link them from relevant pages if they deserve to be indexed, or remove them from the sitemap and decide whether to retire them, as the method for orphan page identification describes.

How do you troubleshoot sitemap errors in Search Console?

Start with the status in the Sitemaps report, read the specific error message, fix the cause at the source and resubmit only if the file location changed. Most errors fall into a handful of patterns.

Which errors appear most often?

"Couldn't fetch" usually means the URL is unreachable, blocked, redirected or slow. "Has errors" points to invalid XML, wrong encoding, unescaped characters or URLs on a different host. "Submitted URL marked noindex" and "Submitted URL has crawl issue" in the Page indexing report show contradictions between the sitemap and the pages.

How do you fix a fetch failure?

Open the sitemap URL in a browser and in a plain HTTP client, confirm a 200 status and an XML content type, check robots.txt and firewall rules for blocking of Googlebot and make sure the file is not served only to logged-in users.

What do exclusions for submitted URLs tell you?

If submitted URLs appear as "Crawled - currently not indexed", the pages may be thin or duplicate. If they appear as "Duplicate, Google chose different canonical than user", your signals disagree. If they appear as redirects or 404s, your generator includes stale URLs. Filter the report by sitemap and read each reason.

Should you remove excluded URLs from the sitemap?

Only if you do not want them indexed. If you do, fix the page quality or canonical problem and leave them listed.

What does a sitemap clean-up look like in practice?

This is a fictional example that illustrates the method. The outcomes are invented.

What was the situation?

A fictional bathroom retailer submitted a single sitemap of 38,000 URLs. Search Console showed only 14,000 indexed. The sitemap was generated by a plugin that updated lastmod to the current date on every build.

What did the audit find?

A list-mode crawl found 5,200 redirected URLs, 3,100 noindexed tag and filter pages, 9,000 parameter variants and 2,300 URLs that returned 404 after a catalog change. Many pages had self-conflicting canonicals, and lastmod showed the same date everywhere.

What did they change?

They rebuilt the generator to include only canonical, indexable, 200-status URLs, stored genuine modification dates, split the output into products, categories, articles and location sitemaps under a sitemap index, referenced the index in robots.txt and submitted it in Search Console. They removed the junk URL sets from the site where possible, and redirected or removed the rest. They monitored the indexed share per sitemap monthly, noting that changes took several weeks and could not be guaranteed.

What did they learn?

The headline count of submitted URLs fell sharply, and the indexed share rose. A smaller, honest sitemap told Google what mattered, and made the next problem easy to see.

Which sitemap mistakes should you avoid?

Most sitemap problems come from automatic generation without rules. Adding a few filters and checks prevents nearly all of them.

What do sites get wrong most often?

Listing redirected, noindexed or blocked URLs, mixing protocols and hosts, using relative URLs, updating lastmod on every build, exceeding limits, forgetting to update after a migration, blocking the sitemap in robots.txt, submitting sitemaps for the wrong property and treating a sitemap as a way to force indexing.

Which mistake is the most damaging?

Untrustworthy lastmod and junk URLs together. They teach Google to ignore the file, which removes its main benefit.

How do you keep sitemaps healthy?

Generate from a single source of truth, validate on every build, monitor the Sitemaps report, review exclusions monthly and re-audit after structural changes.

Who owns the sitemap?

Developers own generation, SEO owns the rules and the checks, and editors own the real modification dates.

How does AH SEO Agency help with XML sitemaps?

We audit your sitemaps against live status codes, canonicals and directives, redesign generation rules and segmentation, correct lastmod handling, set up Search Console submission and monitor indexing by section. We tell you plainly when a sitemap is not the cause of an indexing problem.

Where do you start?

Book a technical SEO review, order a full SEO audit or start with a free SEO audit that includes a sitemap check.

Summary: how do you build a sitemap Google trusts?

List only canonical, indexable, 200-status URLs, keep files within limits, use absolute UTF-8 URLs, set truthful lastmod values, ignore priority and changefreq, submit through Search Console and robots.txt, and segment large sites for diagnosis.

• Limits are 50 MB uncompressed or 50,000 URLs per file, with sitemap index files above that.
• Exclude redirected, noindexed, blocked and duplicate URLs.
• Lastmod is used only when accurate, and priority and changefreq are ignored.
• Submission is a hint, not a guarantee of crawling or indexing.
• Audit regularly and compare submitted with indexed URLs.