How to Extract All URLs from a Sitemap in 2026

Before you can crawl, audit, or feed a website into a RAG pipeline, you need the one thing every site already publishes for you: a complete list of its public URLs. That list lives in sitemap.xml — but on any real site it’s never a single flat file. It’s a sitemap index pointing to dozens of nested sitemaps, sometimes nested again, sometimes gzipped, sometimes split by content type. This guide covers how to discover the entry point, follow the nesting recursively, and produce a clean deduplicated URL list — the boring-but-essential first step of almost every web data project.

Why the sitemap, not a crawl

You could discover URLs by spidering links from the homepage, but that’s slow, expensive, and incomplete — orphan pages with no inbound links never get found. The sitemap is the site owner’s own declaration of what exists. It’s faster (no rendering, no link extraction), more complete (includes pages with no internal links), and politer (one XML fetch instead of crawling every page). For building a URL inventory, it’s strictly the better starting point.

The standard discovery path:

# 1. The conventional location
https://example.com/sitemap.xml

# 2. Declared in robots.txt
https://example.com/robots.txt   ->   Sitemap: https://example.com/sitemap_index.xml

# 3. Common CMS-specific names
https://example.com/sitemap_index.xml      # WordPress / Yoast
https://example.com/sitemap.xml?page=1     # Shopify
https://example.com/wp-sitemap.xml         # WordPress core

A good crawler auto-discovers the entry point: it checks robots.txt first (the authoritative pointer), then falls back to the conventional locations.

The two XML shapes you must handle

Sitemaps come in exactly two flavors, and the recursion is all about telling them apart:

<!-- A sitemap INDEX: points to other sitemaps. Recurse into each. -->
<sitemapindex>
  <sitemap><loc>https://example.com/post-sitemap.xml</loc></sitemap>
  <sitemap><loc>https://example.com/page-sitemap.xml</loc></sitemap>
  <sitemap><loc>https://example.com/product-sitemap1.xml</loc></sitemap>
</sitemapindex>

<!-- A URL SET: the leaf, contains actual page URLs. Extract these. -->
<urlset>
  <url>
    <loc>https://example.com/blog/how-to-bake-bread</loc>
    <lastmod>2026-05-20</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

The crawler’s job: fetch the entry point, detect whether it’s a <sitemapindex> or a <urlset>, recurse into every <sitemap> it finds, extract every <url> from the leaves, and deduplicate the lot (large sites routinely list the same URL in multiple sitemaps). Big e-commerce catalogs can nest three levels deep and span hundreds of sitemap files with hundreds of thousands of URLs total.

A subtle gotcha: many sitemaps are served gzipped (sitemap.xml.gz) even when the index references them without the extension. The fetcher has to transparently decompress.

▶ Run the Sitemap to URL Crawler — recursively follows nested sitemap indexes, decompresses gzipped sitemaps, deduplicates, and returns a flat list of every public URL plus optional SEO metadata. Browserless, zero-config, no proxy required.

No browser, no proxy — why it’s cheap

Sitemaps are static XML. There’s no JavaScript to execute, no anti-bot challenge to solve, no login. That means the crawler is pure HTTP fetching with XML parsing — no headless browser, and for most sites no proxy required. This is the cheapest category of scraping there is: the cost is dominated by the number of sitemap files fetched, not by compute or bandwidth. A 200,000-URL site might be 150 sitemap fetches.

Proxy support exists for the rare case of a site that rate-limits sitemap requests or geo-restricts, but you’ll rarely need it.

The SEO metadata bonus

Beyond the URL itself, the <url> element optionally carries three attributes that are gold for SEO and content work:

• lastmod — when the page was last modified. Diff this across runs to detect new and updated pages without re-crawling content.
• changefreq — the owner’s hint about update cadence (daily, monthly, etc.).
• priority — the owner’s relative importance score (0.0–1.0).

A crawler that extracts these alongside the URL turns a plain link list into a freshness-aware content inventory.

Schema design for downstream use

{
  "url": "https://example.com/blog/how-to-bake-bread",
  "lastmod": "2026-05-20",
  "changefreq": "monthly",
  "priority": 0.8,
  "source_sitemap": "https://example.com/post-sitemap.xml",
  "scraped_at": "2026-05-28T12:00:00Z"
}

Choices worth making early:

• Keep source_sitemap. Knowing which child sitemap a URL came from is invaluable for diagnosing coverage gaps and segmenting by content type (posts vs. products vs. pages).
• Preserve lastmod even when null. Its absence is itself a signal — sites with no lastmod can’t be diffed for freshness.
• Deduplicate on url. The same URL appearing in multiple sitemaps is normal; collapse to one row.
• Stamp scraped_at. So you can compute deltas between runs for content monitoring.

Typical use cases

• RAG and LLM ingestion — generate the complete URL list that feeds a content-extraction and vector-database pipeline. You can’t embed pages you never discovered.
• AI browsing agents — hand an agent a full domain map so it can locate and retrieve relevant pages instead of blindly crawling.
• SEO audits — validate sitemap coverage and freshness at scale; find sections missing from the sitemap or stale lastmod values.
• Competitor analysis — enumerate a competitor’s entire published footprint: every blog post, product, category, and landing page.
• Content monitoring — diff lastmod across runs to detect new or updated pages automatically.
• Site migrations and QA — confirm post-migration that every old URL still has a home and nothing got dropped.

The common thread: this is infrastructure. The URL list isn’t the deliverable — it’s what every subsequent step depends on.

Cost math

This actor prices per dataset item (per URL extracted), and because there’s no browser or proxy, the per-URL cost is about as low as scraping gets. A typical job — a 50,000-URL content site — is a few cents to a couple of dollars depending on pricing tier, and small sites (a few thousand URLs) fit comfortably inside Apify’s free monthly credit.

Building it yourself is a half-day of work, but you’d own: robots.txt parsing, the index-vs-urlset detection, recursion with depth limits and loop protection, gzip decompression, dedupe, and the metadata extraction. It’s the kind of utility that’s easy to write once and annoying to keep robust against every CMS’s sitemap quirks.

Common pitfalls

• Stopping at the first file. If you treat the sitemap index as a urlset, you’ll extract a handful of sitemap URLs and miss the hundreds of thousands of page URLs behind them. Detect the type.
• Forgetting gzip. A surprising number of large sites serve .xml.gz. Skip decompression and you’ll parse garbage or get nothing.
• No loop protection. Misconfigured sites occasionally reference a sitemap that references itself. Cap recursion depth and track visited URLs.
• Ignoring robots.txt. The conventional /sitemap.xml doesn’t always exist; the authoritative location is often only declared in robots.txt.
• Not deduplicating. Counting duplicate URLs inflates your “page count” and wastes downstream embedding cost.
• Assuming a sitemap is complete. Owners sometimes exclude pages. The sitemap is the best starting point, not a guarantee of totality.

Wrapping up

Extracting every URL from a sitemap sounds like a one-liner until you meet a real site with three levels of gzipped, content-typed sitemap indexes. It’s the unglamorous foundation under RAG ingestion, SEO audits, and competitive crawling — and getting it complete and deduplicated is what makes everything downstream trustworthy. Write the recursion yourself for a one-off, or use a zero-config managed crawler when you want a clean URL list without babysitting CMS quirks.

▶ Open the Sitemap to URL Crawler on Apify — recursive, gzip-aware, deduplicated URL extraction with optional SEO metadata. No browser, no proxy, pay per URL. Start with Apify’s free monthly credit.