sitemap-xml-parser

sitemap-xml-parser

Parses sitemap XML files and returns all listed URLs. Can be used as a CLI tool or a Node.js library.

• Follows sitemap index files recursively and decompresses gzip automatically
• Supports custom request headers, concurrency control, and request timeouts
• CLI: outputs plain URLs, TSV, or JSON with configurable field selection (--fields)
• CLI: filters URLs by substring or regular expression

Installation

npm install sitemap-xml-parser

CLI

Run without installing via npx:

npx sitemap-xml-parser <url> [options]

Or, after installing globally (npm install -g sitemap-xml-parser):

sitemap-xml-parser <url> [options]

Fetched URLs are printed to stdout, one per line. Errors are printed to stderr. See Options for available flags.

Examples

# Print all URLs
npx sitemap-xml-parser https://example.com/sitemap.xml

# Save URLs to a file, errors to a log
npx sitemap-xml-parser https://example.com/sitemap.xml > urls.txt 2> errors.log

# Count URLs
npx sitemap-xml-parser https://example.com/sitemap.xml --count

# Stop after 100 entries
npx sitemap-xml-parser https://example.com/sitemap.xml --cap 100

# Filter and count
npx sitemap-xml-parser https://example.com/sitemap.xml --filter "blog" --count

# Filter by regular expression
npx sitemap-xml-parser https://example.com/sitemap.xml --filter-regex "blog/[0-9]{4}/"

# Output as TSV (loc, lastmod, changefreq, priority)
npx sitemap-xml-parser https://example.com/sitemap.xml --format tsv

# Output as JSON with all fields (all fields present in the source XML are included by default)
npx sitemap-xml-parser https://example.com/sitemap.xml --format json

# Discover all fields present in a sitemap
npx sitemap-xml-parser https://example.com/sitemap.xml --list-fields

# Output as TSV with custom columns (e.g. image sitemap extension)
npx sitemap-xml-parser https://example.com/sitemap.xml --format tsv --fields loc,image:image

# Output as TSV with all fields (fetches twice: once to discover fields, once to output)
npx sitemap-xml-parser https://example.com/sitemap.xml --format tsv \
  --fields "$(npx sitemap-xml-parser https://example.com/sitemap.xml --list-fields | paste -sd, -)"

Options

CLI

Library

Features

• Follows Sitemap Index files recursively, including nested indexes (Index within an Index)
• Automatically decompresses gzip: supports both .gz URLs and Content-Encoding: gzip responses
• Batch processing: fetches limit child sitemaps in parallel per batch, then waits delay ms after each batch completes
• Automatically follows redirects (301/302/303/307/308) up to 5 hops; errors beyond that are reported via onError. Custom request headers are forwarded only when the redirect stays on the same origin (same scheme, host, and port); they are stripped on cross-origin redirects.

Usage

const SitemapXMLParser = require('sitemap-xml-parser');

const parser = new SitemapXMLParser('https://example.com/sitemap.xml');

(async () => {
    const urls = await parser.fetch();
    urls.forEach(entry => {
        console.log(entry.loc);
    });
})();

Custom headers

const parser = new SitemapXMLParser('https://example.com/sitemap.xml', {
    headers: {
        'User-Agent': 'MyBot/2.0',
        'Authorization': 'Bearer my-token',
    },
});

Error handling with onError

Failed URLs (network errors, non-2xx responses, malformed XML) are skipped by default. Provide an onError callback to inspect them:

const parser = new SitemapXMLParser('https://example.com/sitemap.xml', {
    onError: (url, err) => {
        console.error(`Skipped ${url}: ${err.message}`);
    },
});

Return value

fetch() resolves to an array of URL entry objects. Each object contains all fields present in the source XML — no field selection is applied at the library level:

[
  {
    loc:        'https://example.com/page1',
    lastmod:    '2024-01-01',
    changefreq: 'weekly',
    priority:   '0.8',
  },
  // ...
]

loc is always a string. Standard fields (lastmod, changefreq, priority) are strings when present, or undefined when absent from the source XML.

Sitemap extension fields (e.g. image:image, news:news, video:video) are also preserved as-is when present in the source XML. Their values reflect the structure parsed by the underlying XML parser — nested elements become objects.