@excsn/sitemap v5.1.1

sitemap.js

sitemap.js is a high-level sitemap-generating library/CLI that makes creating sitemap XML files easy.

Maintainers

• @ekalinin
• @derduher

Table of Contents

• Installation
• Usage
  • CLI
  • Example of using sitemap.js with express
  • Stream writing a sitemap
  • Example of most of the options you can use for sitemap
  • Building just the sitemap index file
  • Auto creating sitemap and index files from one large list
• API
  • Sitemap
  • buildSitemapIndex
  • createSitemapsAndIndex
  • xmlLint
  • parseSitemap
  • SitemapStream
  • XMLToISitemapOptions
  • lineSeparatedURLsToSitemapOptions
  • streamToPromise
  • ObjectStreamToJSON
  • Sitemap Item Options
  • ISitemapImage
  • IVideoItem
  • ILinkItem
  • INewsItem
• License

Installation

npm install --save sitemap

Usage

CLI

Just feed the list of urls into sitemap

npx sitemap < listofurls.txt

Or validate an existing sitemap (requires libxml)

npx sitemap --validate sitemap.xml

Or take an existing sitemap and turn it into options that can be fed into the libary

npx sitemap --parse sitemap.xml

Or prepend some new urls to an existing sitemap

npx sitemap --prepend sitemap.xml < listofurls.json # or txt

As a library

const { SitemapStream, streamToPromise } = require('../dist/index')
// Creates a sitemap object given the input configuration with URLs
const sitemap = new SitemapStream({ hostname: 'http://example.com' });
sitemap.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
sitemap.write('/page-2')
sitemap.end()

streamToPromise(sitemap)
  .then(sm => console.log(sm.toString()))
  .catch(console.error);

Resolves to a string containing the XML data

 <?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"><url><loc>http://example.com/page-1/</loc><changefreq>daily</changefreq><priority>0.3</priority></url><url><loc>http://example.com/page-2</loc></url></urlset>

Example of using sitemap.js with express

const express = require('express')
const { SitemapStream, streamToPromise } = require('sitemap')
const { createGzip } = require('zlib')

const app = express()
let sitemap

app.get('/sitemap.xml', function(req, res) {
  res.header('Content-Type', 'application/xml');
  res.header('Content-Encoding', 'gzip');
  // if we have a cached entry send it
  if (sitemap) {
    res.send(sitemap)
    return
  }
  try {
    const smStream = new SitemapStream({ hostname: 'https://example.com/' })
    const pipeline = smStream.pipe(createGzip())

    smStream.write({ url: '/page-1/',  changefreq: 'daily', priority: 0.3 })
    smStream.write({ url: '/page-2/',  changefreq: 'monthly',  priority: 0.7 })
    smStream.write({ url: '/page-3/'})    // changefreq: 'weekly',  priority: 0.5
    smStream.write({ url: '/page-4/',   img: "http://urlTest.com" })
    smStream.end()

    // cache the response
    streamToPromise(pipeline).then(sm => sitemap = sm)
    // stream the response
    pipeline.pipe(res).on('error', (e) => {throw e})
  } catch (e) {
    console.error(e)
    res.status(500).end()
  }
})

app.listen(3000, () => {
  console.log('listening')
});

Stream writing a sitemap

The sitemap stream is around 20% faster and only uses ~10% the memory of the traditional interface

const fs = require('fs');
const { SitemapStream } = require('sitemap')
// external libs provided as example only
const { parser } = require('stream-json/Parser');
const { streamArray } = require('stream-json/streamers/StreamArray');
const { streamValues } = require('stream-json/streamers/StreamValues');
const map = require('through2-map')
const { createGzip } = require('zlib')

// parsing line separated json or JSONStream
const pipeline = fs
  .createReadStream("./tests/mocks/perf-data.json.txt"),
  .pipe(parser())
  .pipe(streamValues())
  .pipe(map.obj(chunk => chunk.value))
  // SitemapStream does the heavy lifting
  // You must provide it with an object stream
  .pipe(new SitemapStream());

// parsing JSON file
const pipeline = fs
  .createReadStream("./tests/mocks/perf-data.json")
  .pipe(parser())
  .pipe(streamArray())
  .pipe(map.obj(chunk => chunk.value))
  // SitemapStream does the heavy lifting
  // You must provide it with an object stream
  .pipe(new SitemapStream({ hostname: 'https://example.com/' }))
  .pipe(process.stdout)

//
// coalesce into value for caching
//
  let cachedXML
  streamToPromise(
    fs.createReadStream("./tests/mocks/perf-data.json")
    .pipe(parser())
    .pipe(streamArray())
    .pipe(map.obj(chunk => chunk.value))
    .pipe(new SitemapStream({ hostname: 'https://example.com/' }))
    .pipe(createGzip())
  ).then(xmlBuffer => cachedXML = xmlBuffer)

Example of most of the options you can use for sitemap

const { SitemapStream, streamToPromise } = require('sitemap');
const smStream = new SitemapStream({ hostname: 'http://www.mywebsite.com' })
// coalesce stream to value
// alternatively you can pipe to another stream
streamToPromise(smStream).then(console.log)

smStream.write({
  url: '/page1',
  changefreq: 'weekly',
  priority: 0.8,
  lastmodfile: 'app/assets/page1.html'
})

smStream.write({
  url: '/page2',
  changefreq: 'weekly',
  priority: 0.8,
  /* useful to monitor template content files instead of generated static files */
  lastmodfile: 'app/templates/page2.hbs'
})

// each sitemap entry supports many options
// See [Sitemap Item Options](#sitemap-item-options) below for details
smStream.write({
  url: 'http://test.com/page-1/',
  img: [
    {
      url: 'http://test.com/img1.jpg',
      caption: 'An image',
      title: 'The Title of Image One',
      geoLocation: 'London, United Kingdom',
      license: 'https://creativecommons.org/licenses/by/4.0/'
    },
    {
      url: 'http://test.com/img2.jpg',
      caption: 'Another image',
      title: 'The Title of Image Two',
      geoLocation: 'London, United Kingdom',
      license: 'https://creativecommons.org/licenses/by/4.0/'
    }
  ],
  video: [
    {
      thumbnail_loc: 'http://test.com/tmbn1.jpg',
      title: 'A video title',
      description: 'This is a video'
    },
    {
      thumbnail_loc: 'http://test.com/tmbn2.jpg',
      title: 'A video with an attribute',
      description: 'This is another video',
      'player_loc': 'http://www.example.com/videoplayer.mp4?video=123',
      'player_loc:autoplay': 'ap=1'
    }
  ],
  links: [
    { lang: 'en', url: 'http://test.com/page-1/' },
    { lang: 'ja', url: 'http://test.com/page-1/ja/' }
  ],
  androidLink: 'android-app://com.company.test/page-1/',
  news: {
    publication: {
      name: 'The Example Times',
      language: 'en'
    },
    genres: 'PressRelease, Blog',
    publication_date: '2008-12-23',
    title: 'Companies A, B in Merger Talks',
    keywords: 'business, merger, acquisition, A, B',
    stock_tickers: 'NASDAQ:A, NASDAQ:B'
  }
})
// indicate there is nothing left to write
smStream.end()

Building just the sitemap index file

The sitemap index file merely points to other sitemaps

const { buildSitemapIndex } = require('sitemap')
const smi = buildSitemapIndex({
  urls: ['https://example.com/sitemap1.xml', 'https://example.com/sitemap2.xml'],
  xslUrl: 'https://example.com/style.xsl' // optional
})

Auto creating sitemap and index files from one large list

const { createSitemapsAndIndex } = require('sitemap')
const smi = createSitemapsAndIndex({
  hostname: 'http://www.sitemap.org',
  sitemapName: 'sm-test',
  sitemapSize: 1,
  targetFolder: require('os').tmpdir(),
  urls: ['http://ya.ru', 'http://ya2.ru']
})

API

Sitemap

const { Sitemap } = require('sitemap')
const sm = new Sitemap({
    urls: [{ url: '/path' }],
    hostname: 'http://example.com',
    cacheTime: 0, // default
    level: 'warn', // default warns if it encounters bad data
    lastmodDateOnly: false // relevant for baidu
})
sm.toString() // returns the xml as a string

toString

sm.toString(true)

Converts the urls stored in an instance of Sitemap to a valid sitemap xml document as a string. Accepts a boolean as its first argument to designate on whether to pretty print. Defaults to false.

toXML

alias for toString

toGzip

sm.toGzip ((xmlGzippedBuffer) => console.log(xmlGzippedBuffer))
sm.toGzip()

Like toString, it builds the xmlDocument, then it runs gzip on the resulting string and returns it as a Buffer via callback or direct invocation

clearCache

sm.clearCache()

Cache will be emptied and will be bypassed until set again

isCacheValid

sm.isCacheValid()

Returns true if it has been less than cacheTimeout ms since cache was set

setCache

sm.setCache('...xmlDoc')

Stores the passed in string on the instance to be used when toString is called within the configured cacheTimeout returns the passed in string unaltered

add

sm.add('/path', 'warn')

Adds the provided url to the sitemap instance takes an optional parameter level for whether to print a console warning in the event of bad data 'warn' (default), throw an exception 'throw', or quietly ignore bad data 'silent' returns the number of locations currently in the sitemap instance

contains

sm.contains('/path')

Returns true if path is already a part of the sitemap instance, false otherwise.

del

sm.del('/path')

Removes the provided url or url option from the sitemap instance

normalizeURL

Sitemap.normalizeURL('/', 'http://example.com', false)

Static function that returns the stricter form of a options passed to SitemapItem. The third argument is whether to use date-only varient of lastmod. For baidu.

normalizeURLs

Sitemap.normalizeURLs(['http://example.com', {url: '/'}], 'http://example.com', false)

Static function that takes an array of urls and returns a Map of their resolved url to the strict form of SitemapItemOptions

buildSitemapIndex

Build a sitemap index file

const { buildSitemapIndex } = require('sitemap')
const index = buildSitemapIndex({
  urls: [{ url: 'http://example.com/sitemap-1.xml', lastmod: '2019-07-01' }, 'http://example.com/sitemap-2.xml'],
  lastmod: '2019-07-29'
})

createSitemapsAndIndex

Create several sitemaps and an index automatically from a list of urls

const { createSitemapsAndIndex } = require('sitemap')
createSitemapsAndIndex({
  urls: [/* list of urls */],
  targetFolder: 'absolute path to target folder',
  hostname: 'http://example.com',
  cacheTime: 600,
  sitemapName: 'sitemap',
  sitemapSize: 50000, // number of urls to allow in each sitemap
  gzip: true, // whether to gzip the files
})

xmlLint

Resolve or reject depending on whether the passed in xml is a valid sitemap. This is just a wrapper around the xmlLint command line tool and thus requires xmlLint.

const { createReadStream } = require('fs')
const { xmlLint } = require('sitemap')
xmlLint(createReadStream('./example.xml')).then(
  () => console.log('xml is valid'),
  ([err, stderr]) => console.error('xml is invalid', stderr)
)

parseSitemap

Read xml and resolve with the configuration that would produce it or reject with an error

const { createReadStream } = require('fs')
const { parseSitemap, createSitemap } = require('sitemap')
parseSitemap(createReadStream('./example.xml')).then(
  // produces the same xml
  // you can, of course, more practically modify it or store it
  (xmlConfig) => console.log(createSitemap(xmlConfig).toString()),
  (err) => console.log(err)
)

SitemapStream

A Transform for turning a Readable stream of either SitemapItemOptions or url strings into a Sitemap. The readable stream it transforms must be in object mode.

const { SitemapStream } = require('sitemap')
const sms = new SitemapStream({
  hostname: 'https://example.com', // optional only necessary if your paths are relative
  lastmodDateOnly: false // defaults to false, flip to true for baidu
})
const readable = // a readable stream of objects
readable.pipe(sms).pipe(process.stdout)

XMLToISitemapOptions

Takes a stream of xml and transforms it into a stream of ISitemapOptions. Use this to parse existing sitemaps into config options compatible with this library

const { createReadStream, createWriteStream } = require('fs');
const { XMLToISitemapOptions, ObjectStreamToJSON } = require('sitemap');

createReadStream('./some/sitemap.xml')
// turn the xml into sitemap option item options
.pipe(new XMLToISitemapOptions())
// convert the object stream to JSON
.pipe(new ObjectStreamToJSON())
// write the library compatible options to disk
.pipe(createWriteStream('./sitemapOptions.json'))

lineSeparatedURLsToSitemapOptions

Takes a stream of urls or sitemapoptions likely from fs.createReadStream('./path') and returns an object stream of sitemap items.

streamToPromise

Takes a stream returns a promise that resolves when stream emits finish.

const { streamToPromise, SitemapStream } = require('sitemap')
const sitemap = new SitemapStream({ hostname: 'http://example.com' });
sitemap.write({ url: '/page-1/', changefreq: 'daily', priority: 0.3 })
sitemap.end()
streamToPromise(sitemap).then(buffer => console.log(buffer.toString())) // emits the full sitemap

ObjectStreamToJSON

A Transform that converts a stream of objects into a JSON Array or a line separated stringified JSON.

• @param lineSeparated=false whether to separate entries by a new line or comma

const stream = Readable.from([{a: 'b'}])
  .pipe(new ObjectStreamToJSON())
  .pipe(process.stdout)
stream.end()
// prints {"a":"b"}

Sitemap Item Options

ISitemapImage

Sitemap image https://support.google.com/webmasters/answer/178636?hl=en&ref_topic=4581190

IVideoItem

Sitemap video. https://support.google.com/webmasters/answer/80471?hl=en&ref_topic=4581190

ILinkItem

https://support.google.com/webmasters/answer/189077

INewsItem

https://support.google.com/webmasters/answer/74288?hl=en&ref_topic=4581190

License

See LICENSE file.