@bluecadet/launchpad-content NPM

Launchpad Content

The content package downloads and locally caches content from various common web APIs.

To download content, all you need to do is define content sources and provide credentials as needed.

The following launchpad.json would download jsons and images from the Flickr API to .downloads/flickr-images (a combination of the default .downloads/ directory and the id field of the content source):

{
  "content": {
    "sources": [
      {
        "id": "flickr-images",
        "type": "json",
        "files": {
            "spaceships.json": "https://api.flickr.com/services/feeds/photos_public.gne?format=json&nojsoncallback=1&tags=spaceship",
            "rockets.json": "https://api.flickr.com/services/feeds/photos_public.gne?format=json&nojsoncallback=1&tags=rocket"
        }
      }
    ]
  }
}

See ContentOptions Parameters for a full list of content settings.

Source Settings

Every source needs:

id: ID of the individual source (required)
type: Source type (default: json)
Additional source-specific settings

Currently supported sources are:

Authentication

Some content sources require credentials to access their APIs.

These can all be stored in a .env or .env.local file which will be automatically loaded by launchpad.

`.env.local`

AIRTABLE_API_KEY=<YOUR_AIRTABLE_API_KEY>

CONTENTFUL_PREVIEW_TOKEN=<YOUR_CONTENTFUL_PREVIEW_TOKEN>
CONTENTFUL_DELIVERY_TOKEN=<YOUR_CONTENTFUL_DELIVERY_TOKEN>
CONTENTFUL_USE_PREVIEW_API=false

SANITY_API_TOKEN=<YOUR_API_TOKEN>

STRAPI_IDENTIFIER=<YOUR_API_USER>
STRAPI_PASSWORD=<YOUR_API_PASS>

`launchpad.config.js`

export default defineConfig({
	content: {
		sources: [
			{
				id: "airtable-cms",
				type: "airtable",
				apiKey: process.env.AIRTABLE_API_KEY,
			},
			{
				id: "contentful-cms",
				type: "contentful",
				previewToken: process.env.CONTENTFUL_PREVIEW_TOKEN,
				deliveryToken: process.env.CONTENTFUL_DELIVERY_TOKEN,
				usePreviewApi: false,
			},
			{
				id: "sanity-cms",
				type: "sanity",
				apiToken: process.env.SANITY_API_TOKEN,
			},
			{
				id: "strapi-cms",
				type: "strapi",
				identifier: process.env.STRAPI_IDENTIFIER,
			},
		],
	},
});

Post Processing

Once content is downloaded it can be processed to transform text and images. For example, launchpad can convert markdown to html and create scaled derivatives of each image.

ContentOptions

Options for all content and media downloads. Each of these settings can also be configured per ContentSource.

Property	Type	Default	Description
`sources`	Array.<SourceOptions>	[]	A list of content source options. This defines which content is downloaded from where.
`imageTransforms`	Array.<Object.<string, number>>	[]	A list of image transforms to apply to a copy of each downloaded image.
`contentTransforms`	Object.<string, string>	{}	A list of content transforms to apply to all donwloaded content.
`downloadPath`	string	'.downloads/'	The path at which to store all downloaded files.
`credentialsPath`	string	'.credentials.json'	The path to the json containing credentials for all content sources.
`tempPath`	boolean	'%DOWNLOAD_PATH%/.tmp/'	Temp file directory path.
`backupPath`	boolean	'%DOWNLOAD_PATH%/.backups/'	Temp directory path where all downloaded content will be backed up before removal.
`keep`	boolean	''	Which files to keep in `dest` if `clearOldFilesOnSuccess` or `clearOldFilesOnStart` are `true`. E.g. `'\.json\\|\.csv\\|\.xml\\|\.git\*'`
`strip`	string	''	Strips this string from all media file paths when saving them locally
`backupAndRestore`	boolean	true	Back up files before downloading and restore originals for all sources on failure of any single source.
`maxConcurrent`	number	4	Max concurrent downloads.
`maxTimeout`	number	30000	Max request timeout in ms.
`clearOldFilesOnSuccess`	boolean	true	Remove all existing files in dest dir when downloads succeed. Ignores files that match `keep`
`clearOldFilesOnStart`	boolean	false	Will remove all existing files _before_ downloads starts. `false` will ensure that existing files are only deleted after a download succeeds.
`ignoreCache`	boolean	false	Will always download files regardless of whether they've been cached
`enableIfModifiedSinceCheck`	boolean	true	Enables the HTTP if-modified-since check. Disabling this will assume that the local file is the same as the remote file if it already exists.
`enableContentLengthCheck`	boolean	true	Compares the HTTP header content-length with the local file size. Disabling this will assume that the local file is the same as the remote file if it already exists.
`abortOnError`	boolean	true	If set to `true`, errors will cause syncing to abort all remaining tasks immediately
`ignoreImageTransformCache`	boolean	false	Set to `true` to always re-generate transformed images, even if cached versions of the original and transformed image already exist.
`ignoreImageTransformErrors`	boolean	true	Set to `false` if you want to abort a content source from downloading if any of the image transforms fail. Leaving this to `true` will allow for non-image files to fail quietly.
`forceClearTempFiles`	boolean	true	Set to `false` if you want to keep all contents of the tempPath dir before downloading