3.37.8 • Published 8 months ago

@sanity/import v3.37.8

Weekly downloads
25,067
License
MIT
Repository
github
Last release
8 months ago

@sanity/import

Imports documents from an ndjson-stream to a Sanity dataset

Installing

npm install --save @sanity/import

Usage

const fs = require('fs')
const sanityClient = require('@sanity/client')
const sanityImport = require('@sanity/import')

const client = sanityClient({
  projectId: '<your project id>',
  dataset: '<your target dataset>',
  token: '<token-with-write-perms>',
  useCdn: false,
})

// Input can either be a readable stream (for a `.tar.gz` or `.ndjson` file), a folder location (string), or an array of documents
const input = fs.createReadStream('my-documents.ndjson')

const options = {
  /**
   * A Sanity client instance, preconfigured with the project ID and dataset
   * you want to import data to, and with a token that has write access.
   */
  client: client,

  /**
   * Which mutation type to use for creating documents:
   * `create` (default)  - throws error if document IDs already exists
   * `createOrReplace`   - replaces documents with same IDs
   * `createIfNotExists` - skips document with IDs that already exists
   *
   * Optional.
   */
  operation: 'create',

  /**
   * Function called when making progress. Gets called with an object of
   * the following shape:
   * `step` (string) - the current step name of the import process
   * `current` (number) - the current progress of the step, only present on some steps
   * `total` (number) - total items before complete, only present on some steps
   */
  onProgress: (progress) => {
    /* report progress */
  },

  /**
   * Whether or not to allow assets in different datasets. This is usually
   * an error in the export, where asset documents are part of the export.
   *
   * Optional, defaults to `false`.
   */
  allowAssetsInDifferentDataset: false,

  /**
   * Whether or not to allow failing assets due to download/upload errors.
   *
   * Optional, defaults to `false`.
   */
  allowFailingAssets: false,

  /**
   * Whether or not to replace any existing assets with the same hash.
   * Setting this to `true` will regenerate image metadata on the server,
   * but slows down the import.
   *
   * Optional, defaults to `false`.
   */
  replaceAssets: false,

  /**
   * Whether or not to skip cross-dataset references. This may be required
   * when importing a dataset with cross-dataset references to a different
   * project, unless a dataset with the referenced name exists.
   *
   * Optional, defaults to `false`.
   */
  skipCrossDatasetReferences: false,

  /**
   * Whether or not to import system documents (like permissions and custom retention). This
   * is usually not necessary, and may cause conflicts if the target dataset
   * already contains these documents. On a new dataset, it is recommended that roles are re-created
   * manually, and that any custom retention policies are re-created manually.
   *
   * Optional, defaults to `false`.
   */
  allowSystemDocuments: false,
}

sanityImport(input, options)
  .then(({numDocs, warnings}) => {
    console.log('Imported %d documents', numDocs)
    // Note: There might be warnings! Check `warnings`
  })
  .catch((err) => {
    console.error('Import failed: %s', err.message)
  })

CLI-tool

This functionality is built in to the sanity package as sanity dataset import, but is also usable through the sanity-import CLI tool, part of this package:

$ sanity-import --help

  CLI tool that imports documents from an ndjson file or URL

  Usage
    $ sanity-import -p <projectId> -d <dataset> -t <token> sourceFile.ndjson

  Options
    -p, --project <projectId> Project ID to import to
    -d, --dataset <dataset> Dataset to import to
    -t, --token <token> Token to authenticate with
    --asset-concurrency <concurrency> Number of parallel asset imports
    --replace Replace documents with the same IDs
    --missing Skip documents that already exist
    --allow-failing-assets Skip assets that cannot be fetched/uploaded
    --replace-assets Skip reuse of existing assets
    --skip-cross-dataset-references Skips references to other datasets
    --help Show this help

  Examples
    # Import "./my-dataset.ndjson" into dataset "staging"
    $ sanity-import -p myPrOj -d staging -t someSecretToken my-dataset.ndjson

    # Import into dataset "test" from stdin, read token from env var
    $ cat my-dataset.ndjson | sanity-import -p myPrOj -d test -

  Environment variables (fallbacks for missing flags)
    --token = SANITY_IMPORT_TOKEN

Future improvements

  • When documents are imported, record which IDs are actually touched
    • Only upload assets for documents that are still within that window
    • Only strengthen references for documents that are within that window
    • Only count number of imported documents from within that window
  • Asset uploads and strengthening can be done in parallel, but we need a way to cancel the operations if one of the operations fail
  • Introduce retrying of asset uploads based on hash + indexing delay
  • Validate that dataset exists upon start
  • Reference verification
    • Create a set of all document IDs in import file
    • Create a set of all document IDs in references
    • Create a set of referenced ID that do not exist locally
    • Batch-wise, check if documents with missing IDs exist remotely
    • When all missing IDs have been cross-checked with the remote API (or a max of say 100 items have been found missing), reject with useful error message.

License

MIT-licensed. See LICENSE.

3.37.7

9 months ago

3.37.8

8 months ago

3.37.6

9 months ago

3.37.5

1 year ago

3.37.4

1 year ago

3.37.5-alpha.1

1 year ago

3.37.0

1 year ago

3.37.1

1 year ago

3.37.2

1 year ago

3.37.3

1 year ago

3.36.5-next.85

1 year ago

3.36.5-next.86

1 year ago

3.37.2-canary.9

1 year ago

3.36.5-next.82

1 year ago

3.36.3

1 year ago

3.36.4

1 year ago

3.36.2

1 year ago

3.36.1

1 year ago

3.36.0

1 year ago

3.35.2

1 year ago

3.35.1

1 year ago

3.35.0

1 year ago

3.34.0

1 year ago

3.33.0

1 year ago

3.32.0

1 year ago

3.31.0

1 year ago

3.30.1

1 year ago

3.30.0

1 year ago

3.29.1

1 year ago

3.29.0

1 year ago

3.28.0

1 year ago

3.27.1

1 year ago

3.27.0

1 year ago

3.26.1

1 year ago

3.26.0

1 year ago

3.25.0

1 year ago

3.24.2-canary.1

1 year ago

3.24.0

1 year ago

3.24.1

1 year ago

3.23.4-canary.1

1 year ago

3.23.4-canary.0

1 year ago

3.23.3

1 year ago

3.23.4

1 year ago

3.22.3

1 year ago

3.23.2-canary.8

1 year ago

3.23.2

1 year ago

3.23.2-EDX937.1

1 year ago

3.22.0

1 year ago

3.22.2

1 year ago

3.22.1

1 year ago

3.23.1-canary.1

1 year ago

3.23.1

1 year ago

3.23.0

1 year ago

3.21.4-canary.6

2 years ago

3.21.3

2 years ago

3.21.2

2 years ago

3.21.2-i18n.15

2 years ago

3.21.2-canary.17

2 years ago

3.21.1

2 years ago

3.21.0

2 years ago

3.20.3-i18n.262

2 years ago

3.14.5-i18n.46

2 years ago

3.14.5-i18n.47

2 years ago

3.14.5-i18n.45

2 years ago

3.16.8-sdx-736.1

2 years ago

3.18.2-i18n.179

2 years ago

3.18.1

2 years ago

3.18.0

2 years ago

3.16.7

2 years ago

3.16.1

2 years ago

3.16.0

2 years ago

3.16.3

2 years ago

3.16.2

2 years ago

3.16.4

2 years ago

3.20.3-i18n.242

2 years ago

3.20.3-i18n.243

2 years ago

3.20.3-i18n.247

2 years ago

3.14.2-i18n.10

2 years ago

3.15.1-canary.6

2 years ago

3.20.0

2 years ago

3.20.2

2 years ago

3.20.1

2 years ago

3.14.6-pte.52

2 years ago

3.20.1-i18n.244

2 years ago

3.19.3

2 years ago

3.19.0

2 years ago

3.19.2

2 years ago

3.19.1

2 years ago

3.18.2-next.66

2 years ago

3.17.0

2 years ago

3.15.0

2 years ago

3.15.1

2 years ago

3.19.3-sdx-722.2

2 years ago

3.16.2-canary.19

2 years ago

3.14.1

2 years ago

3.14.0

2 years ago

3.14.3

2 years ago

3.14.2

2 years ago

3.14.5

2 years ago

3.14.4

2 years ago

3.15.2-canary.44

2 years ago

3.20.2-i18n.241

2 years ago

3.20.2-i18n.239

2 years ago

3.15.2-canary.1

2 years ago

3.19.4-canary.2

2 years ago

3.19.4-canary.6

2 years ago

3.13.0

2 years ago

3.10.1-canary.1

2 years ago

3.9.2-canary.18

2 years ago

3.9.2-canary.26

2 years ago

3.11.3

2 years ago

3.11.5

2 years ago

3.11.0

2 years ago

3.11.2

2 years ago

3.11.1

2 years ago

3.9.2-next.27

2 years ago

3.10.2-alpha.13

2 years ago

3.12.1

2 years ago

3.12.0

2 years ago

3.10.1

2 years ago

3.10.0

2 years ago

3.10.3

2 years ago

3.10.2

2 years ago

3.12.2

2 years ago

3.8.1-canary.5

2 years ago

3.8.1-canary.6

2 years ago

3.5.1

2 years ago

3.9.2-canary.12

2 years ago

3.9.2-canary.16

2 years ago

3.8.0

2 years ago

3.8.2-canary.4

2 years ago

3.8.2-canary.2

2 years ago

3.8.3

2 years ago

3.8.2

2 years ago

3.8.1

2 years ago

3.8.4-next.27

2 years ago

3.5.2-next.19

2 years ago

2.35.7

2 years ago

3.6.0

2 years ago

3.7.2-canary.49

2 years ago

3.9.1

2 years ago

3.9.0

2 years ago

3.7.1

2 years ago

3.7.0

2 years ago

2.36.1

2 years ago

3.9.1-canary.38

2 years ago

3.9.1-canary.39

2 years ago

3.9.1-canary.24

2 years ago

3.5.0

2 years ago

3.3.1

2 years ago

3.3.0

2 years ago

3.2.6

2 years ago

3.4.0

2 years ago

3.5.1-next.11

2 years ago

3.1.3

2 years ago

3.1.2

2 years ago

3.1.1

2 years ago

3.1.0

2 years ago

3.1.4

2 years ago

3.0.0-v3-pte.88

2 years ago

3.0.7-canary.14

3 years ago

3.1.5-next.38

2 years ago

3.1.5-next.39

2 years ago

3.1.5-next.43

2 years ago

3.1.5-next.44

2 years ago

3.1.5-next.42

2 years ago

3.1.5-next.45

2 years ago

3.1.5-next.46

2 years ago

3.1.5-next.40

2 years ago

3.0.7-canary.7

3 years ago

3.2.2-next.0

2 years ago

3.2.2

2 years ago

3.2.1

2 years ago

3.2.0

2 years ago

3.2.5

2 years ago

3.2.4

2 years ago

3.2.3

2 years ago

3.0.4

3 years ago

3.0.3

3 years ago

3.0.2

3 years ago

3.0.1

3 years ago

3.0.6

3 years ago

3.0.5

3 years ago

3.1.5-canary.15

2 years ago

3.1.5-canary.14

2 years ago

3.2.4-next.5

2 years ago

3.0.1-canary.23

3 years ago

3.1.5-canary.40

2 years ago

3.0.0-v3-pte.87

3 years ago

3.0.0-v3-pte.30

3 years ago

3.0.0-v3-pte.31

3 years ago

3.0.0-v3-pte.33

3 years ago

2.34.3-cdr-ga.20

3 years ago

2.34.1-canary.0

3 years ago

3.0.0

3 years ago

3.0.0-rc.2

3 years ago

3.0.0-rc.1

3 years ago

3.0.0-rc.0

3 years ago

3.0.0-rc.3

3 years ago

3.0.0-v3-pte.58

3 years ago

3.0.0-v3-pte.55

3 years ago

3.0.0-v3-pte.69

3 years ago

3.0.0-v3-pte.80

3 years ago

3.0.0-v3-pte.79

3 years ago

2.33.0

3 years ago

2.33.2

3 years ago

2.33.4-shopify.8

3 years ago

2.33.3-edge.8

3 years ago

2.31.1

3 years ago

3.0.0-v3-pte.48

3 years ago

3.0.0-v3-pte.49

3 years ago

3.0.0-v3-esm.54

3 years ago

3.0.0-v3-esm.52

3 years ago

3.0.0-v3-esm.53

3 years ago

2.32.1-next.22

3 years ago

3.0.0-v3-pte.56

3 years ago

2.29.5-edge.1030

3 years ago

3.0.0-v3-pte.57

3 years ago

3.0.0-v3-pte.59

3 years ago

2.29.5-edge.1044

3 years ago

3.0.0-v3-pte.61

3 years ago

3.0.0-v3-pte.12

3 years ago

3.0.0-v3-pte.13

3 years ago

3.0.0-v3-pte.16

3 years ago

3.0.0-v3-pte.17

3 years ago

2.30.3

3 years ago

2.29.5-edge.1050

3 years ago

2.30.3-shopify.5

3 years ago

2.30.3-shopify.3

3 years ago

2.30.3-shopify.4

3 years ago

2.30.2-shopify.2

3 years ago

2.30.2-shopify.0

3 years ago

2.29.3

3 years ago

2.28.0

3 years ago

2.25.5-next.6

3 years ago

2.24.1-pte.127

3 years ago

2.24.1-pte.113

3 years ago

2.24.1-pte.115

3 years ago

2.23.4-pte.99

3 years ago

2.23.4-pte.96

3 years ago

2.23.4-pte.95

3 years ago

2.23.5-next.19

3 years ago

2.27.0

3 years ago

2.23.4-pte.107

3 years ago

2.26.0

3 years ago

2.23.4-pte.101

3 years ago

2.23.4-pte.102

3 years ago

2.24.3-pte.128

3 years ago

2.23.2

4 years ago

2.23.1

4 years ago

2.22.3

4 years ago

2.23.3-canary.84

3 years ago

2.23.3-next.34

3 years ago

2.23.3-next.35

3 years ago

2.23.3-next.4

3 years ago

2.22.6-next.121

4 years ago

2.23.2-canary.28

4 years ago

2.22.3-next.5

4 years ago

2.22.6-next.104

4 years ago

2.22.6-next.102

4 years ago

2.21.10

4 years ago

2.23.3-canary.59

3 years ago

2.23.2-next.3

4 years ago

2.23.1-next.10

4 years ago

2.23.0

4 years ago

2.21.8-canary.4

4 years ago

2.21.7

4 years ago

2.21.3

4 years ago

2.21.0

4 years ago

2.21.1-canary.2

4 years ago

2.20.0

4 years ago

2.19.0

4 years ago

2.19.1-canary.1

4 years ago

2.18.1-canary.9

4 years ago

2.18.0

4 years ago

2.17.3-canary.0

4 years ago

2.16.0

4 years ago

2.15.1-canary.4

4 years ago

2.15.0

4 years ago

2.14.1-canary.4

4 years ago

2.14.1-canary.36

4 years ago

2.14.0

4 years ago

2.13.2-canary.13

4 years ago

2.13.2-canary.9

4 years ago

2.13.2-canary.7

4 years ago

2.12.3-rc.0

4 years ago

2.13.0

4 years ago

2.11.3-canary.6

4 years ago

2.11.0

4 years ago

2.10.5

4 years ago

2.10.2

4 years ago

2.10.2-next.0

4 years ago

2.9.1-canary.15

4 years ago

2.8.2-canary.40

4 years ago

2.8.0

4 years ago

2.7.5-next.41

4 years ago

2.7.4

4 years ago

2.7.1

4 years ago

2.7.0

4 years ago

2.6.2

4 years ago

2.5.1-canary.79

4 years ago

2.4.4-canary.81

4 years ago

2.4.4-canary.75

4 years ago

2.3.9-canary.32

4 years ago

2.3.7-canary.22

4 years ago

2.3.5-canary.23

4 years ago

2.3.5-canary.22

4 years ago

2.3.5-canary.17

4 years ago

2.3.5-canary.18

4 years ago

2.2.6

4 years ago

2.1.7-next.22

4 years ago

2.1.7-next.21

4 years ago

2.1.5-canary.3

4 years ago

2.1.4

4 years ago

2.1.2-ui.11

5 years ago

2.1.1-ui.13

5 years ago

2.1.1-ui.18

5 years ago

2.1.1-ui.14

5 years ago

2.1.0

5 years ago

2.0.10-next.46

5 years ago

2.0.10-next.47

5 years ago

2.0.10-canary.11

5 years ago

2.0.10-canary.0

5 years ago

2.0.9

5 years ago

2.0.1

5 years ago

1.150.7

5 years ago

1.150.5-canary.6

5 years ago

1.150.2-canary.0

5 years ago

1.150.0

5 years ago

1.150.1

5 years ago

1.149.18

5 years ago

1.149.16

5 years ago

1.149.13

5 years ago

1.149.9

5 years ago

1.149.7

5 years ago

1.149.1

5 years ago

1.149.2

5 years ago

1.149.0

5 years ago

1.148.7

5 years ago

1.148.5

5 years ago

1.148.3

5 years ago

1.148.0

5 years ago

1.148.1

5 years ago

0.147.10-next.23

5 years ago

0.147.8

5 years ago

0.147.9

5 years ago

0.147.3

6 years ago

0.147.0

6 years ago

0.147.0-alpha.1

6 years ago

0.147.0-alpha.2

6 years ago

0.147.0-alpha.0

6 years ago

0.146.0

6 years ago

0.146.0-alpha.2

6 years ago

0.146.0-alpha.1

6 years ago

0.145.0

6 years ago

0.145.0-alpha.32

6 years ago

0.144.0

6 years ago

0.142.6

6 years ago

0.142.0

6 years ago

0.141.7-next.20

6 years ago

0.141.7-next.19

6 years ago

0.141.7-next.18

6 years ago

0.141.6

6 years ago

0.141.5

6 years ago

0.141.4

6 years ago

0.141.3

6 years ago

0.141.1

6 years ago

0.140.36

6 years ago

0.140.32

6 years ago

0.140.30

6 years ago

0.140.25

6 years ago

0.140.20

6 years ago

0.140.17

6 years ago

0.140.11

6 years ago

0.140.10

6 years ago

0.141.0-alpha.12

6 years ago

0.140.8

6 years ago

0.140.7

6 years ago

0.140.6

6 years ago

0.140.0

6 years ago

0.139.3-parrot.0

6 years ago

0.139.3-canary.0

6 years ago

0.139.1

6 years ago

0.140.0-alpha.4

6 years ago

0.140.0-alpha.3

6 years ago

0.140.0-alpha.1

6 years ago

0.139.0

6 years ago

0.139.0-rc.0

7 years ago

0.138.0

7 years ago

0.138.0-alpha.21

7 years ago

0.138.0-alpha.20

7 years ago

0.138.0-alpha.14

7 years ago

0.137.0

7 years ago

0.136.3

7 years ago

0.136.2

7 years ago

0.136.1-canary.5

7 years ago

0.136.0

7 years ago

0.135.3

7 years ago

0.135.3-next.6

7 years ago

0.135.0

7 years ago

0.135.0-next.2

7 years ago

0.135.0-next.1

7 years ago

0.134.2

7 years ago

0.134.1

7 years ago

0.135.0-beta.0

7 years ago

0.134.0

7 years ago

0.133.2

7 years ago

0.133.1

7 years ago

0.133.0

7 years ago

0.132.10

7 years ago

0.132.8

7 years ago

0.132.6

7 years ago

0.132.5

7 years ago

0.132.4-next.0

7 years ago

0.132.2

7 years ago

0.132.0

7 years ago

0.128.12

7 years ago

0.128.5

7 years ago

0.128.3

7 years ago

0.128.1

7 years ago

0.128.0

7 years ago

0.127.1

7 years ago

0.127.0

7 years ago

0.126.2

7 years ago

0.126.0

7 years ago

0.125.8

7 years ago

0.125.6

7 years ago

0.125.4

7 years ago

0.125.0

7 years ago

0.125.0-rc.4

7 years ago

0.124.12

7 years ago

0.125.0-rc.3

7 years ago

0.125.0-rc.2

7 years ago

0.125.0-rc.1

7 years ago

0.125.0-rc.0

7 years ago

0.124.9

7 years ago

0.124.5

7 years ago

0.124.0

7 years ago

0.123.1

7 years ago

0.123.0

8 years ago

0.122.0

8 years ago

0.121.0

8 years ago

0.120.0

8 years ago

0.119.0

8 years ago

0.118.4

8 years ago

0.118.0

8 years ago

0.117.0

8 years ago

0.116.2

8 years ago

0.116.0

8 years ago

0.116.0-rc.0

8 years ago

0.115.2

8 years ago

0.115.1

8 years ago

0.115.0

8 years ago

0.114.0

8 years ago

0.113.7

8 years ago

0.113.3

8 years ago

0.113.1

8 years ago

0.113.0

8 years ago

0.113.0-rc.0

8 years ago

0.112.0

8 years ago

0.111.6-beta.0

8 years ago

0.0.1

8 years ago