sfdx-data-deploy-plugin v2.5.3

SFDX Data Deploy Plugin

SFDX plugin to deploy/retrieve data to/from Salesforce

This Salesforce CLI plugin can be used to retrieve records from any Salesforce org and save them to local JSON files. These data files can be stored in Git repositories to share them with other team members. The plugin can use the data files to deploy records to several other Salesforce orgs. Practical use cases are:

• Maintaining test data as source files in a Git repository
• Rolling out configuration records as part of deployments

TIP: Enhance your developer experience with the corresponding Visual Studio Code extension.

• SFDX Data Deploy Plugin
• Installation
• Usage
• Known Issues
• Commands
• Version History
• License

Installation

1. Download and install Salesforce CLI.

2. Install the plugin:

   sfdx plugins:install sfdx-data-deploy-plugin

TIP: If you want to install the plugin on a CI server you must add it to the whitelisted plugins as described in this blog post.

Usage

Creating a Deployment Descriptor

The deploy and retrieve commands of the plugin require a directory as the context for the deployment. This directory must contain a special JSON file named datadeploy.json which describes the deployment characteristics. The major content of this deployment descriptor is the declaration of jobs which are executed one after another. Each job represents the handling of records for a specific Salesforce object.

Creating a new deployment context requires the following steps.

1. Create a new directory in your local file system.
2. Create a new JSON file named datadeploy.json in that directory.

3. Insert the following content into the JSON file:

   {
     "jobs": []
   }

The following sections explains how to create jobs for data deployment and retrieval.

Creating a Job

The jobs array in the deployment descriptor must be populated with the configurations of jobs to be executed sequentially in the given order. Each job configuration has general properties which apply to both deployment and retrieval. These properties are described in the following table.

The following example shows a basic job configuration for the Account object.

{
  "sObjectApiName": "Account",
  "dataFileName": "Account.json"
}

Creating a Deployment Job Configuration

For deployment of records from the data file to Salesforce there are some further configuration properties which can be declared as an object in the deployConfig property of the job configuration. The deployment properties are described in the following table.

The following example shows a deployment job configuration for the Account object, assuming that there is an external ID field named AccountId__c. Deployment would continue even if some error occurred in the job (e.g. invalid external ID field or some records failing to deploy).

{
  "sObjectApiName": "Account",
  "dataFileName": "Account.json",
  "deployConfig": {
    "externalIdFieldApiName": "AccountId__c",
    "maxWaitMinutes": 2,
    "failOnError": false
  }
}

Data is deployed using the Salesforce Bulk API. The plugin checks the results of each Bulk API job before continuing with the next one. By default, the deployment fails with an error if at least one record could not be deployed properly. Note that in this case some records from this job and all previous jobs may still have been deployed because the Bulk API is not transactional.

Creating a Retrieval Job Configuration

For retrieval of records from Salesforce to the data file there are some further configuration properties which can be declared as an object in the retrieveConfig property of the job configuration. The retrieval properties are described in the following table.

The following example shows a retrieval job configuration for the Account object, assuming there is the custom field AccountId__c. It retrieves only records where the Name begins with "Demo ", fetches only the Name and AccountId__c fields and sorts the results by Name in ascending order.

{
  "sObjectApiName": "Account",
  "dataFileName": "Account.json",
  "retrieveConfig": {
    "includeFieldApiNames": ["AccountId__c", "Name"],
    "filterCriteria": {
      "Name": { "$like": "Demo %" }
    },
    "sortFieldApiNames": ["Name"]
  }
}

Filtering Retrieved Records

The retrieval configuration property filterCriteria enables the selection of specific records to include in the result. The property value must be a JSON-based condition expression (like MongoDB) as supported by the JSForce SObject#find() method.

The following table shows some examples of filter criteria.

Another more complex example with OR-combined criteria.

{
  "$or": [{ "Name": { "$like": "Test%" } }, { "GrossValue": { "$lte": 500 } }]
}

It selects those records with a name starting with "Test" OR with a gross value less than/equal to 500.

Sorting Retrieved Records

The retrieval configuration property sortFieldApiNames enables sorting of the records in the result by multiple fields. The property value must be an array of the field API names. The first field in the array is the strongest sorting criterion. If a second field is given then records with the same value in the first sorting field are sorted by the value of the second field and so on.

If a field name is given without any prefix (e.g. FirstName) then records are sorted by this field in ascending order.

If a field name is prefixed by a dash (e.g. -LastName) then records are sorted by this field in descending order.

Deploying Data to Salesforce

The plugin requires a connection to a Salesforce org in order to deploy records from data files. Once the deployment descriptor has been prepared and the data files contain the proper records follow these steps to perform the deployment.

1. Connect Salesforce CLI to the target Salesforce org (see the auth commands in the documentation).

2. Run the deploy command (here assuming that the deployment descriptor is located in the subdirectory testdata):

   sfdx datadeploy:deploy -d testdata -u yourname@yourorg.com

TIP: If the deployment descriptor is in the current working directory you can omit the -d flag.

The plugin runs the deployment for all jobs defined the deployment descriptor by default. Use the command line options -i or -x followed by a comma-separated list of data file paths to specify which jobs to include or exclude from deployment. The file paths are resolved inside the deployment directory.

sfdx datadeploy:deploy -d testdata -i IncludedFile1.json,IncludedFile2.json -u yourname@yourorg.com

Retrieving Data from Salesforce

The plugin requires a connection to a Salesforce org in order to retrieve records to data files. Once the deployment descriptor has been prepared with the proper retrieval configuration follow these steps to retrieve records.

1. Connect Salesforce CLI to the target Salesforce org (see the auth commands in the documentation).

2. Run the retrieve command (here assuming that the deployment descriptor is located in the subdirectory testdata):

   sfdx datadeploy:retrieve -d testdata -u yourname@yourorg.com

TIP: If the deployment descriptor is in the current working directory you can omit the -d flag.

The plugin runs the retrieval for all jobs defined the deployment descriptor by default. Use the command line options -i or -x followed by a comma-separated list of data file paths to specify which jobs to include or exclude from retrieval. The file paths are resolved inside the deployment directory.

sfdx datadeploy:retrieve -d testdata -i IncludedFile1.json,IncludedFile2.json -u yourname@yourorg.com

Tips and Examples

Same Set of Fields for Each Record in JSON File

Be careful when crafting JSON files with records by hand. Each record must have the same set of fields! If only one record does not contain a field that other records have then this field will be ignored for all records in the file. The order of the fields does not matter.

In the following example the Type of the second Account record will not be deployed because the field is missing for the first record.

[
  {
    "AccountId__c": "b7845971-2677-43e0-9316-4909060da942",
    "Name": "Demo Company 1"
  },
  {
    "AccountId__c": "01898b4a-555b-4010-ab1c-e6e9aeb3f20e",
    "Name": "Demo Company 2",
    "Type": "Prospect"
  }
]

Deploying and Retrieving Lookup Relationships

In the data file use the relationship name followed by the external ID field API name of the related object and provide the other record's external ID value. In the following example for the Contact object the reference to the Account is created by referring to the custom external ID field AccountId__c defined on the Account object.

{
  "Account.AccountId__c": "b7845971-2677-43e0-9316-4909060da942"
}

Note that custom lookup relationships must be suffixed by __r as in the following example.

{
  "OtherObject__r.OtherExternalId__c": "b7845971-2677-43e0-9316-4909060da942"
}

When retrieving a lookup relationship using the external ID field of the related object use the following syntax in the retrieval configuration of the job. The example fetches the Account relationship for Contact records using the custom external ID field AccountId__c defined on the Account object.

{
  "sObjectApiName": "Contact",
  "dataFileName": "Contact.json",
  "retrieveConfig": {
    "includeFieldApiNames": ["ContactId__c", "Account.AccountId__c"]
  }
}

If you want to retrieve all fields but still include further lookup relationships specify an asterisk as in the following example for the Contact object.

{
  "sObjectApiName": "Contact",
  "dataFileName": "Contact.json",
  "retrieveConfig": {
    "includeFieldApiNames": ["*", "Account.AccountId__c"],
    "excludeFieldApiNames": ["AccountId"]
  }
}

Example Configuration

See the subdirectory data in this Git repository for an example.

Known Issues

Setting NULL for custom lookup relationship does not work

Assuming that an object has a custom lookup relationship to another custom object (not a standard object) then using the data deployment to remove an existing relation to another record will not work. Pay attention to null values in relationship fields as in the following example.

{
  "OtherObject__c": null
}

Records with existing relations in the target org will still have their relation after deployment. This is an issue in the JSforce library, see https://github.com/jsforce/jsforce/issues/943

Commands

• sfdx datadeploy:deploy [-d <directory>] [-i <array>] [-x <array>] [-u <string>] [--apiversion <string>] [--json] [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL]
• sfdx datadeploy:retrieve [-d <directory>] [-i <array>] [-x <array>] [-u <string>] [--apiversion <string>] [--json] [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL]

sfdx datadeploy:deploy [-d <directory>] [-i <array>] [-x <array>] [-u <string>] [--apiversion <string>] [--json] [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL]

Deploy records from data files to Salesforce

USAGE
  $ sfdx datadeploy:deploy [-d <directory>] [-i <array>] [-x <array>] [-u <string>] [--apiversion <string>] [--json]
  [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL]

OPTIONS
  -d, --deploydir=deploydir                                                         Directory containing the deployment
                                                                                    descriptor 'datadeploy.json'
                                                                                    (default: current working directory)

  -i, --include=include                                                             Include only the given list of data
                                                                                    files (default: all)

  -u, --targetusername=targetusername                                               username or alias for the target
                                                                                    org; overrides default target org

  -x, --exclude=exclude                                                             Exclude the given list of data files
                                                                                    (default: none)

  --apiversion=apiversion                                                           override the api version used for
                                                                                    api requests made by this command

  --json                                                                            format output as json

  --loglevel=(trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL)  [default: warn] logging level for
                                                                                    this command invocation

EXAMPLES
  $ sfdx datadeploy:deploy --deploydir ./testdata --targetusername myOrg@example.com
  $ sfdx datadeploy:deploy --deploydir ./testdata --include Account.json,Contact.json --targetusername myOrg@example.com

sfdx datadeploy:retrieve [-d <directory>] [-i <array>] [-x <array>] [-u <string>] [--apiversion <string>] [--json] [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL]

Retrieve records from Salesforce to data files

USAGE
  $ sfdx datadeploy:retrieve [-d <directory>] [-i <array>] [-x <array>] [-u <string>] [--apiversion <string>] [--json]
  [--loglevel trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL]

OPTIONS
  -d, --deploydir=deploydir                                                         Directory containing the deployment
                                                                                    descriptor 'datadeploy.json'
                                                                                    (default: current working directory)

  -i, --include=include                                                             Include only the given list of data
                                                                                    files (default: all)

  -u, --targetusername=targetusername                                               username or alias for the target
                                                                                    org; overrides default target org

  -x, --exclude=exclude                                                             Exclude the given list of data files
                                                                                    (default: none)

  --apiversion=apiversion                                                           override the api version used for
                                                                                    api requests made by this command

  --json                                                                            format output as json

  --loglevel=(trace|debug|info|warn|error|fatal|TRACE|DEBUG|INFO|WARN|ERROR|FATAL)  [default: warn] logging level for
                                                                                    this command invocation

EXAMPLES
  $ sfdx datadeploy:retrieve --deploydir ./testdata --targetusername myOrg@example.com
  $ sfdx datadeploy:retrieve --deploydir ./testdata --include Account.json,Contact.json --targetusername
  myOrg@example.com

Version History

• Release 2.5.3
  • UPDATE: Upgraded dependencies due to security vulnerabilities
• Release 2.5.2
  • UPDATE: Dependencies updated to most recent versions
• Release 2.5.1
  • UPDATE: Dependencies updated to most recent versions
• Release 2.5.0
  • NEW: Job retrieval configuration option excludeSystemFields
• Release 2.4.5
  • UPDATE: Dependencies updated to most recent versions
• Release 2.4.4
  • UPDATE: Dependencies updated to most recent versions
• Release 2.4.3
  • UPDATE: Dependencies updated to most recent versions
• Release 2.4.2
  • UPDATE: Dependencies updated to most recent versions
• Release 2.4.1
  • UPDATE: Mention Visual Studio Code extension in README file
• Release 2.4.0
  • NEW: Command line options --include and --exclude support real paths relative to deployment directory
  • UPDATE: Log messages no longer use emoji but text prefixes
  • UPDATE: Dependencies updated to most recent versions
• Release 2.3.7
  • UPDATE: Dependencies updated to most recent versions
• Release 2.3.6
  • FIX: #11 Bulk Data Load jobs are not closed after deployment
• Release 2.3.5
  • UPDATE: Dependencies updated to most recent versions
• Release 2.3.4
  • UPDATE: Dependencies updated to most recent versions
• Release 2.3.3
  • FIX: #6 Jenkins deployment fails because of json output
• Release 2.3.2
  • FIX: #4 Import will abort with no message when wrong externalId is set in deployment descriptor
  • FIX: #5 Import will abort with no message when import file is empty
• Release 2.3.1
  • FIX: #3 retrieveConfig.maxRecordCount is limited to 500
• Release 2.3.0
  • NEW: Command line options --include and --exclude to define which jobs to include or exclude from deployment or retrieval
• Release 2.2.0
  • NEW: Job deployment configuration option failOnError
  • UPDATE: Internal refactoring to move logic to separate modules
  • FIX: Error message definition errorDeploymentFileNotReadable missing for retrieve command
• Release 2.1.0
  • NEW: Summary table after deployment or retrieval
  • NEW: Icon prefixes for console log messages
• Release 2.0.3
  • UPDATE: Dependencies updated to most recent versions
• Release 2.0.2
  • NEW: System fields LastActivityDate, LastViewedDate and LastReferencedDate are excluded from retrieval
  • NEW: Data of a record is printed when it fails to deploy
  • UPDATE: Dependencies updated to most recent versions
• Release 2.0.1
  • UPDATE: Dependencies updated to most recent versions
• Release 2.0.0
  • NEW: Direct internal usage of Bulk API instead of using other CLI commands
  • NEW: JSON format for data files
  • NEW: Command to retrieve data from Salesforce
• Release 1.0.2
  • FIX: #1 Incorrect default values for options displayed in CLI help
• Release 1.0.0
  • Initial version

License

MIT