Railflow NPM | npm.io

Railflow CLI

What is Railflow CLI?

Railflow NPM based CLI is meant to be a cross-platform stand-alone utility so that engineers can easily integrate their CICD systems or testing framework with TestRail.

Why use the CLI?

If you are using Jenkins or TeamCity, you can use the respective plugins or the Railflow CLI. If you are using any of the SAAS CICD systems like Gitlab, GitHub, CircleCI, Travis CI, etc., you can use the Railflow CLI via NPM or Docker to easily integrate Railflow into your CICD process.

How much does Railflow CLI cost?

Railflow CLI has a free edition for JUnit reports, and it includes full access to our support portal and Slack channel.

Railflow CLI also has Enterprise and Professional editions for advanced features such as JIRA defect automation, smart assignment, and supporting a rich set of reporting formats.

You can get a free or 14-day Enterprise evaluation license on our website.

What test frameworks does the Railflow CLI support?

Railflow CLI supports a host of testing framework reporting schemas, and we're always adding support for new ones. Currently, Railflow supports JUnit, TestNG, NUnit, Allure, Cucumber, Robot Framework, TRX, Postman, PyTest, ReadyAPI, and Katalon.

Railflow CLI Requirements

NodeJS - v14.17.0 or higher.
Enable TestRail API

Installation

Railflow CLI is a npm based utility. Installing Railflow is as simple as installing any other npm package.

npm install railflow

CLI Command Reference

Note: Use double quotes for argument values with spaces. Example: --project "demo project"
Note: Railflow CLI will automatically create tests, runs, plans, milestones, etc. if they do not exist.

Key	Required	Description	Example
-v, --version	No	Outputs Railflow version number	-v
-k, --key	-k or -l	(online activation) License key. Can be set with RAILFLOW_LICENSE environment variable	-k XXXXX-XXXXX-XXXXX-XXXXX
-l, --license-file	-k or -l	(offline activation) File path or remote url license file	-l /files/ActivationFile.skm
-url, --url	Yes	TestRail instance URL. Can be set with RAILFLOW_TR_URL environment variable	-url https://example.testrail.io
-u, --username	Yes	TestRail username. Can be set with RAILFLOW_TR_USER environment variable	-u test-username
-p, --password	Yes	TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable	-p XtpHXiPLEODyhF
-pr, --project	Yes	TestRail project name	-pr "example project"
-path, --test-path	Yes	TestRail test cases path	-path "Section1/subsection2/ShoppingCart
-f, --format	Yes	Report format: JUnit, JUnit-Steps, TestNg, TestNg-Steps, Cucumber, NUnit, NUnit-SpecFlow, Allure, Robot, TRX, xUnit, PyTest-Railflow, Playwright (case insensitive)	-f junit
-r, --report-files	Yes	The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as*/surefire-reports/.xml can be used.E.g. use *target/surefire-reports/.xml to capture all XML files in target/surefire-reports** directory.	-r target/surefire-reports/.xml target/failsafe-reports/.xml
-sm, --search-mode	Yes	Specifies the test case lookup algorithm. `name:` search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified `-path` `path:` search for test case matching the name within the specified `-path`. If test case found, update the test case. If test case not found, create a new test case within specified `-path`	-sm path
-px, --proxy	No	HTTP or SOCKS proxy configuration E.g. socks://username:password@127.0.0.1:1080	-px socks://username:password@127.0.0.1:1080
-t, --timeout	No	Upload timeout (seconds)	-t 10
-tr, --test-run	No	TestRail test run name	-tr "Chrome Regression Run"
-tp, --test-plan	No	TestRail test plan Name	-tp "Shopping Cart Test Plan"
-mp, --milestone-path	No	TestRail milestone path	-mp Milestone1/Milestone2
-cf, --case-fields	No	TestRail test case custom fields. The format is `[Field label]=[value]` pairs, separated with space. E.g. "Case Field 1=value 1" "Case Field 2=value 2" ...	-cf "Case Field 1=value 1" "Case Field 2=value 2"
-rf, --result-fields	No	TestRail test result custom fields. The format is `[Field label]=[value]` pairs, separated with space. E.g. "Result Field 1=value 1" "Result Field 2=value 2" ...	-rf "Result Field 1=value 1" "Result Field 2=value 2"
-a, --assign	No	Smart Test Failure Assignment. Comma-separated list of TestRail users (email addresses). Railflow will assign failures based on a round robin algorithm.	-a user1@email.com,user2@email.com
-af, --assign-file	No	Smart Test Failure Assignment. File path containing list of TestRail users (email addresses).`Note: One user per line`	-af /assignees.txt
-cn, --config-names	No	TestRail test plan configuration options.Configuration format is: config_group_name/config_name. E.g. "Operating Systems/Linux" "Browsers/Chrome" ...	-cn "Operating Systems/Linux" "Browsers/Chrome"
-cr, --close-run	No	If Railflow should close the corresponding run after uploading test results	-cr
-cp, --close-plan	No	If Railflow should close the corresponding plan after uploading test results	-cp
-dg, --disable-grouping	No	If Railflow should ignore report structure and just upload all tests into a folder which is set by test-path parameter	-dg
-tn, --template-name	No	The name of a template to use in TestRail. If it is not set, 'Test Case (Steps)' or the default one will be used	-tn "Test Case (Steps)"
-cst, --case-type	No	The name of a type for cases	-cst other
-csp, --case-priority	No	The name of a priority for cases	-csp medium
-th, --thread-number	No	The number of concurrent threads for exporting data. Default is 4	-th 8
-um, --upload-mode	No	Upload mode. Available values are: 0 (default) - create new test cases and do not overwrite existing ones; 1 - create new cases and overwrite existing ones; 2 - do not create new cases and overwrite existing ones; 3 - do not create new cases and do not overwrite existing ones	-um 1
-csf, --case-search-field	No	The name of the case field in TestRail which will be using for searching for existing test cases instead of test case title	-csf "Custom field"
-ds, --disable-stats	No	If Railflow should disable collecting usage and error logs	-ds
-fqtn, --fully-qualified-test-name	No	If Railflow should allow exporting test cases using qualified name as the case name	-fcn
-us, --untested-status	No	The name of the status to use in TestRail for untested/skipped tests	-us Skipped
-ams, --attachment-max-size	No	Maximum size of an attachment. Can be set in Gb, Mb, Kb or b. If the unit is not provided considered as Mb. E.g. 1Gb, 2, 20Kb, 3Mb	-ams 1Gb
-atw, --attachment-type-whitelist	No	The comma-separated list of file extensions which are allowed to upload. E.g.`json, .html, .xml`	-atw "json, .html, .xml"
-atb, --attachment-type-blacklist	No	The comma-separated list of file extensions which are not allowed to upload. E.g.`json, .html, .xml`	-atb "json, .html, .xml"
-nr, --no-run	No	Do not create or update Test Run in TestRail	-nr
-tfn, --tags-field-name	No	The name of a test case field which will be holding tags from the report file if any. E.g. `Cucumber Tags`	-tfn "Cucumber Tags"
-h, --help	No	Show the help information	-h

Railflow general usage format:

npx railflow -k <license key> -url <testrail address> -u <username> -p <password> -pr <project name> -path <suite name>/<section name>/<subsection name> -f junit -r <report files pattern> -sm <search mode> -tp [test plan name] -mp [milestone path]

Using Environment Variables

Sometimes it is not secure or not convenient to pass sensitive data directly in the CLI invocation script (e.g. in CICD environments), so Railflow exposes several environment variables which users can use instead of setting corresponding arguments directly. If --key, --url, --username or --password arguments are not provided, Railflow will use the values from the corresponding environment variables: RAILFLOW_LICENSE, RAILFLOW_TR_URL, RAILFLOW_TR_USER or RAILFLOW_TR_PASSWORD.

Example usage

Export JUnit reports into TestRail with a license key (online activation method)

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt

Export JUnit reports into TestRail with a license file (offline activation method)

npx railflow -l /home/user/ActivationFile20201020.skm -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path Master/section1/section2 -f junit -r target/surefire-reports/*.xml -sm path -tr TestRunDemo -tp TestPlanDemo -mp Milestone1/Milestone2 -cn Browsers/Firefox -af assignees.txt