proficiency v0.0.8
Proficiency
Regression test tool for rasa_nlu.
What is it for?
Proficiency allows bot developers to stay confident of the bot behaviour while its language model grows and becomes more complex. Developers can
- define what intent and entites are expected given a certain query
- define confidence levels required for certain, acceptable and uncertain matches
- get a visual report of the test run
see details of the issues
Warning
In order to understand the following instructions you should be familiar with Rasa NLU and its terms. Please take a look at the docs if you feel you're missing something.
Setup
Install the library
npm install -g proficiency
Create a project folder where you will store
- config.json: the configuration file, and
suite.json: the test suite.
For your convenience the folder src/sample-project
contains example files that you can copy and customize.
Sample configuration is
{
"project": "new_bot",
"endpoints" : {
"dev": {
"name": "Dev",
"url": "http://localhost:5000/parse"
},
"prod": {
"name": "Production",
"url": "https://example.com:5000/parse",
"token": "rasa_nlu_server_token_if_any"
}
},
"highConfidence": 0.75,
"mediumConfidence": 0.65
}
As you can see, you can specify connection tokens for your endpoints. If null or missing, no token will be sent. Note: Be sure to use the same project name you used in Rasa. Otherwise the NLU serivce will be unable to answer.
Write regression tests in suite.json.
{
"examples": [
{
"sentence": "Hi!",
"expected": {
"intent": {
"name": "greetings"
},
"entities": []
}
},
{
"sentence": "Hi Sarah!",
"expected": {
"intent": {
"name": "greetings"
},
"entities": [
{
"start": 3,
"end": 8,
"value": "Sarah",
"entity": "name"
}
]
}
}
]
}
The format of each example is compatible with rasa_nlu responses. This way you can add new examples by
- filling the sentence attribute, then
- copying and pasting intent and entities from the response you received from rasa_nlu service, then
possibly removing fields like confidence, extractor, processors. They are not useful in this file.
Look at the available options
proficiency -h
Launch the test suite
Using your development endpoint:
npx proficiency <regression test folder path>
In production:
npx proficiency <regression test folder path> -p
Happy testing!