0.3.5 • Published 7 years ago

@naktibalda/testmybot v0.3.5

Weekly downloads
1
License
MIT
Repository
github
Last release
7 years ago

Test My Bot

NPM

npm version license

TestMyBot is a test automation framework for your chatbot project. It is unopinionated and completely agnostic about any involved development tools. Best of all, it’s free and open source.

Your test cases are recorded by Capture & Replay tools and can be run against your Chatbot implementation automatically over and over again. It is meant to be included in your continuous integration pipeline, just as with your unit tests.

Blog Articles

Here are links to some articles about TestMyBot:

Serverless Monitoring Of Your Facebook Chatbot in 3 Easy Steps

Capture & Replay: Bringing Chatbot Code Quality To a New Level

Continuous Automated Testing for Your Chatbot With Open Source Tool “TestMyBot”

No More Excuse: Automated Testing of your Chatbot with “TestMyBot”

TestMyBot "Container Modes"

TestMyBot supports three modes to run automated tests against your Chatbot.

docker

In this container mode, your Chatbot is transfered into a docker container, see below. This is the most versatile container mode.

local

In this container mode, you have to wire your Chatbot with the TestMyBot library. Examples and helper classes for doing it with Botkit exist - see this sample.

This mode is available for Chatbots developed with Node.js only

fbdirect

You can use TestMyBot to run your conversations against an already deployed Facebook Chatobt - see this sample

This mode is available for Facebook Chatbots only

What mode to use ?

This depends on your Chatbot project and what you want to achieve.

How does container mode "Docker" work ?

The key (and only) concept of this framework is to simulate ("mock out") the "real" Chatbot APIs (like the Facebook Messenger Platform API), hijacking them through docker networks. Your chatbot is transfered into a local docker container, the API mocks are possible by manipulating the DNS of the docker image. For example, any access to "graph.facebook.com" is redirected to another local docker container simulating the Facebook Messenger Platform API.

TestMyBot injects your Chatbots behaviour into the test runner, and your test case specifications into your Chatbot.

TestMyBot heavily relies on Docker to provide it’s outstanding capabilities in test automation.

Showcase: Your Chatbot as Node.js application with Facebook Messenger Platform API

In "normal" operation mode, your Chatbot connects to the Facebook Messenger Platform API at http://graph.facebook.com and receives Callbacks from there on a registered webhook - similar for Slack oder other platforms. So the architecture in the most simple scenario looks like this:

Architecture without Docker

Showcase: TestMyBot is running automated tests on your Chatbot

When TestMyBot runs your Chatbot, it won't connect to the real Chatbot API, but it won't notice. It runs within a docker container providing a network seperated from the physical network. TestMyBot simulates Facebook Messenger Platform API and runs your test cases on it.

Architecture with Docker

Requirements

Please install Docker and Docker Compose (>= Version 1.10.0) first on your development machines.

Of course, you need Node.js (>= Version 4.0) and npm installed and working.

Current Requirements on your Chatbot:

  • Developed in Node.js or any other programming language where a Docker image is available
  • Developed with Facebook Messenger Platform or Slack API
  • Webhook has to listen on all interfaces (listen to 0.0.0.0)
  • Accept self-signed SSL certificates (for Node.js, NODE_TLS_REJECT_UNAUTHORIZED environment variable is set to 0 automatically by TestMyBot)
  • Request verification has to be disabled (for Botkit: validate_requests should be set to false)

Special considerations for Facebook Messenger Platform

See README-Facebook.md

Special considerations for Slack API

See README-Slack.md

Quick Start

Please check out one of the samples to get a quick overview.

$ git clone https://github.com/codeforequity-at/testmybot.git
$ cd testmybot/samples/facebook
$ npm install
$ npm test

Installation and Basic Usage

Usually, you won't install this project on it's own, but you will include it in your Chatbot projects.

To install it to your chatbot project, type:

$ npm install testmybot --save-dev

Please note that you have to install it in your local development directory (not in global registry with -g).

With Jasmine, the setup looks like this:

$ npm install testmybot --save-dev
$ npm install jasmine --save-dev
$ ./node_modules/.bin/jasmine init

Add a file named "testmybot.json" to your project directory. A very basic configuration for a Facebook Chatbot looks like this:

{
  "docker": {
    "container": {
      "testmybot-fbmock": {
        "run": true,
      }
    }
  }
}

Add a file named "docker-compose.testmybot.override.yml" to your project directory. A very basic configuration for a Facebook Chatbot looks like this:

version: "2"
services:
    testmybot-fbmock:
	environment:
	    TESTMYBOT_FACEBOOK_WEBHOOKPORT: 5000
	    TESTMYBOT_FACEBOOK_WEBHOOKPATH: "webhook"

You tell TestMyBot that the Facebook Webhook of your chatbot runs on port 5000, and the url path is /webhook. You don't have to tell the hostname, because it will run in a docker container with a fixed hostname ("testmybot").

Add a file spec/testmybot.spec.js with a basic test case:

describe('TestMyBot Sample Conversation Test Suite', function() {
  var bot = require('testmybot');

  beforeAll(function(done) {
    bot.beforeAll().then(done);
  }, 120000);

  beforeEach(function(done) {
    bot.beforeEach().then(done);
  }, 60000);

  afterEach(function(done) {
    bot.afterEach().then(done);
  }, 60000);

  afterAll(function(done) {
    bot.afterAll().then(done);
  }, 60000);

  it('should answer to hello', function(done) {

    bot.hears('hello');

    bot.says().then((msg) => {
      expect(msg.messageText).toMatch(/echo/);
      done();
    }).catch((err) => {
      throw new Error(err);
    });
  });

  it('should send a generic payload', function(done) {

    bot.hears('Generic');

    bot.says().then((msg) => {
      expect(msg.message.attachment.type).toEqual('template');
      expect(msg.message.attachment.payload.template_type).toEqual('generic');
      done();
    }).catch((err) => {
      throw new Error(err);
    });
  });
});

Take special care for:

  • All test are asynchronous
  • Setup and Teardown has high timeouts, because buildling, running and stopping Docker containers can take some time. Especially on first run, it will take very long. Afterwards, the Docker cache speeds up things.
  • TestMyBot uses Bluebird Promises
  • The test API is rather simple
    • bot.hears: send a text (or structured content) to your chatbot
    • bot.says: receive a text (or structured content) from your chatbot

In your package.json, define a script for TestMyBot names start_testmybot, which is run in the Docker container.

...
"scripts": {
  "start_testmybot": "node index.js",
},
...

You can hand over environment variables to your chatbot here. And finally, run your tests with Jasmine:

$ ./node_modules/.bin/jasmine

You will see some output from Docker, and in the end, your Jasmine tests should succeed (of course).

How to Compose Test Cases

As is it very hard to compose your test cases from a combination of bot.hears / bot.says instructions to the test runner, TestMyBot is able to run your test cases from previously recorded conversations. The conversations are loaded from simple text files. The conversation files should be included in your project and added to your source code control, just as with other test specs (usually to folder ./spec/convo/*.convo.txt).

Convo Files

We made a strong decision to not use any standard file format like JSON or XML for writing the test cases, as they should be kept extremly simple. It should be so simple that everyone could compose the conversation files manually. Here is an example for a simple test conversation:

Call Me Captain
A simple Test Case 

#me
hello

#bot
Try: `what is my name` or `structured` or `call me captain`

#me
call me captain

#bot
Got it. I will call you captain from now on.

#me
who am i

#bot
Your name is captain

The semantics are simple:

  • The first line is the name of the test case
  • The second line up to the first line starting with # is an optional description text
  • A line starting with #me will send the following text to your Chatbot
    • Anything following will be the channel to send to - for example: #me #private will send the message to the private channel (Slack only)

  • A line starting with #bot will expect your Chatbot to answer accordingly

    • Anything following will be the channel to listen to - for example: #bot #general will wait for a message on the #general-channel (Slack only)

That's it.

Structured Messages

Actually, for sending structured messages, you can include the message content in this conversation files as well:

#bot
{
    "message": {
        "attachment": {
            "type": "template",
            "payload": {
                ....
            }
        }
    }
}

#me
{
    "postback": {
        "payload": "White T-Shirt"
    }
}

Capture & Replay Tools

Especially with structured messages, it can become uncomfortable to write those conversation files manually. TestMyBot contains two tools to support you with writing your conversation files:

TestMyBot IDE

The TestMyBot IDE provides a simple browser interface to record and organize your test cases, and to interact with your Chatbot.

TestMyBot IDE

Installation and running it is simple:

$ npm install testmybot-ide --save-dev
$ node ./node_modules/testmybot-ide/ide.js

It will show all test cases from the ./spec/convo/-Folder and will write new conversations to this directory as well.

More Information

TestMyBot Chat

The TestMyBot Chat is a basic command line interface to your Chatbot running within TestMyBot. You can record and save your conversation files.

TestMyBot Chat

Installation and running it is simple:

$ npm install testmybot-chat --save-dev
$ node ./node_modules/testmybot-chat/index.js

More Information

Running the conversation files

You have to advice the TestMyBot library within your test specification how to add test cases to your test runner. In the following example, you advice it to add the test cases to Jasmine:

describe('TestMyBot Sample Conversation Test Suite', function() {
    var bot = require('testmybot');

    ... setup / teardown code
    
    bot.setupTestSuite(
        (testcaseName, testcaseFunction) => {
          it(testcaseName, testcaseFunction, 60000);
        },
        (response, tomatch) => {
          expect(response).toContain(tomatch);
        },
        (err) => fail(err)
    )        
});

You have to provide the callback functions for:

  • adding a test case to your test suite. TestMyBot provides the name of the test case and the function to call.
  • comparing your bot responses with the actual responses. You can choose your matcher here (regexp, contains, ...)
  • failing the test case because the bot doesn't respond (or other failures)

So TestMyBot is agnostic about the actual test runner you are using. The example above can be adapted to other test runners easily.

Test Suite Setup for Jasmine and Mocha

There exist helper functions for setting up the whole test suite for Jasmine and Mocha. Place this code in your spec/testmybot.spec.js:

const bot = require('testmybot');
const botHelper = require('testmybot/helper/jasmine');

botHelper.setupJasmineTestSuite(60000);

This will create a test case for all your conversation files automatically.

Configuration

There are three steps for buildling the configuration:

  • ./node_modules/testmybot/testmybot.default.json
  • ./testmybot.default
  • "config" Parameter to "beforeAll" method

The subsequent steps are overwriting the configuration parameters from the previous steps. The .json-files are converted into a plain object in Node.js.

Please see ./node_modules/testmybot/testmybot.default.json for information what settings you have to override in your project configuration.

The docker containers are configured with seperate docker-compose.yml-files:

  • ./node_modules/testmybot/docker-compose.testmybot.yml
  • ./node_modules/testmybot-fbmock/docker-compose.testmybot-fbmock.yml
  • ./node_modules/testmybot-slackmock/docker-compose.testmybot-slackmock.yml
  • ./docker-compose.testmybot.override.yml

The files are handed over to docker-compose in this order, so you can include your project-specific settings in the last file.

API

  • testmybot.beforeAll(config) - builds Docker networking and containers from your configuration
    • config (optional) - you can pass an optional configuration argument (merged into other configuration)
    • returns a bluebird Promise
  • testmybot.afterAll() - removes Docker networking and containers
    • returns a bluebird Promise
  • testmybot.beforeEach() - starts Docker containers and waits until online
    • returns a bluebird Promise
  • testmybot.afterEach() - stops Docker containers
    • returns a bluebird Promise
  • testmybot.hears(msg, from, channel) - send a text (or structured content) to your chatbot
    • msg - text or structured content
    • from - sender of the message
    • channel (optional) - the channel to send your message to (Slack only). You can mock parallel conversations with multiple users

  • testmybot.says(channel, timeoutMillis) - receive a text (or structured content) from your chatbot

    • channel (optional) - receive for this channel (Slack only)
    • timeoutMillis (optional) - timeout when not receiving anything (default: 5000)
    • returns a bluebird Promise, which resolves to the the received message.

Here is an example for a received message. It contains (for brevity) the message text (if applicable), the full original message (in "orig") and the message body (in "message). For structured messages, there is no messageText.

  {
      "orig": {
          "recipient": {
            "id":4440090943
          },
        "message": { 
            "text": "Text received, echo: hello"
        }
      },
      "messageText": "Text received, echo: hello",
      "message": {
          "text": "Text received, echo: hello"
      },
      "channel": "#private"
  }

Outlook

Work is ongoing

  • Support Facebook Chatbots
  • Support Slack Chatbots
  • Support Wechat Chatbots
  • Support Node.js Chatbots
  • Support Microsoft Bot Framework Chatbots
  • Support Python Chtabots
  • Define Test Cases with TestMyBot API calls
  • Define Test Cases by conversation transcripts
  • Run your Tests in live environment with real Endpoints

License

MIT License

Copyright (c) 2017 Code For Equity

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

continuoustestingchatbot
asyncbluebirdfacebook-chat-apiis-jsonlodashmkdirpn-readlinesrequestsocket.io-clienttcp-port-usedtestmybot-fbmocktestmybot-slackmock
@infinitebrahmanuniverse/nolb-_nak@everything-registry/sub-chunk-632
0.3.5

7 years ago

0.3.4

7 years ago

0.3.3

7 years ago

0.3.2

7 years ago

0.3.1

7 years ago