Karma-cukes NPM

Karma-Cukes

Karma plugin for running Gherkin feature files against CucumberJS in end-to-end tests.

Idea

Karma is a developer-friendly tool for running tests in real browsers, with direct access to client-side markup and code (without needing a Selenium tunnel).
CucumberJS enables end-to-end tests and stakeholder-friendly, living documentation.

Bringing these two together could simplify the development workflow from story to spec through a unified set of tools.

Main features of this Karma plugin

Comes bundled with "Progress", "Pretty", and "JSON" reporters that are more in line with the official Cucumber reporters (no HTML reporter yet, this package is more targeted at command-line users).
The API and syntax for registering step definitions and other support code is compatible with CucumberJS (i.e. step definition files can use module.exports).
Native access to the browser, jQuery, etc.
A single toolkit for both Story BDD and Spec BDD.
Tested with Chrome, Firefox, PhantomJS

Existing alternatives

Selenium/Webdriver with built-in CucumberJS framework

While Webdriver (Selenium) offers near out-of-the-box support for Cucumber, the generated reports lack the level of detail that "native" Cucumber provides. They don't allow to auto-generate living documentation from report files (e.g. feature descriptions are missing and both feature and scenario names get normalized to underscored identifiers).

Other Karma-CucumberJS adapters

They basically have the same issues as Webdriver-Cucumber. There are good jUnit-reporters but I could not find proper "Progress", "Pretty", or "JSON" formatters that behave like the ones a Cucumber (or Behat in my case) developer is used to. The ones I tested also did not work with PhantomJS and Firefox.

Solution

The two main options were:

writing a custom adapter + custom Cucumber-compatible reporters for Webdriver, or
writing a custom adapter + custom Cucumber-compatible reporters for Karma.

This package goes for the second option, which –from looking at available adapters—, seemed to be the lower-hanging fruit. I'm not so sure about this anymore after writing the plugin, but here we are ;-)

Limitations compared to Webdriver

Only PhantomJS offers an API for taking screenshots through Karma (AFAIK), while Webdriver provides this feature for all browsers.
The Browser reports are not fully identical, e.g. PhantomJS and Firefox report step locations instead of step definition locations (due a patched Error.captureStackTrace).
There is only a minimal Browser API for visiting URLs in end-to-end tests. The rest is native browser access and up to you (which I personally think is actually an advantage).
No access to response headers (except for AJAX-based requests), but could be doable via Service Workers
For each non-CORS-enabled target server to be used in end-to-end tests, a proxy needs to be defined in the Karma configuration file.

Installation

npm install karma-cukes --save-dev

Peer dependencies: jquery (any version), cucumber (1.*), and of course karma (any version)

Configuration (karma.conf.js)

config.set({

    // activate framework
    frameworks: ['karma-cukes'],

    // specify feature files and support code
    files: [
        { pattern: 'dev/**/*.feature', included: false, watched: true, served: true },
        { pattern: 'dev/**/hooks.js', included: true, watched: true, served: true },
        { pattern: 'dev/**/step-definitions.js', included: true, watched: true, served: true }
    ],

    // forward command line arguments to CucumberJS
    client: {
        args: process.argv.slice(4),
        captureConsole: true
    },

    // activate reporters (kc-progress, kc-pretty, and/or kc-json)
    reporters: ['kc-pretty', 'kc-json'],

    // configure the JSON formatter
    kcJsonReporter: {
        outputDir: 'dev/reports/behaviour',
        outputFile: 'karma-cukes-{shortBrowserName}.json' // supported placeholders: `shortBrowserName`, `browserName`
    },

    // enable colors in the output
    colors: true,

    // proxy test server requests for end-to-end testing
    proxies: {
        "/": "http://localhost:8889/"
    },
    
    // use a root URL for the karma runner that does not interfere with proxied sites
    urlRoot: "/__karma__/",

    ...

})

Running features

// all scenarios
$ ./node_modules/.bin/karma start path/to/karma.conf.js

// tagged scenarios
$ ./node_modules/.bin/karma start path/to/karma.conf.js -- --tags @ui

Assertion utils

Karma-Cukes comes with 3 (basic) assertion methods that are accessible from step definitions.

this.assert.ok(condition, callback, message)
this.assert.equal(actual, expected, callback, message)
this.assert.contain(haystack, needle, callback, message)

Using the Browser object

The World context provides a Promise-based browser interface for loading local (proxied) paths or CORS-enabled URIs:

var world = this;
this.browser
    .url('/my/path')
    .then(function () {
        var window = world.browser.window;      // native browser window (an iframe)
        var document = world.browser.document;  // native browser document object
        var $title = $(document).find('title'); // just use jQuery from here on
        var pageContent = $(document).text();
        ...
    })
    // wait for an element to appear
    .then (function() {
        var $form = $(world.browser.document).find('#my-form');
        $form
            .find('input#search').val('cucumber').end()
            .find('button[type="submit"]').click();
        ;
        return world.browser.waitFor('#search-results');
    });

AJAX requests can be issued as well:

var world = this;
this.browser
    .http('my/api', 'GET', { search: 'cucumber' })
    .then(function(response) {
        var $xhr = world.browser.$ajax;
        if ($ajax.status === 200 && $ajax.getResponseHeader('Content-Type').match(/application\/json/)) {
            return JSON.parse(response);
        }
    })

Built-in Step Definitions

Karma-Cukes provides basic step definitions to get you started:

I go to "$path"
- Calls browser.url
- Example: When I go to "/test"
I should see "$html" in the "$element" element
- Calls assert.contain
- $element can be a tag name or a CSS selector
- Example: Then I should see "Hello" in the "body" element
the "$element" element should be "$expected"
- Calls assert.equal
- $element can be a tag name or a CSS selector
- Example: And the "title" element should be "Hello World"

Formatters

Karma-Cukes has 3 built-in reporters:

"kc-progress"
"kc-pretty"

"kc-json"

  [{
      "id": "test-feature",
      "uri": "path/to/test.feature",
      "name": "Test feature",
      "description": "    In order to test the library\n    As a developer\n    I can test a few dummy features",
      "line": 3,
      "keyword": "Feature",
      "tags": [
          {
              "name": "@active",
              "line": 2
          }
      ],
      "browser": "Chrome 51.0.2704 (Mac OS X 10.11.5)",
      "elements": [
          {
              "id": "test-feature;open-page",
              "name": "Open Page",
              "line": 8,
              "keyword": "Scenario",
              "type": "scenario",
              "tags": [
                  {
                      "name": "@active",
                      "line": 2
                  }
              ],
              "examples": [],
              "steps": [
                  {
                      "keyword": "When ",
                      "name": "I go to \"/\"",
                      "line": 9,
                      "match": {
                          "location": "/path/to/KarmaCukesStepDefinitions.js:9"
                      },
                      "result": {
                          "status": "passed",
                          "error_message": "",
                          "duration": 276000000
                      },
                      "log": []
                  },
                  ...
              ]
          }
          ...
      ]
  }
  ...
  ]