odrl-test-suite v0.1.1

Test Suite

The test suite is software that can execute, validate and measure the compliance of an evaluator over a set of test cases.

A test case consists of the following:

• A Policy
• The state of the world
• The expected compliance report
• Description of the test case
• (optional) A Request (in case of an Access Control Scenario)

How does it work

First, it loads in the test cases. A test case has a description in RDF which instantiated looks like the following:

<urn:uuid:68023514-bfd3-435f-bf2b-e1ef93e3015e> a ex:TestCase;
    dct:title "Title of the test case.";
    ex:policy <PolicyIdentifier> ;
    ex:request <RequestIdentifier> ;
    ex:sotw <StateOfTheWorldIdentifier> ;
    ex:expectedReport <ReportIdentifier> .

In data/test_cases/ examples can be found of an instantiated test case.

The function loadTestCases is responsible for this task.

Next, each test case is evaluated using an Evaluator. That is the evaluator is given a policy, a request and the state of the world. The result (a compliance report) is then compared against the expected compliance report.

During this process, count is tracked of (i) the amount of test cases, (ii) the total amount of successful validations and (iii) the total amount of expected vs evaluated compliance report equality.

Note: currently, the state of the world and the expected compliance report are added in the serialization of a test case.

How to run the test-suite

The test-suite.ts contains the code that runs the test suite.

The directories for the loadTestCases are hardcoded constants which expect the following structure:

data/
└── ODRL
    ├── policies
    ├── requests
    ├── sotw
    └── test_cases

First, the packages need to be installed

npm install

Then, run the test suite:

npx ts-node demo/test-suite.ts

Note: only Nodejs v18.18.2 and npm 9.8.1 is tested

Following code shows how to run the test suite.

import { ComplianceReportComparator, runTestSuite, TestCase, TestCaseEvaluation, TestCaseEvaluator} from "./src";
import { ODRLEvaluator } from "./ODRL-Evaluator";

async function main() {

    const testCases: TestCase[] = []; // a list of Test Cases conforming to the TestCase interface
    const odrlEvaluator = new ODRLEvaluator();
    const comparison = ComplianceReportComparator.simple; // A comparison metric used to compare an expected and evaluated compliance report
    const testCaseEvaluator = new TestCaseEvaluator(odrlEvaluator, comparison); 

    // running the test suite, which does both the evaluations on all test cases and the comparison
    const evaluations: TestCaseEvaluation[] = await runTestSuite(testCases, testCaseEvaluator);
}
main()

Result of the test suite

Engine: ODRLEngineMultipleSteps Evaluator: ODRLEvaluator Comparison metric: Rule-Activation-similarity There were 67 ODRL evaluations correct compared to the Expected Compliance Report out of 68.