@fua/module.testing v1.1.2

@fua/module.testing

Thoughts

• parameters: The necessary parameters for organizing and testing, e.g. applicant, connector, certs, routes. Should only be used as setup and not changed after starting the session.
  • A parameter should have an identifier which acts as the key for the parameter.
  • The value of a parameter can be only a native js type (boolean, number or string).
• identification: (see parameters)
  • Identifiers are also only parameters with a string value.
• properties: Any additional properties that are managed by the test session and not intended to be changed by any test.
  • The properties are meta information that are not set by the individual tests but for the test run. It would contain the starting and ending times of test runs, the testing system, operator of test, supervisor, etc.
• statements: During testing any validation results can be saved as statement about the tested component. Can also be used for claims.
  • Because statements must be used for a variety of different testing result and not defined specifically for one purpose, the model for statements must have much room for different expressions and claim types.
  • Statements have in common that they can be identified by some kind of identifier.
  • The creation or removal (historization) of a statement is subject to timestamping.
  • Types of statements include: IDS certification criteria, GAIA-X testing claims, custom testing topics.
• claims: (see statements)
• cache: (see statements, properties, history)
  • For cached validation results the statements and the history is the right place.
  • For cached runtime properties that should be shared throughout validations the properties is the right place.
• result: Must be generated out of the statements, maybe after the session has finished. A result can be decoupled from the test session if it is used in a verifiable presentation and a test session that is packed as a verifiable credential.
  • The statements are the results from the validation.
  • A complete testing result is the conclusion out of the complete test session.
• history: The history is important if a statement changes during one testing session, especially if it is a continued testing session.
  • The history is mainly for changing statements.
  • Historization of statements should only occur for a changing setup, e.g. component rework or fixed config.
• runs: A test session can be separated into multiple test runs. Although the statements of a test session can change, the statements in a run should not change, because that would mean that different validations concluded different results for the same statement.
  • Runs should differentiate component changes and configuration changes.
  • A run might need its individual parameters and properties.
  • The run concept could be an argument to save sessions in a folder and the run as separate file.
• logs: Logs can give more information about the flow of validation and testing and might help resolve issues.
  • Log entries can be overwhelming and should be sorted and categorized somehow.
  • Log entries must definitely contain a timestamp.
• errors: (see logs)
• timestamps: Timestamps are important for each statement, log and the history. They can also be used to denote the start and end of a test session.
• persistence: The session file must be saved as json with a context or as ld-format, e.g. turtle. Optionally multiple file formats might be supported.
  • A session could also be saved over multiple files. Therefore, a folder should be reserved for the session.
• serialization: (see persistence, presentation)
• presentation: Generating a verifiable credential might not be the task of the session class, because it requires certificates for the proof which are not useful otherwise for the session. Also, the usage of a verifiable credential as serialization format would not be practical. It can easily be archived though by using normalized json-ld as credential subject.
  • One kind of presentation can also be markdown to have a more human-readable format to look at.
• tokens: A token acts as identifiable object throughout a test. As different components in the testing framework produce additional results, the token can bundle those results and reference the individual test. This is important if the same testing function is called with different setups to differentiate them and attribute logs and results accordingly.

Components

Model

    ┌─────────────────────┐
   /  TestingComponents  / <────┐
  └─────────────────────┘       │               ┌────────────┐
                                │            ┌─>│ TestMethod │────┐
                                │            │  └────────────┘<─┐ │
  ┌─────────────────┐   ┌──────────────────┐ │  ┌──────────┐────┘ │
  │ TestingProvider │──>│ TestingEcosystem │─┼─>│ TestCase │──────┤
  └─────────────────┘   └──────────────────┘ │  └──────────┘<─┐   │
       ^                                     │  ┌───────────────┐ │
       │                                     └─>│ TestProcedure │─┤
       │                                        └───────────────┘ │
    ┌───────────┐                                                 │
   /  Testbed  /      ┌───────────┐     ┌──────────────┐          │
  └───────────┘       │ Statement │<─┐  │ TestToken    │          │
                      └───────────┘  ├──├──────────────┤<─────────┘
                       ┌──────────┐  │  │ - param      │
    ┌─────────────┐    │ LogEntry │<─┤  │ - data       │
   /  Testsuite  /     └──────────┘  │  │ - result     │
  └─────────────┘    ┌────────────┐  ├─>│ - validation │<─────────┐
       │             │ ErrorEntry │<─┘  │ - steps      │          │
       │             └────────────┘     └──────────────┘          │
       v                                                          │
  ┌─────────────────┐   ┌─────────────┐   ┌─────────┐             │
  │ TestingConsumer │──>│ TestSession │──>│ TestRun │─────────────┘
  └─────────────────┘   └─────────────┘   └─────────┘

namespace fua.module.testing {
    // TODO
}

Interface

Example

// TODO

Improvements

• PresentationGenerator: Generator for different presentations of the test session.
• Naming for test runs could be better to prevent collision of runs that get created during one second.
  • A different naming should support the same benefits of lexical order than unix time.
  • It can be argued whether its sensible to assume multiple test runs would occur in the same second.
• TypeScript Definition
  • https://www.typescriptlang.org/docs/handbook/declaration-files/publishing.html
  • https://www.typescriptlang.org/docs/handbook/declaration-files/templates/module-d-ts.html
  • https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html