bash-universal-tester v1.9.5

Superquick installation via npm install -g bash-universal-tester

:white_check_mark: bash-universal-tester             

Universal testing script for bash

General purpose awesome testing-script

What?

Currently this script supports:

• Testing native executables with text input/out files (.in/.out)

• Testing native executables with .in/.out files packed to a zip file

• Using custom testing scripts

• Some nice types of output formatting

Requirements

This is stand-alone script.

• Works on Linux with bash shell

• Works on Windows (BUT YOU NEED BASH EMULATION LIKE CYGWIN)

Installation

• Install via npm:

npm install -g bash-universal-tester

• Download through provided link     

• Alternatively on bash type:

wget https://raw.githubusercontent.com/styczynski/bash-universal-tester/master/utest.sh && chmod u+x ./utest.sh && mv ./utest.sh utest && PATH=$PATH:$PWD

Basic usage

Basic usage: utest <prog>

The script tries to autodetect folder with input/output test files. And sometimes gives suggestions what program you may want to test.

Basic in + out + non empty err

To test input folder (only .in and .out):

utest <prog> <folder>

Basic in + out + ignore err (err is not checked anytime)

To test input folder (only .in and .out):

utest --tierr <prog> <folder>

Basic in + out + err (missing .err file cause error)

To test input folder (.in, .out, and .err files):

utest --tnerr <prog> <folder>

Basic in + out + err (missing .err files are ignored)

To test input folder (.in, .out, and .err files):

utest --tgerr <folder> <prog> <folder>

Globbing input files

Let's suppose we have the following file structure:

tests
|- test1 
|  |- input.in
|  \- output.out
|
|- test2
   |- input.in
   \- output.out

We want to feed utest with input files nested in subdirectories. For that purpose just use:

utest <prog> "./tests/**/*.in"

Note that globbing must be provided in quotes otherwise it will be parsed by shell and won't work!

Custom input layout

Let's suppose we have the following file structure (even more unfriendly!):

tests
|- test1 
|  |- input.txt
|  \- out
|
|- test2
   |- input.txt
   \- out

We want to feed utest with input files nested in subdirectories. And the input files have custom extensions. We must tell utest where to find output files. We use --tgout flag that utilizes dynamic variable to generate output path. You can read more about dynamic variables in variables section.

utest <prog> --tgout "%input_file_folder/out" "./tests/**/input.txt"

Note that globbing must be provided in quotes otherwise it will be parsed by shell and won't work!

Advanced usage

utest [test_flags] <prog> <dir> [prog_flags]

• <prog> is path to the executable, you want to test

• <dir> is the path to folder containing .in/.out files

• [prog_flags] are optional conmmand line argument passed to program <prog>

• [test_flags] are optional flags for test script

Wherever -help, --help flags are placed the script always displays its help info.

About minimalistic modes:

• In --tm mode OKs are not printed / erors are atill full with diff

• In --tmm mode errors are only generally descripted / OK at output on success

• In --tmmm only names of error files are printed / Nothing at output on success

Variables

In <prog>, --tgerr <dir>, --tgout <dir> and config files you can use special dynamic variables. These are the following:

Example usage:

utest "echo %input_file" <folder>

Moreover you can use formatting variables (that are set via formatting switches). Please use them instead of hard-coded values, because it's easy and improves customizability of your output.

Example usage:

input: ./test
executions:
    - prog
hooks:
    test_case_start:
        - @echo %{bwarn}Hello%{ewarn} %input_file %{bok} %ok_index %{eok}
prog:
    command: echo Something
    

Piping

Utest provides easy way to preprocess your input file or postprocess program outputs.

All you have to do is to use --tpipe-in <command>, --tpipe-out <command> or --tpipe-out-err <command>.

Pipes are provided with additional variables:

For example let's sort program output alphabetically:

utest.sh --tpipe-out "cat %input | sort > %output" <prog> <folder>

Advantage of pipes are that you do not modify in/out files directly. And you can test programs that may potentailly give not exactly the same answers but which are still correct.

Configuration file

Global configuration

Instead of passing all parameters by command line we offer the ability to put everything into single YAML file!

Utest seek for utest.yaml file in current directory. It can contain all configuration available via command line switches and flags!

All config options are listed there:

input: test/*.in
silent: false
good_output: test/%input_file_name.out
good_err: test/%input_file_name.err
need_error_files: false
testing_script_out: ignore
testing_script_err: ignore
executions:
    - prog1
    - prog2
prog1:
    cwd: ./somefolder
    command: ./test/totest.sh
    args: %input_file_name
    pipes_out:
        - echo 123 > %output
    pipes_in:
        - echo 123 > %output
    pipes_out_err:
        - echo 123 > %output
        - cat %input | echo 123 > %output
prog2:
    command: echo 99

Single test configuration

You can also configure environment for single test case! Just put xyz.config.yaml file next to your input file xyz.in.

All options of config file for single test are listed below:

prog2:
    cwd: ./somefolder
    args: %input_file_name some additional args
    input: override_input_file.in

You must identify program by the command it calls.

Hooks

You can provide hooks commands for any of testing lifecycle state.

All available hooks are:

• init - called when testing begins
• deinit - called when testing ends
• test_case_start - called when new test file is tested
• test_case_finish - called when the test file was tested
• test_case_fail - called when test fails
• test_case_fail_out - called when test fails on std output (launched after test_case_fail)
• test_case_fail_err - called when test fails on error output (launched after test_case_fail)
• test_case_fail_success - called when test succeeded

Please note that: You can add mutliple commands that are executed in order from up to the bottom. If the command begins with @ character then it's output is directly displayed. If not then utest can change it to be more readable to the user!

input: test/*.in
good_output: test/%input_file_name.out
need_error_files: false
executions:
    - prog1
    - prog2
hooks:
    init:
        - @echo Testing
        - @echo Prepare input...
    deinit:
        - @echo Goodbye
    test_case_fail:
        - @echo [%{input_file}]  Test case failed for %{param_prog}
    test_case_success:
        - @echo [%{input_file}]  Test case successed for %{param_prog}
    test_case_fail_out:
        - @echo [%{input_file}]  FAILED ON OUTPUT
    test_case_fail_err:
        - @echo [%{input_file}]  FAILED ON ERR
        - @echo Whats a shame
    test_case_start:
        - @echo New test case jsut started %{input_file}
    test_case_finish:
        - @echo The test case was finished
prog1:
    command: ./test/totest.sh
    args: %input_file_name
    pipes_out:
        - echo 15 > %output
prog2:
    command: echo 159

Custom output format

Using --tsilent flags allows only hooks to write output. So if you use @ sign along with hooks (see Hooks section) you can make utest output any format of the output you want!

Example of outputing ERR <file> only on errors.

hooks:
    test_case_fail:
        - @echo ERR %{input_file}

Simple enough, right?