fhtagn.awk is a tiny CLI tool for literate testing for command-line programs.
File tests.fhtagn:
$ command --that --should --execute correctly
| expected stdout output
$ command --that --will --cause error
@ expected stderr output
? expected-exit-code
Run the tests:
./fhtagn.awk tests.fhtagnIn fact, this is a re-implementation of darius/tush, adolfopa/tush. But simpler (single tiny AWK script) and faster, because:
- it uses
/dev/shmwhere available instead of/tmp - it compares the expected result with the actual in the code and only calls
diffto show the difference if they don't match - it doesn't create a sandbox folder for each test
- it doesn't use
mktempbut rather generates random name in the code
./fhtagn.awk f1.fhtagn [ f2.fhtagn [ f3.fhtagn [...] ] ]This will stop on the first error found.
Example:
$ ./fhtagn.awk tests/1.fhtagn tests/2.fhtagn tests/3.fhtagn
tests/2.fhtagn:12: $ echo "hello world"; echo "error msg" >&2; exit 7
--- expected
+++ actual
@@ -1,4 +1,4 @@
| hello world
-@ error msg 444
-? 8
+@ error msg
+? 7
Set ALL=1 environment variable.
This runs all tests, collects all errors, and shows the stats at the end.
ALL=1 ./fhtagn.awk f1.fhtagn [ f2.fhtagn [ f3.fhtagn [...] ] ]Useful for running in CI.
Example:
$ ALL=1 ./fhtagn.awk tests/1.fhtagn tests/2.fhtagn tests/3.fhtagn
tests/2.fhtagn:12: $ echo "hello world"; echo "error msg" >&2; exit 7
--- expected
+++ actual
@@ -1,4 +1,4 @@
| hello world
-@ error msg 444
-? 8
+@ error msg
+? 7
tests/3.fhtagn:7: $ echo bbb
--- expected
+++ actual
@@ -1,2 +1,2 @@
-| BBB
+| bbb
result=FAIL, failure=2, success=4, total=6, files=3
Use |~pattern to match a stdout line or @~pattern to match a stderr line:
$ echo 'version: 1.2.3'; echo 'warning: 42' >&2
|~version: [0-9]+\.[0-9]+\.[0-9]+
@~warning: [0-9]+
Patterns use AWK extended regular expressions and must match the whole line.
For a substring match, surround the pattern with .*, for example
|~.*connection refused.*. Each assertion matches exactly one line; extra or
missing output lines still fail, and output order and stream must match.
Literal and regex assertions can be mixed. Exit-code assertions remain literal.
Everything after ~ is the pattern, including whitespace and backslashes.
There are no /…/ delimiters or additional escaping: use \. for a literal dot.
|~ abc requires a leading space, |~ matches only an empty line, and |~.*
matches any single line, including an empty one. To expect literal output starting
with ~, use the usual space: | ~text.
On failure, the diff shows unmatched patterns against actual output. Successful regex assertions appear unchanged in the diff, even when another assertion fails.
Patterns must be valid for your AWK implementation. An invalid regex causes a
native AWK error and aborts the run, even with ALL=1. It may leave temporary
files because AWK can skip the cleanup in END on such errors.
You can use variable substitution to parametrize tests. This works like so:
# in your .fhtagn file you have
$ command {{VAR1}} '{{VAR2}}'
| expected output
And you run it like so:
./fhtagn.awk tests.fhtagn VAR1=value VAR2='other value'The example use case here is to check the correctness of the same command when ran by different versions of a programming language. This is what we do to test fhtagn itself with different AWKs: link1, link2, link3.
Note that in case of test failure, you will see an expanded command line, which is helpful for debugging.