1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
|
Input validation for *Units*
---------------------------------------------------------------------
:Maintainer: Masatake YAMATO <yamato@redhat.com>
----
We have to maintain parsers for languages that we don't know well. We
don't have enough time to learn the languages.
*Units* test cases help us not introduce wrong changes to a parser.
However, there is still an issue; a developer who doesn't know a
target language well may write a broken test input file for the
language. Here comes "Input validation."
You can validate the test input files of *Units* with *validate-input*
make target if a validator for a language is defined.
How to run and an example session
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here is an example validating an input file for JSON.
.. code-block:: console
$ make validate-input VALIDATORS=jq
...
Category: ROOT
------------------------------------------------------------
simple-json.d/input.json with jq valid
Summary
------------------------------------------------------------
#valid: 1
#invalid: 0
#skipped (known invalidation) 0
#skipped (validator unavailable) 0
This example shows validating *simple-json.d/input.json* as an input
file with *jq* validator. With VALIDATORS variable passed via
command-line, you can specify validators to run. Multiple validators
can be specified using a comma-separated list. If you don't give
VALIDATORS, the make target tries to use all available validators.
The meanings of "valid" and "invalid" in "Summary" are apparent. In
two cases, the target skips validating input files:
#skipped (known invalidation)
A test case specifies KNOWN-INVALIDATION in its *validator* file.
#skipped (validator unavailable)
A command for a validator is not available.
*validator* file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*validator* file in a *Units* test directory specifies which
validator the make target should use.
.. code-block:: console
$ cat Units/simple-json.d/validator
jq
If you put *validator* file to a category directory (a directory
having *.r* suffix), the make target uses the validator specified in
the file as default. The default validator can be overridden with a
*validator* file in a subdirectory.
.. code-block:: console
$ cat Units/parser-puppetManifest.r/validator
puppet
# cat Units/parser-puppetManifest.r/puppet-append.d/validator
KNOWN-INVALIDATION
In the example, the make target uses *puppet* validator for validating
the most of all input files under *Units/parser-puppetManifest.r*
directory. An exception is an input file under
*Units/parser-puppetManifest.r/puppet-append.d* directory. The
directory has its specific *validator* file.
If a *Unit* test case doesn't have *expected.tags* file, the make
target doesn't run the validator on the file even if a default
validator is given in its category directory.
If a *Unit* test case specifies KNOWN-INVALIDATION in its *validator*
file, the make target just increments "#skipped (known invalidation)"
counter. The target reports the counter at the end of execution.
validator command
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A validator specified in a *validator* file is a command file put
under *misc/validators* directory. The command must have "validator-"
as prefix in its file name. For an example,
*misc/validators/validator-jq* is the command for "jq".
The command file must be an executable. *validate-input* make target
runs the command in two ways.
*is_runnable* method
Before running the command as a validator, the target runs
the command with "is_runnable" as the first argument.
A validator command can let the target know whether the
validator command is runnable or not with exit status.
0 means ready to run. Non-zero means not ready to run.
The make target never runs the validator command for
validation purpose if the exit status is non-zero.
For an example, *misc/validators/validator-jq* command uses *jq*
command as its backend. If *jq* command is not available on a
system, *validator-jq* can do nothing. If such case,
*is_runnable* method of *validator-jq* command should exit with
non-zero value.
*validate* method
The make target runs the command with "validate* and an input
file name for validating the input file. The command exits
non-zero if the input file contains invalid syntax. This method
will never run if *is_runnable* method of the command exits with
non-zero.
|
