1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
== Testing Graphite2 ==
During development Graphite2 is regularly checked against its test suite of over
a 100 test cases. This happens automatically on every check-in to the default
branch thanks to our continuous build server for Windows and Linux. Prior to
each major release it's tested using an automated fuzzing system, which checks
for robustness in the face of corrupted font files. This set of fuzz tests uses
valgrind to check for rogue memory accesses and runs several thousand tests on
each of the 4 major test fonts and takes considerably longer to run. We also
have a growing suite of fuzz test regressions culled from logs generated by the
above fuzz test system, which can be run with a make command. These are intended
to be run before bug fix release and other point releases.
=== Running the tests ===
==== Running the standard tests ====
The standard test suite, the same one run on every checkin to the graphite2
project repository, can be run with a simple:
----
make test
----
==== Runnging the fuzztest regressions ====
----
make fuzztest
----
These should be run before point and bug fix releases.
==== Running the full fuzz test script ====
----
full-fuzz-test.sh script [fuzztest options]
----
This script exercises graphite over 4 scripts Myanmar, Devangari, extended Latin
and Arabic using 4 fonts and text in the Myanmar, Nepalese, Yoroba and Arabic
languages. It uses the `fuzzcomparerender` script to fuzz every byte of each
font with a random value and enforce generous enough runtime and memory resource
limits to detect infinite loops or memory leaks. A successfull run of this
script will produce four empty log files. Passing `--valgrind` to the script
this is passed down to the `fuzztest` program which will run the test program
with valgrind, this increases the runtime of script considerably. Normally the
script can run all four tests within 24 hours, fully loading our 4 core
hyperthreaded Xeon/Core i7 system. Using the valgrind option this takes
approximately a week on the same system.
==== Running fuzzcomparerender ====
----
tests/fuzzcomparerender <font name> <text name> [-r] [fuzztest options]
----
The `font name` must be the basename of a font file under `tests/fonts` and the
`text name` must be the basename of a text file from `tests/texts`. The `-r`
option if present is passed to comparerenderer and tells it the text is
right-to-left. Any further options are passed to the fuzztest program, this is
typically one of `--valgrind` or `--input`, or less frequently `--passes`,
`--jobs` or `--status`. See `tests/fuzztest --help` for more information. This
script runs `fuzztest` so that it corrupts every byte of the required TrueType
and Graphite specific tables with a random value, but excludes OpenType and AAT
related tables. It also imposes a runtime limit of 10 seconds (most test should
compete in a fraction of a second) and a memory limit 200MiB, again a normal run
should only use a tiny fraction of that. If the `comparerenderer` test
segfaults, exceeds those limits or returns an error value it is logged to a file
named on the following pattern: `fuzzfont-<font name>-<text name>.log`, which is
written to the script's current directory.
==== Running fuzztest ====
----
tests/fuzztest --font=font [options] -- <test harness>
----
A multiprocess fuzz test framework that generates corrupt fonts to run a user
supplied test harness against. This will check each byte in every table of a
TTF or subset of it's tables if specified, by overwriting with a random or user
specified value. Using the `--input` option it can also re-run tests using
previous output logs as input, it will ignore any line that doesn't match the
format of a fuzz line generated by the program, so such input fuzzes can be well
annotated. It is this facility that is used to drive the `make fuzztest`
regression check.
By default this will try to ensure there is always one test harness running on
each OS reported core at all times, the `--jobs` option can be used to limit
this if need to limit the load. Unless told otherwise the program will display a
status line inidcating the percentage of the tests run so far, the rate of
testing and an estimated time and date for completion. Once the status has
updated 4-5 times the estimate usually settles down to a frequently accurate
estimate.
=== Adding fuzz tests ===
Fuzz regression test files are simply copies of the log generated by the
`fuzzcomparerender` script. Once you have a log that generates a test case, you
can edit it to produce a more targeted set of fuzz lines (a log can generate
several different bugs). You should try to produce a minimal set of fuzz lines
for that particular test case, however the more fuzz lines that casue the same
bug the better.
The test case should be placed as follows:
----
tests/fuzz-tests/<font name>/<text name>/<bug description>.fuzz
----
where:
font name::
The name of a font, minus the .ttf extension from tests/fonts.
text name::
The name of a text file, minus .txt extension from tests/texts.
bug description::
A short description of the bug, this cannot contain any ":" characters or
other characters forbidden by Windows or Unix filename schemes. For
`class::member` references use an `_` e.g. `class_member`.
=== LibFuzzer based fuzzing ===
We now build a number of Clang libFuzzer test targets in
`<src>/tests/fuzz-tests` whenever `fuzzer` is one of the sanitizers found in
`GRAPHITE2_SANITIZERS`. There are a series of seed inputs in the
`<src>/tests/fuzz-tests/libfuzzer-corpus` directory each one coresponding to
one of the existing unit tests pairing a font with text and parameters. These
are binary files which consist of:
.libFuzzer target seed file-format for fuzz targets
[cols="3,10",options="unbreakable,header,compact",width="90%",frame="topbot",grid="none"]
|=============================================================================
| Length | Input parameter
| <variable> | TrueType file of font
| utint16 * N | Feature values (N = number of feature ids in the font)
| uint32_t | Text Script ID
| uint8_t | UTF encoding form uint size (1,2 or 4) see `gr_encform`
| uint32_t | Text directionality see `gr_bidirtl`
| uint_8_t[128] | Text in the UTF encoding form specified
|=============================================================================
Run as follows:
----
cd <src>/build
cp ../tests/fuzz-tests/libfuzz-corpus corpus
tests/fuzz-tests/gr-fuzzer-font corpus
----
To rerun a test just do:
----
tests/fuzz-tests/gr-fuzz-font crash-test-case-produced-by-fuzzer
----
See the libFuzzer documentation for more advanced options standard to all
libFuzzer produced fuzzing targets.
|
