1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
|
# test-file-plugins
Version 0.5
Jacob Boerema, 2021-2024
## Introduction
This is a Python 3 framework for testing GIMP file import and export plug-ins.
It can be run both as a plug-in from within GIMP (in case it is installed),
and from the command line in batch mode.
## Installation and set-up
1. For running as a plug-in in GIMP this plug-in needs to be in a location
that GIMP knows of for plug-ins. By default it is only installed for
unstable builds. For using the batch version, it is not necessary to
have it installed.
2. There is a `tests` folder inside the plug-in folder for
configuration files. The main configuration file is
`config.ini` for the plug-in; I use a separate `batch-config.ini` for
command line testing, but that's not required.
This configuration file is where you define the file import and export
plug-ins to test. Each plug-in has its own separate configuration
file that defines the tests and test files which should be in the same
folder.
The location and name of the main configuration file can be changed by
setting an environment variable: `GIMP_TESTS_CONFIG_FILE`.
3. The plug-in specific configuration file, e.g.
`png-tests.ini` defines the specific test sets. This configuration file is
expected to be in the same location as the main configuration file.
Each test set can be enabled/disabled and needs to
have a folder set where the test images can be found.
This folder is expected to be relative to the base test data folder.
By default this is the `./data/` folder, but it can also be
changed, see below.
The specific test images are defined in a file set in `files=`. These are
expected to be in the same folder as the images they are testing.
The root folder for all test images is by default `./data/`. This folder can
be changed by setting the environment variable `GIMP_TESTS_DATA_FOLDER`.
4. The file that defines the images to be tested, should be a text file
where each line contains the filename of one test image, optionally
followed by a ',' and then a code for the expected result.
Currently there are four results defined:
- `EXPECTED_OK` (the default if omitted, if loading should succeed)
- `EXPECTED_FAIL` (if loading should fail)
- `EXPECTED_TODO` (if loading is currently expected to fail, but the
intention is to fix this in the future)
- `SKIP` (this temporarily skips testing a file, useful when a file causes
problems working on other test files) Note: this is **only** intended
for temporary debugging purposes. They should not be committed like that
to our repository.
We may add `EXPECTED_NO_CRASH` in the future, to define fuzzed images
where we don't care whether they load or not, as long as there is no crash.
## How to run
Start GIMP, then start the plug-in from:
- Menu Filters, Development, Python-Fu, Test file import plug-ins.
Note: this plug-in is not installed for stable builds.
Or, from the commandline or a shell script:
```
export GIMP_TESTS_CONFIG_FILE="/location/of/config.ini"
export GIMP_TESTS_LOG_FILE="/location/of/logfile.log"
export GIMP_TESTS_DATA_FOLDER="/location/of/test-images/rootfolder/"
export PYTHONPATH="/location/of/your/python/script/"
cd location/of/plug-in/gimp-file-plugin-tests
cat batch-import-tests.py | gimp-console-2.99 -idf --batch-interpreter python-fu-eval -b - --quit
```
In case you run the batch version, the log file is written to the folder
where the plug-in resides. This location can be adjusted by setting the
environment variable `GIMP_TESTS_LOG_FILE`. Note that the path to this file
needs to exist!
To test for success or regressions in batch mode you can test whether there
were regressions.
You should grep for `Total number of regressions: 0`.
### Status
- The only thing currently being tested is whether the image loads or not. No
testing is done to see if the image is correctly shown.
- When running all tiff tests from inside GIMP, it may crash, at least on
Windows, due to the error console running out of resources because of
the amount of messages.
- There is some preliminary export testing support, see `batch-export-tests.py`,
but it is incomplete. That is: there is no error tracking yet, nor support
for comparing exported images with reference images.
### File import plug-ins being tested
At this moment the following file import plug-ins have
tests available:
- file-bmp
- file-dcx
- file-dds
- file-fits
- file-gif
- file-jpeg
- file-pcx
- file-png
- file-pnm
- file-psb
- file-psd
- file-psd-load-merged
- file-psp
- file-sgi
- file-tga
- file-tiff
## Future enhancements
- Add tests for all our image loading plug-ins.
- Test if loaded images are loaded correctly, i.e.
do they look like they should.
- Fuzzing of images before loading to test our handling of corrupt images.
We may need to add another result type: `EXPECTED_NO_CRASH`, meaning we
don't care whether it loads or not, as long as there is no crash.
- GUI to select/deselect tests.
- Improve file export testing.
- It would be nice to be able to capture all stdout/stderr output and
possibly filter it for crashes, criticals and warnings, and add that to
the log file. Maybe with an option to filter out certain kinds of messages.
|
