1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
|
Copyright (C) 2004-2009 The PaGMO development team,
Advanced Concepts Team (ACT), European Space Agency (ESA)
This directory contains the data for the hypervolume indicator tests.
Purpose of this test suite is to verify the validity of the algorithms, and comparison on common data.
These tests do not validate the proper usage of the classes or the user interface - tests validating that can be found in PyGMO test suite.
************************************
* 1. Structure of this directory *
************************************
This directory contains the following:
- testcases_list.txt
- testcases/
- README (file you're reading right now)
testcases/ directory contains the individual test files, often sharing similar characteristics or taken from given source.
Single test file under testcases/ contains T fronts, each describing a front of dimension D and the size of N (number of points).
The name of the test usually contains the information on T,D and N, e.g.: file "testcases/c_max_t1_d3_n128" contains a single front with D = 3 and N = 128.
Some names the values N or T if the size/dimensions are inconsistent inside the file or irrelevant if the test is small (we provide values D, N and T so its easier to estimate how long given test will run).
************************************
* 2. The testcases_list.txt file *
************************************
testcases_list.txt file contains a list of files from testcases/ that are run by default when the "make test" is executed.
Given test description in testcases_list.txt consists of the following:
[test_type] [algorithm_name] [testcase_filename] [epsilon]
[test_type] can tale the following values (and is equivalent to the name of the method inside pagmo::util::hypervolume class):
compute - hypervolume computation test (hypervolume indicator)
exclusive - exclusive hypervolume computation test
least_contributor - test for the least contributor
greatest_contributor - test for the greatest contributor
[algorithm_name] can take the following value (and is equivalent to the class name under pagmo::util::hv_algorithm namespace):
hv2d
hv3d
hv4d
hoy
wfg
bf_approx
bf_fpras
[testcase_filename] is a path to the test file (relatively from the hypervolume_test_data/testcases/ directory!)
[epsilon] is a floating point value describing the accuracy. It is used to determine whether given algorithm "passes" the test (i.e. (test_answer - algorithm_answer) < abs([epsilon])).
*********************************
* 3. Structure of a test file *
*********************************
Single test file is written in the following format:
T
D
N
R (reference point)
[N D-dimensional points]
[optional parameter for "exclusive" test type]
[correct answer]
Contents of a simple 2-dimensional test may look like this (minus the "#" comments):
1 # definition of a single test will follow
2 # 2-dimensional front
2 # with 2 points
3 3 # reference point at (3,3)
2 1 # first point
0.5 1.7 # second point
3.95 # value of the hypervolume
Format above is the same for the test type "compute", "least_contributor" and the "greatest_contributor".
Format differs slightly for the "exclusive" tests, as we need to provide the index of the point we are interested in.
Let us look at the file testcases/e_max_d2:
2 # 2 descriptions of fronts will follow
2 # 2-dimensional front
3 # with 3 points
1 1 # reference point is at (1,1)
0 0.7 # first point
0.5 0.5 # second point
0.9 0 # third point
2 # exclusive hypervolume of point at index 2 (this extra line exists only in "exclusive" test type)
0.05 # answer
2 # 2-dimensional front (description of the next front follows)
1 # single point
1 1 # reference point at (1,1)
0 0 # first point
0 # exclusive hypervolume of point at index 0
1.0 # answer
For more detail on each algorithm, please refer to the corresponding class under pagmo::util::hv_algorithm.
Please notice that some algorithms have certain limitations, e.g. hv4d algorithm will refuse to work with 3-dimensional fronts etc.
********************************
* 4. Origin of the testcases *
********************************
Some tests were generated by us, using pagmo DTLZ benchmark problems, and then the answer to given hypervolume problem was derived using the Fast Dimension-Sweep algorithm (http://ls11-www.cs.uni-dortmund.de/rudolph/hypervolume/start):
For larger fronts, we used the DTLZ fronts provided by the Walking Fish Group (http://www.wfg.csse.uwa.edu.au/hypervolume/) and reformatted them to fit our test suite, the answer was derived using the implementation of WFG (version 1.03): (http://www.wfg.csse.uwa.edu.au/hypervolume/code/WFG_1.03.tar.gz)
*********************************
* 5. How to add a custom test *
*********************************
Adding a custom test is very simple:
Prepare the front in a format described in the point 3 above (NOTE: Every hypervolume algorithm in pagmo assumes minimization).
Either add the entry with the file to the testcases_list.txt file, or execute the ./hypervolume_test directly, providing a custom testcases file as the first argument:
(navigate to pagmo/build/tests/)
./hypervolume_test my_testcases.txt
NOTE: paths to the testcase files inside the custom testcases_list file must describe the path to the test as it goes from the /tests/hypervolume_test_data/testcases/ directory!
|
