1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
GStreamer QA system TODO
Milestone 1
-----------
Goal : Compatible behaviour as gst-media-test, but using the new API.
* bin/gst-media-test
* document the usage
* testrun
* Add possibility to modify the environment
* Monitor support
* gdb/backtrace monitor
* Reporting support
Milestone 2
-----------
Goal : Database support
* DB storage support
* Store monitors
* Remote DB storage support
* Might have to be done with Milestone 3+4
* Important speed issue for search
* Add a 'similarity' table that contains tests that have:
* similar type and
* similar arguments
* This can be done either on the first run of compare.py or
when importing one database in another.
Milestone 3
-----------
Goal : Centralized scheduling/control system
Using buildbot
Milestone 4
-----------
Goal : Search/Query/Reporting centralized system
Non-milestone-specific / Bonus features
---------------------------------------
* Speed optimizations
* DBusRemoteTest : Add features so that, for some tests, we don't spawn a new
remote process for every instance, but instead reuse existing ones (and
re-spawning new instances if that one crashes).
* Smarter introspection
* Make __test_arguments__ have more information :
* default value
* full documentation on that argument
* Make __test_checklist__ have more information :
* full documentation , especially explaining what could have caused
that step to not be validated, to make debugging/fixing easier.
* Ordered list of tuples instead of dictionnaries of tuples
* Maybe this could also apply to other properties
* Database
* Make sure we use cPickle only for generic (non-gstqa-specific) objects
* Add convenience functions for merging two DB together (or one in another).
* Add methods for updating old-versioned DB
* Tests
* Add some form of versioning so we can track changes.
* Maybe upgrade version based on variations of arguments/checklist/extrainfo/behaviour
* DBUS
* Fix issue in a 'asv' when one of the values is a list, we currently avoid this
by removing all values of type 'list'
* Regression testing of GStreamer modules
* Currently we use the old gst-media-test system (with all its inconveniences).
* Create an easy script to test regressions
DBUS ISSUES
-----------
* the type detection is extremely dumb for integers.
Ex : python integers have a maximum range of 2**63-1 on 64bit
-> this will be considered by dbus-python as an int
-> dbus-python will convert that to an Int32
-> BOOM
We currently solve this by pre-processing some values to make sure
they have the correct type.
|
