1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
|
baycomp
=======
Baycomp is a library for Bayesian comparison of classifiers. (For those who
don't know what they do and how, and why you should use them instead of
testing null hypotheses, we prepared a
:doc:`short introduction for dummies <introduction>`).
Functions in the library compare two classifiers on one or on multiple data
sets. They compute three probabilities: the probability that the first
classifier has higher scores than the second, the probability that differences
are within the region of practical equivalence (rope), or that the second
classifier has higher scores. We will refer to this probabilities as
`p_left`, `p_rope` and `p_right`. If the argument `rope` is omitted (or set
to zero), functions return only `p_left` and `p_right`.
The region of practical equivalence (rope) is specified by the caller and
should correspond to what is "equivalent" in practice; for instance,
classification accuracies that differ by less than 1 % may be called
equivalent. So we can set a rope of 1 (if accuracies are on scale from 0 to
100, or to 0.01, if they are on a scale from 0 to 1).
Similarly, whether higher scores are better or worse depends upon the type
of the score.
The library can also plot the posterior distributions.
The library can be used in three ways.
1. Two :doc:`shortcut functions <functions>` can be used for comparison on
single and on multiple data sets. If `nbc` and `j48` contain a list of
average classification accuracies of naive Bayesian classifier and J48 on a
collection of data sets, we can call
>>> two_on_multiple(nbc, j48, rope=1)
(0.23124, 0.00666, 0.7621)
(Actual outputs may differ due to Monte Carlo sampling.)
With some additional arguments, the function can also plot the posterior
distribution from which these probabilities came.
2. Tests are packed into :doc:`test classes <classes>`. The above call is
equivalent to
>>> SignedRankTest.probs(nbc, j48, rope=1)
(0.23124, 0.00666, 0.7621)
and to get a plot, we call
>>> SignedRankTest.plot(nbc, j48, rope=1, names=("nbc", "j48"))
To switch to another test, use another class::
>>> SignTest.probs(nbc, j48, rope=1)
(0.26508, 0.13274, 0.60218)
3. Finally, we can construct and query sampled
:doc:`posterior distributions <posterior>`.
>>> posterior = SignedRankTest(nbc, j48, rope=1)
>>> posterior.probs()
(0.23124, 0.00666, 0.7621)
>>> posterior.plot(names=("nbc", "j48"))
Detailed documentation is given on the following pages.
.. toctree::
:maxdepth: 2
functions
classes
posterior
introduction
|
