1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
=head1 NAME
heri-eval - evaluate classification algorithm
=head1 SYNOPSIS
B<heri-eval> [OPTIONS] I<dataset> [-- SVM_TRAIN_OPTIONS]
=head1 DESCRIPTION
B<heri-eval> runs training algorithm on I<dataset> and then
evaluate it using testing set, specified by option I<-e>.
If option I<-n> was applied,
cross-validation is used for evaluation, training and testing on different folds
are run in parallel, thus utilizing available CPUs. If I<-r> is used, the
dataset is splitted into training and testing datasets randomly with
the specified ratio, and then holdout is run.
=head1 OPTIONS
=over 6
=item B<-h, --help>
Display help information.
=item B<-f>
Enable output of per-fold statistics. See B<-M>I<f>.
=item B<-n> I<N>
Enable T*I<N>-fold cross-validation mode and set the number of folds to I<N>.
=item B<-r> I<ratio>
Split the dataset into training and testing parts with the specified ratio
of their sizes (in percents).
=item B<-t> I<T>
Enable I<T>*N-fold cross-validation mode and set the number of runs to I<T>
which 1 by default.
=item B<-e> I<testing_dataset>
Enable hold-out mode and set the testing dataset.
=item B<-T> I<threshold>
Set the minimum threshold for making a classification decision. If this flag is applied,
micro-average precision, recall, and F1 are calculated instead of accuracy.
=item B<-o> I<filename>
Save predictions from testing sets
to the specified file.
Format: outcome_class prediction_class [score]
=item B<-O> I<filename>
Save incorrectly classified objects
to the specified file.
Format: #object_number: outcome_class prediction_class [score])
=item B<-m> I<filename>
Save confusion matrix to the specified file.
Format: frequency : outcome_class prediction_class
=item B<-p> I<opts>
Pass the specified I<opts> to B<heri-stat(1)>.
=item B<-s> I<opts>
Pass the specified I<opts> to B<heri-split(1)>.
=item B<-M> I<chars>
Sets the output mode where chars are:
t -- output total statistics,
f -- output per-fold statistics,
c -- output cross-fold statistics.
The default is "-M tc".
=item B<-S> I<seed>
Pass the specified I<seed> to B<heri-split(1)>.
=item B<-K>
Keep temporary directory after exiting.
=item B<-D>
Turn on the debugging mode, implies -K.
=back
=head1 EXAMPLES
=over 1
heri-eval -e testing_set.libsvm training_set.libsvm -- -s 0 -t 0
export SVM_TRAIN_CMD='liblinear-train'
export SVM_PREDICT_CMD='liblinear-predict'
heri-eval -p '-mr' -n 5 training_set.libsvm -- -s 4 -q
heri-eval -p '-mr' -n 5 training_set.libsvm -- -s 4 -q
export SVM_TRAIN_CMD='scikit_rf-train --estimators=400'
export SVM_PREDICT_CMD='scikit_rf-predict'
heri-eval -p '-c' -Mt -t 50 -r 70 dataset.libsvm
=back
=head1 ENVIRONMENT
=over 6
=item I<SVM_TRAIN_CMD>
Training utility, e.g., liblinear-train
(the default is svm-train).
=item I<SVM_PREDICT_CMD>
Predicting utility, e.g., liblinear-predict
(the default is svm-predict).
=item I<SVM_HERI_STAT_CMD>
Utility for calculating statistics (the default is B<heri-stat(1)>).
=item I<SVM_HERI_STAT_ADDONS_CMD>
Utility for calculating additional statistics (the default is B<heri-stat-addons(1)>).
=item I<SVM_HERI_SPLIT_CMD>
Utility for splitting the dataset (the default is B<heri-split(1)>).
=item I<TMPDIR>
Temporary directory (the default is /tmp).
=back
=head1 HOME
L<http://github.com/cheusov/herisvm>
=head1 SEE ALSO
L<heri-split(1)>
L<heri-stat(1)>
|
