1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
|
=============
Term Prefixes
=============
Xapian itself doesn't put any restrictions on the contents of a term, other
than that terms can't be empty, and there's an upper limit on the length
(which is backend dependent - flint and quartz allow just over 240 characters,
except that zero bytes count double in this length).
However, Omega and ``Xapian::QueryParser`` impose some rules to aid
interoperability and make it easier to write code that doesn't require
excessive configuring. It's probably wise to follow these rules unless
you have a good reason not to. Right now you might not intend to use Omega
or the QueryParser, not to combine a search with another database. But if
you later find you do, it'll be much easier if you're using compatible
rules!
The basic idea is that terms won't begin with a capital letter (since they're
usually lower-cased and often stemmed), so any term which starts with a capital
letter is assumed to have a prefix. For all letters apart from X, this is a
single character prefix and these have predefined standard meanings (or are
reserved for standard meanings but currently unallocated).
X starts a multi-capital letter user-defined prefix. If you want a prefix for
something without a standard prefix, you create your own starting with an X
(e.g. XSHOESIZE). The prefix ends with the first non-capital. If the term
you're prefixing starts with a capital, add a ":" between prefix and term to
resolve ambiguity about where the prefix ends and the term begins.
Here's the current allocation list:
A
Author
D
Date (numeric format: YYYYMMDD or "latest" - e.g. D20050224 or Dlatest)
G
newsGroup (or similar entity - e.g. a web forum name)
H
Hostname
K
Keyword
L
ISO Language code
M
Month (numeric format: YYYYMM)
N
ISO couNtry code (or domaiN name)
P
Pathname
Q
uniQue id
R
Raw (i.e. unstemmed) term (unused by Xapian since 1.0.0)
S
Subject (or title)
T
mimeType
U
full URL of indexed document - if the resulting term would be > 240
characters, a hashing scheme is used to prevent overflowing
the Xapian term length limit (see omindex for how to do this).
W
"weak" (approximately 10 day intervals, taken as YYYYMMD from
the D term, and changing the last digit to a '2' if it's a '3')
(unused by Xapian since 0.9.7)
X
longer prefix for user-defined use
Y
year (four digits)
Z
stemmed term
Reserved but currently unallocated: BCEFIJOV
There are two main uses for prefixes - boolean filters and probabilistic
(i.e. free text) fields.
Boolean Filters
===============
If the documents being indexed represent people, you might have a gender
field (e.g. M for Male, F for Female, X for Unknown). Gender doesn't have
a standard prefix, so you might allocated "XGENDER". And then lowercase
the field contents to avoid needing to always add a colon. So documents
will be indexed by one of XGENDERm, XGENDERf, or XGENDERx.
If you're indexing using scriptindex, and have a field in the input file
which can be "gender=M", etc, then your index script would have a rule
such as::
gender : lower boolean=XGENDER
You can then restrict a search in Omega by passing a B parameter with one
of these as the value, e.g. B=XGENDERf
In your HTML search form, you can allow the user to select this using a set of
radio buttons::
Gender:<br>
<input type="radio" name="B" value=""> any<br>
<input type="radio" name="B" value="XGENDERf"> female<br>
<input type="radio" name="B" value="XGENDERm"> male<br>
If you want to have multiple sets of radio buttons for selecting different
boolean filters, you can make use of Omega's preprocessing of CGI parameter
names by calling them "B 1", "B 2", etc (names are truncated at the first
space - see `cgiparams.html <cgiparams.html>`_ for full details).
You can also use a select tag::
Gender:
<select name="B">
<option value="">any</option>
<option value="XGENDERf">female</option>
<option value="XGENDERm">male</option>
<option value="XGENDERx">unknown</option>
</select>
You can also allow the user to restrict a search with a boolean filter
specified in text query (e.g. sex:f -> XGENDERf) by adding this to the
start of your omegascript template::
$setmap{boolprefix,sex,XGENDER}
Multiple aliases are allowed::
$setmap{boolprefix,sex,XGENDER,gender,XGENDER}
This decoupling of internal and external names is also useful if you want
to offer search frontends in more than one language, as it allows the
prefixes the user sees to be translated.
Probabilistic Fields
====================
Say you want to index the title of the document such that the user can
search within the title by specifying title:report (for example) in their
query.
Title has standard prefix S, so you'd generate terms as normal, but then
add an "S" prefix. If you're using scriptindex, then you do this by
adding "index=S" to the scriptindex rule like so::
title : field=title index=S
You then need to tell Xapian::QueryParser that "title:" maps to an "S" prefix.
If you're using Omega, then you do so by adding this to your omegascript
template (at the start is best)::
$setmap{prefix,title,S}
Or if you're writing your own search frontend, like this::
Xapian::QueryParser qp;
qp.add_prefix("subject", "S");
// And similar lines for other probabilistic prefixes...
// And any other QueryParser configuration (e.g. stemmer, stopper).
Xapian::Query query = qp.parse_query(user_query_string);
You can add multiple aliases for a prefix (e.g. title and subject for S), and
the decoupling of "UI prefix" and "term prefix" means you can easily translate
the "UI prefixes" if you have frontends in different languages.
Note that if you want words from the subject to be found without a prefix, you
need to generate unprefixed terms as well as the prefixed ones.
|
