File: overview.html

package info (click to toggle)
xapian-omega 1.4.3-3
links: PTS, VCS
area: main
in suites: stretch
size: 3,684 kB
ctags: 1,703
sloc: cpp: 13,129; sh: 4,529; perl: 864; makefile: 322
file content (1239 lines) | stat: -rw-r--r-- 50,425 bytes
parent folder | download | duplicates (2)
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.13.1: http://docutils.sourceforge.net/" />
<title>Omega overview</title>
<style type="text/css">

/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $
:Copyright: This stylesheet has been placed in the public domain.

Default cascading style sheet for the HTML output of Docutils.

See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/

/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
  border: 0 }

table.borderless td, table.borderless th {
  /* Override padding for "table.docutils td" with "! important".
     The right padding separates the table cells. */
  padding: 0 0.5em 0 0 ! important }

.first {
  /* Override more specific margin styles with "! important". */
  margin-top: 0 ! important }

.last, .with-subtitle {
  margin-bottom: 0 ! important }

.hidden {
  display: none }

.subscript {
  vertical-align: sub;
  font-size: smaller }

.superscript {
  vertical-align: super;
  font-size: smaller }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dl.docutils dd {
  margin-bottom: 0.5em }

object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
  overflow: hidden;
}

/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

/* Uncomment (and remove this text!) to get reduced vertical space in
   compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
  margin-bottom: 0.5em }

div.compound .compound-last, div.compound .compound-middle {
  margin-top: 0.5em }
*/

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em ;
  margin-right: 2em }

div.footer, div.header {
  clear: both;
  font-size: smaller }

div.line-block {
  display: block ;
  margin-top: 1em ;
  margin-bottom: 1em }

div.line-block div.line-block {
  margin-top: 0 ;
  margin-bottom: 0 ;
  margin-left: 1.5em }

div.sidebar {
  margin: 0 0 0.5em 1em ;
  border: medium outset ;
  padding: 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
  margin-top: 0.4em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr.docutils {
  width: 75% }

img.align-left, .figure.align-left, object.align-left, table.align-left {
  clear: left ;
  float: left ;
  margin-right: 1em }

img.align-right, .figure.align-right, object.align-right, table.align-right {
  clear: right ;
  float: right ;
  margin-left: 1em }

img.align-center, .figure.align-center, object.align-center {
  display: block;
  margin-left: auto;
  margin-right: auto;
}

table.align-center {
  margin-left: auto;
  margin-right: auto;
}

.align-left {
  text-align: left }

.align-center {
  clear: both ;
  text-align: center }

.align-right {
  text-align: right }

/* reset inner alignment in figures */
div.align-right {
  text-align: inherit }

/* div.align-center * { */
/*   text-align: left } */

.align-top    {
  vertical-align: top }

.align-middle {
  vertical-align: middle }

.align-bottom {
  vertical-align: bottom }

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font: inherit }

pre.literal-block, pre.doctest-block, pre.math, pre.code {
  margin-left: 2em ;
  margin-right: 2em }

pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

span.section-subtitle {
  /* font-size relative to parent (h1..h6 element) */
  font-size: 80% }

table.citation {
  border-left: solid 1px gray;
  margin-left: 1px }

table.docinfo {
  margin: 2em 4em }

table.docutils {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.footnote {
  border-left: solid 1px black;
  margin-left: 1px }

table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

table.docutils th.field-name, table.docinfo th.docinfo-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap ;
  padding-left: 0 }

/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
  border: 0px;
  border-top: 2px solid;
  border-bottom: 2px solid;
  border-collapse: collapse;
}
table.docutils.booktabs * {
  border: 0px;
}
table.docutils.booktabs th {
  border-bottom: thin solid;
  text-align: left;
}

h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
  font-size: 100% }

ul.auto-toc {
  list-style-type: none }

</style>
</head>
<body>
<div class="document" id="omega-overview">
<h1 class="title">Omega overview</h1>

<p>If you just want a very quick overview, you might prefer to read the
<a class="reference external" href="quickstart.html">quick-start guide</a>.</p>
<p>Omega operates on a set of databases.  Each database is created and updated
separately using either omindex or <a class="reference external" href="scriptindex.html">scriptindex</a>.  You can
search these databases (or any other Xapian database with suitable contents)
via a web front-end provided by omega, a CGI application.  A search can also be
done over more than one database at once.</p>
<p>There are separate documents covering <a class="reference external" href="cgiparams.html">CGI parameters</a>, the
<a class="reference external" href="termprefixes.html">Term Prefixes</a> which are conventionally used, and
<a class="reference external" href="omegascript.html">OmegaScript</a>, the language used to define omega's web
interface.  Omega ships with several OmegaScript templates and you can
use these, modify them, or just write your own.  See the &quot;Supplied Templates&quot;
section below for details of the supplied templates.</p>
<p>Omega parses queries using the <tt class="docutils literal"><span class="pre">Xapian::QueryParser</span></tt> class - for the supported
syntax, see queryparser.html in the xapian-core documentation
- available online at: <a class="reference external" href="https://xapian.org/docs/queryparser.html">https://xapian.org/docs/queryparser.html</a></p>
<div class="section" id="term-construction">
<h1>Term construction</h1>
<p>Documents within an omega database are stored with two types of terms:
those used for probabilistic searching (the CGI parameter 'P'), and
those used for boolean filtering (the CGI parameters 'B' and 'N' - the
latter is a negated variant of 'B' and was added in Omega 1.3.5).</p>
<p>Boolean terms start with an initial capital letter denoting the 'group' of the
term (e.g. 'M' for MIME type), while probabilistic terms are all lower-case,
and are also stemmed before adding to the database.</p>
<p>The &quot;english&quot; stemmer is used by default - you can configure this for omindex
and scriptindex with <tt class="docutils literal"><span class="pre">--stemmer=LANGUAGE</span></tt> (use <tt class="docutils literal"><span class="pre">--stemmer=none</span></tt> to disable
stemming, see omindex <tt class="docutils literal"><span class="pre">--help</span></tt> for the list of accepted language names).  At
search time you can configure the stemmer by adding <tt class="docutils literal">$set{stemmer,LANGUAGE}</tt>
to the top of your OmegaScript template.</p>
<p>The two term types are used as follows when building the query:</p>
<p>The 'P' parameter is parsed using <cite>Xapian::QueryParser</cite> to give a
<cite>Xapian::Query</cite> object denoted as <cite>P-terms</cite> below.</p>
<p>There are two ways that 'B' and 'N' parameters are handled, depending if the
term-prefix has been configured as &quot;non-exclusive&quot; or not.  The default is
&quot;exclusive&quot; (and in versions before 1.3.4, this was how all 'B' parameters
were handled).</p>
<div class="section" id="exclusive-boolean-prefix">
<h2>Exclusive Boolean Prefix</h2>
<p>B(oolean) terms from 'B' parameters with the same prefix are ORed together,
like so:</p>
<pre class="literal-block">
      [   OR   ]
     /    | ... \
B(F,1) B(F,2)...B(F,n)
</pre>
<p>Where B(F,1) is the first boolean term with prefix F from a 'B' parameter, and
so on.</p>
</div>
<div class="section" id="non-exclusive-boolean-prefix">
<h2>Non-Exclusive Boolean Prefix</h2>
<p>For example, <tt class="docutils literal">$setmap{nonexclusiveprefix,K,true}</tt> sets prefix <cite>K</cite> as
non-exclusive, which means that multiple filter terms from 'B' parameters will
be combined with &quot;AND&quot; instead of &quot;OR&quot;, like so:</p>
<pre class="literal-block">
      [   AND   ]
     /     | ... \
B(K,1) B(K,2)... B(K,m)
</pre>
</div>
<div class="section" id="combining-the-boolean-filters">
<h2>Combining the Boolean Filters</h2>
<p>The subqueries for each prefix from &quot;B&quot; parameters are combined with AND,
to make this (which we refer to as &quot;B-filter&quot; below):</p>
<pre class="literal-block">
              [     AND     ]
             /       |  ...  \
            /                 \
      [   OR   ]               [   AND  ]
     /    | ... \             /    | ... \
B(F,1) B(F,2)...B(F,n)   B(K,1) B(K,2)...B(K,m)
</pre>
</div>
<div class="section" id="negated-boolean-terms">
<h2>Negated Boolean Terms</h2>
<p>All the terms from all 'N' parameters are combined together with &quot;OR&quot;, to
make this (which we refer to as &quot;N-filter&quot; below):</p>
<pre class="literal-block">
      [       OR       ]
     / ... |     |  ... \
N(F,1)...N(F,n) N(K,1)...N(K,m)
</pre>
</div>
<div class="section" id="putting-it-all-together">
<h2>Putting it all together</h2>
<p>The P-terms are filtered by the B-filter using &quot;FILTER&quot; and by the N-filter
using &quot;AND_NOT&quot;:</p>
<pre class="literal-block">
                 [ AND_NOT ]
                /           \
               /             \
     [ FILTER ]             N-terms
      /      \
     /        \
P-terms      B-terms
</pre>
<p>The intent here is to allow filtering on arbitrary (and, typically,
orthogonal) characteristics of the document. For instance, by adding
boolean terms &quot;Ttext/html&quot;, &quot;Ttext/plain&quot; and &quot;J/press&quot; you would be
filtering the probabilistic search for only documents that are both in
the &quot;/press&quot; site <em>and</em> which are either of MIME type text/html or
text/plain. (See below for more information about sites.)</p>
<p>If B-terms or N-terms is absent, that part of the query is simply omitted.</p>
<p>If there is no probabilistic query, the boolean filter is promoted to
be the query, and the weighting scheme is set to boolean.  This has
the effect of applying the boolean filter to the whole database.  If
there are only N-terms, then <tt class="docutils literal"><span class="pre">Query::MatchAll</span></tt> is used for the left
side of the &quot;AND_NOT&quot;.</p>
<p>In order to add more boolean prefixes, you will need to alter the
<tt class="docutils literal">index_file()</tt> function in omindex.cc. Currently omindex adds several
useful ones, detailed below.</p>
<p>Probabilistic terms are constructed from the title, body and keywords
of a document. (Not all document types support all three areas of
text.) Title terms are stored with position data starting at 0, body
terms starting 100 beyond title terms, and keyword terms starting 100
beyond body terms. This allows queries using positional data without
causing false matches across the different types of term.</p>
</div>
</div>
<div class="section" id="sites">
<h1>Sites</h1>
<p>Within a database, Omega supports multiple sites. These are recorded
using boolean terms (see 'Term construction', above) to allow
filtering on them.</p>
<p>Sites work by having all documents within them having a common base
URL. For instance, you might have two sites, one for your press area
and one for your product descriptions:</p>
<ul class="simple">
<li>http://example.com/press/index.html</li>
<li>http://example.com/press/bigrelease.html</li>
<li>http://example.com/products/bigproduct.html</li>
<li>http://example.com/products/littleproduct.html</li>
</ul>
<p>You could index all documents within http://example.com/press/ using a
site of '/press', and all within http://example.com/products/ using
'/products'.</p>
<p>Sites are also useful because omindex indexes documents through the
file system, not by fetching from the web server. If you don't have a
URL to file system mapping which puts all documents under one
hierarchy, you'll need to index each separate section as a site.</p>
<p>An obvious example of this is the way that many web servers map URLs
of the form &lt;http://example.com/~&lt;username&gt;/&gt; to a directory within
that user's home directory (such as ~&lt;username&gt;/pub on a Unix
system). In this case, you can index each user's home page separately,
as a site of the form '/~&lt;username&gt;'. You can then use boolean
filters to allow people to search only a specific home page (or a
group of them), or omit such terms to search everyone's pages.</p>
<p>Note that the site specified when you index is used to build the
complete URL that the results page links to. Thus while sites will
typically want to be relative to the hostname part of the URL (e.g.
'/site' rather than 'http://example.com/site'), you can use them
to have a single search across several different hostnames. This will
still work if you actually store each distinct hostname in a different
database.</p>
</div>
<div class="section" id="omindex-operation">
<h1>omindex operation</h1>
<p>omindex is fairly simple to use, for example:</p>
<pre class="literal-block">
omindex --db default --url http://example.com/ /var/www/example.com
</pre>
<p>For a full list of command line options supported, see <tt class="docutils literal">man omindex</tt>
or <tt class="docutils literal">omindex <span class="pre">--help</span></tt>.</p>
<p>You <em>must</em> specify the database to index into (it's created if it doesn't
exist, but parent directories must exist).  You will often also want to specify
the base URL (which is used as the site, and can be relative to the hostname -
starts '/' - or absolute - starts with a scheme, e.g.
'http://example.com/products/').  If not specified, the base URL defaults to
<tt class="docutils literal">/</tt>.</p>
<p>You also need to tell omindex which directory to index. This should be
either a single directory (in which case it is taken to be the
directory base of the entire site being indexed), or as two arguments,
the first being the directory base of the site being indexed, and the
second being a relative directory within that to index.</p>
<p>For instance, in the example above, if you separate your products by
size, you might end up with:</p>
<ul class="simple">
<li>http://example.com/press/index.html</li>
<li>http://example.com/press/bigrelease.html</li>
<li>http://example.com/products/large/bigproduct.html</li>
<li>http://example.com/products/small/littleproduct.html</li>
</ul>
<p>If the entire website is stored in the file system under the directory
/www/example, then you would probably index the site in two
passes, one for the '/press' site and one for the '/products' site. You
might use the following commands:</p>
<pre class="literal-block">
$ omindex -p --db /var/lib/omega/data/default --url /press /www/example/press
$ omindex -p --db /var/lib/omega/data/default --url /products /www/example/products
</pre>
<p>If you add a new large products, but don't want to reindex the whole of
the products section, you could do:</p>
<pre class="literal-block">
$ omindex -p --db /var/lib/omega/data/default --url /products /www/example/products large
</pre>
<p>and just the large products will be reindexed. You need to do it like that, and
not as:</p>
<pre class="literal-block">
$ omindex -p --db /var/lib/omega/data/default --url /products/large /www/example/products/large
</pre>
<p>because that would make the large products part of a new site,
'/products/large', which is unlikely to be what you want, as large
products would no longer come up in a search of the products
site. (Note that the <tt class="docutils literal"><span class="pre">--depth-limit</span></tt> option may come in handy if you have
sites '/products' and '/products/large', or similar.)</p>
<p>omindex has built-in support for indexing HTML, PHP, text files, CSV
(Comma-Separated Values) files, SVG, Atom feeds, and AbiWord documents.  It can
also index a number of other formats using external programs.  Filter programs
are run with CPU, time and memory limits to prevent a runaway filter from
blocking indexing of other files.</p>
<p>The way omindex decides how to index a file is based around MIME content-types.
First of all omindex will look up a file's extension in its extension to MIME
type map.  If there's no entry, it will then ask libmagic to examine the
contents of the file and try to determine a MIME type.</p>
<p>The following formats are supported as standard (you can tell omindex to use
other filters too - see below):</p>
<ul class="simple">
<li>HTML (.html, .htm, .shtml)</li>
<li>PHP (.php) - our HTML parser knows to ignore PHP code</li>
<li>text files (.txt, .text)</li>
<li>SVG (.svg)</li>
<li>CSV (Comma-Separated Values) files (.csv)</li>
<li>PDF (.pdf) if pdftotext is available (comes with poppler or xpdf)</li>
<li>PostScript (.ps, .eps, .ai) if ps2pdf (from ghostscript) and pdftotext (comes
with poppler or xpdf) are available</li>
<li>OpenOffice/StarOffice documents (.sxc, .stc, .sxd, .std, .sxi, .sti, .sxm,
.sxw, .sxg, .stw) if unzip is available</li>
<li>OpenDocument format documents (.odt, .ods, .odp, .odg, .odc, .odf, .odb,
.odi, .odm, .ott, .ots, .otp, .otg, .otc, .otf, .oti, .oth) if unzip is
available</li>
<li>MS Word documents (.dot) if antiword is available (.doc files are left to
libmagic, as they may actually be RTF (AbiWord saves RTF when asked to save
as .doc, and Microsoft Word quietly loads RTF files with a .doc extension),
or plain-text).</li>
<li>MS Excel documents (.xls, .xlb, .xlt, .xlr, .xla) if xls2csv is available
(comes with catdoc)</li>
<li>MS Powerpoint documents (.ppt, .pps) if catppt is available (comes with
catdoc)</li>
<li>MS Office 2007 documents (.docx, .docm, .dotx, .dotm, .xlsx, .xlsm, .xltx,
.xltm, .pptx, .pptm, .potx, .potm, .ppsx, .ppsm) if unzip is available</li>
<li>Wordperfect documents (.wpd) if wpd2text is available (comes with libwpd)</li>
<li>MS Works documents (.wps, .wpt) if wps2text is available (comes with libwps)</li>
<li>MS Outlook message (.msg) if perl with Email::Outlook::Message and
HTML::Parser modules is available</li>
<li>MS Publisher documents (.pub) if pub2xhtml is available (comes with libmspub)</li>
<li>AbiWord documents (.abw)</li>
<li>Compressed AbiWord documents (.zabw)</li>
<li>Rich Text Format documents (.rtf) if unrtf is available</li>
<li>Perl POD documentation (.pl, .pm, .pod) if pod2text is available</li>
<li>reStructured text (.rst, .rest) if rst2html is available (comes with
docutils)</li>
<li>Markdown (.md, .markdown) if markdown is available</li>
<li>TeX DVI files (.dvi) if catdvi is available</li>
<li>DjVu files (.djv, .djvu) if djvutxt is available</li>
<li>XPS files (.xps) if unzip is available</li>
<li>Debian packages (.deb, .udeb) if dpkg-deb is available</li>
<li>RPM packages (.rpm) if rpm is available</li>
<li>Atom feeds (.atom)</li>
<li>MAFF (.maff) if unzip is available</li>
<li>MHTML (.mhtml, .mht) if perl with MIME::Tools is available</li>
<li>MIME email messages (.eml) and USENET articles if perl with MIME::Tools and
HTML::Parser is available</li>
<li>vCard files (.vcf, .vcard) if perl with Text::vCard is available</li>
</ul>
<p>If you have additional extensions that represent one of these types, you can
add an additional MIME mapping using the <tt class="docutils literal"><span class="pre">--mime-type</span></tt> option.  For
instance, if your press releases are PostScript files with extension
<tt class="docutils literal">.posts</tt> you can tell omindex this like so:</p>
<pre class="literal-block">
$ omindex --db /var/lib/omega/data/default --url /press /www/example/press --mime-type posts:application/postscript
</pre>
<p>The syntax of <tt class="docutils literal"><span class="pre">--mime-type</span></tt> is 'ext:type', where ext is the extension of
a file of that type (everything after the last '.').  The <tt class="docutils literal">type</tt> can be any
string, but to be useful there either needs to be a filter set for that type
- either using <tt class="docutils literal"><span class="pre">--filter</span></tt> or by <tt class="docutils literal">type</tt> being understood by default:</p>
<table border="1" class="docutils">
<colgroup>
<col width="84%" />
<col width="16%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">MIME Type</th>
<th class="head">Extensions</th>
</tr>
</thead>
<tbody valign="top">
<tr><td>text/csv</td>
<td>csv</td>
</tr>
<tr><td>text/html</td>
<td>html
htm
shtml</td>
</tr>
<tr><td>text/markdown</td>
<td>markdown
md</td>
</tr>
<tr><td>text/plain</td>
<td>txt
text</td>
</tr>
<tr><td>text/rtf</td>
<td>rtf</td>
</tr>
<tr><td>text/vcard</td>
<td>vcf
vcard</td>
</tr>
<tr><td>text/x-perl</td>
<td>pl
pm
pod</td>
</tr>
<tr><td>text/x-php</td>
<td>php</td>
</tr>
<tr><td>text/x-rst</td>
<td>rst
rest</td>
</tr>
<tr><td>application/atom+xml</td>
<td>atom</td>
</tr>
<tr><td>application/msword</td>
<td>dot</td>
</tr>
<tr><td>application/pdf</td>
<td>pdf</td>
</tr>
<tr><td>application/postscript</td>
<td>ps
eps
ai</td>
</tr>
<tr><td>application/vnd.debian.binary-package</td>
<td>deb
udeb</td>
</tr>
<tr><td>application/vnd.ms-excel</td>
<td>xls
xlb
xlt
xla
xlr</td>
</tr>
<tr><td>application/vnd.ms-outlook</td>
<td>msg
oft</td>
</tr>
<tr><td>application/vnd.ms-powerpoint</td>
<td>ppt
pps</td>
</tr>
<tr><td>application/vnd.ms-works</td>
<td>wps
wpt</td>
</tr>
<tr><td>application/vnd.ms-xpsdocument</td>
<td>xps</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.chart</td>
<td>odc</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.chart-template</td>
<td>otc</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.database</td>
<td>odb</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.formula</td>
<td>odf</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.formula-template</td>
<td>otf</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.graphics</td>
<td>odg</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.graphics-template</td>
<td>otg</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.image</td>
<td>odi</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.image-template</td>
<td>oti</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.presentation</td>
<td>odp</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.presentation-template</td>
<td>otp</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.spreadsheet</td>
<td>ods</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.spreadsheet-template</td>
<td>ots</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.text</td>
<td>odt</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.text-master</td>
<td>odm</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.text-template</td>
<td>ott</td>
</tr>
<tr><td>application/vnd.oasis.opendocument.text-web</td>
<td>oth</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.presentationml.presentation</td>
<td>pptx
pptm</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.presentationml.slideshow</td>
<td>ppsx
ppsm</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.presentationml.template</td>
<td>potx
potm</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.spreadsheetml.sheet</td>
<td>xlsx
xlsm</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.spreadsheetml.template</td>
<td>xltx
xltm</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.wordprocessingml.document</td>
<td>docx
docm</td>
</tr>
<tr><td>application/vnd.openxmlformats-officedocument.wordprocessingml.template</td>
<td>dotx
dotm</td>
</tr>
<tr><td>application/vnd.sun.xml.calc</td>
<td>sxc</td>
</tr>
<tr><td>application/vnd.sun.xml.calc.template</td>
<td>stc</td>
</tr>
<tr><td>application/vnd.sun.xml.draw</td>
<td>sxd</td>
</tr>
<tr><td>application/vnd.sun.xml.draw.template</td>
<td>std</td>
</tr>
<tr><td>application/vnd.sun.xml.impress</td>
<td>sxi</td>
</tr>
<tr><td>application/vnd.sun.xml.impress.template</td>
<td>sti</td>
</tr>
<tr><td>application/vnd.sun.xml.math</td>
<td>sxm</td>
</tr>
<tr><td>application/vnd.sun.xml.writer</td>
<td>sxw</td>
</tr>
<tr><td>application/vnd.sun.xml.writer.global</td>
<td>sxg</td>
</tr>
<tr><td>application/vnd.sun.xml.writer.template</td>
<td>stw</td>
</tr>
<tr><td>application/vnd.wordperfect</td>
<td>wpd</td>
</tr>
<tr><td>application/x-abiword</td>
<td>abw</td>
</tr>
<tr><td>application/x-abiword-compressed</td>
<td>zabw</td>
</tr>
<tr><td>application/x-dvi</td>
<td>dvi</td>
</tr>
<tr><td>application/x-maff</td>
<td>maff</td>
</tr>
<tr><td>application/x-mimearchive</td>
<td>mht
mhtml</td>
</tr>
<tr><td>application/x-mspublisher</td>
<td>pub</td>
</tr>
<tr><td>application/x-redhat-package-manager</td>
<td>rpm</td>
</tr>
<tr><td>image/svg+xml</td>
<td>svg</td>
</tr>
<tr><td>image/vnd.djvu</td>
<td>djv
djvu</td>
</tr>
<tr><td>message/rfc822</td>
<td>eml</td>
</tr>
</tbody>
</table>
<p>You can specify <tt class="docutils literal">*</tt> as the MIME sub-type for <tt class="docutils literal"><span class="pre">--filter</span></tt>, for example if you
have a filter you want to apply to any video files, you could specify it using
<tt class="docutils literal"><span class="pre">--filter</span> <span class="pre">'video/*:index-video-file'</span></tt>.  Note that this is checked right after
checking for the exact MIME type, so will override any built-in filters which
would otherwise match.  Also you can't use arbitrary wildcards, just <tt class="docutils literal">*</tt> for
the entire sub-type.  And be careful to quote <tt class="docutils literal">*</tt> to protect it from the
shell.  Support for this was added in 1.3.3.</p>
<p>If there's no specific filter, and no subtype wildcard, then <tt class="docutils literal">*/*</tt> is checked
(assuming the mimetype contains a <tt class="docutils literal">/</tt>), and after that <tt class="docutils literal">*</tt> (for any
mimetype string).  Combined with filter command <tt class="docutils literal">true</tt> for indexing by
meta-data only, you can specify a fall back case of indexing by meta-data
only using <tt class="docutils literal"><span class="pre">--filter</span> <span class="pre">'*:true'</span></tt>.  Support for this was added in 1.3.4.</p>
<p>There are also two special values that can be specified instead of a MIME
type:</p>
<ul class="simple">
<li>ignore - tells omindex to quietly ignore such files</li>
<li>skip - tells omindex to skip such files</li>
</ul>
<p>By default no extensions are marked as &quot;skip&quot;, and the following extensions are
marked as &quot;ignore&quot;:</p>
<blockquote>
<ul class="simple">
<li>a</li>
<li>adm</li>
<li>bin</li>
<li>com</li>
<li>css</li>
<li>cur</li>
<li>dat</li>
<li>db</li>
<li>dll</li>
<li>dylib</li>
<li>exe</li>
<li>fon</li>
<li>ico</li>
<li>jar</li>
<li>js</li>
<li>lib</li>
<li>lnk</li>
<li>msi</li>
<li>msp</li>
<li>o</li>
<li>obj</li>
<li>pyc</li>
<li>pyd</li>
<li>pyo</li>
<li>so</li>
<li>sqlite</li>
<li>sqlite-journal</li>
<li>sqlite3</li>
<li>tmp</li>
<li>ttf</li>
</ul>
</blockquote>
<p>If you wish to remove a MIME mapping, you can do this by omitting the type -
for example if you have <tt class="docutils literal">.dot</tt> files which are inputs for the graphviz
tool <tt class="docutils literal">dot</tt>, then you may wish to remove the default mapping for <tt class="docutils literal">.dot</tt>
files and let libmagic be used to determine their type, which you can do
using: <tt class="docutils literal"><span class="pre">--mime-type=dot:</span></tt> (if you want to <em>ignore</em> all <tt class="docutils literal">.dot</tt> files,
instead use <tt class="docutils literal"><span class="pre">--mime-type=dot:ignore</span></tt>).</p>
<p>The lookup of extensions in the MIME mappings is case sensitive, but if an
extension isn't found and includes upper case ASCII letters, they're converted
to lower case and the lookup is repeated, so you effectively get case
insensitive lookup for mappings specified with a lower-case extension, but
you can set different handling for differently cased variants if you need
to.</p>
<p>You can add support for additional MIME content types (or override existing
ones) using the <tt class="docutils literal"><span class="pre">--filter</span></tt> option to specify a command to run.  At present,
this command needs to produce output in either HTML or plain text format
(as of 1.3.3, you can specify the character encoding that the output will be
in; in earlier versions, plain text output had to be UTF-8).</p>
<p>As of 1.3.3, the command can include certain placeholders which are substituted
by omindex:</p>
<ul>
<li><p class="first">Any <tt class="docutils literal">%f</tt> in this command will be replaced with the filename of the file to
extract (suitably escaped to protect it from the shell, so don't put quotes
around <tt class="docutils literal">%f</tt>).</p>
<p>If you don't include <tt class="docutils literal">%f</tt> in the command, then the filename of the file to
be extracted will be appended to the command, separated by a space.</p>
</li>
<li><p class="first">Any <tt class="docutils literal">%t</tt> in this command will be replaced with a filename in a temporary
directory (suitably escaped to protect it from the shell, so don't put
quotes around <tt class="docutils literal">%t</tt>).  The extension of this filename will reflect the
expected output format (either <tt class="docutils literal">.html</tt> or <tt class="docutils literal">.txt</tt>).  If you don't use
<tt class="docutils literal">%t</tt> in the command, then omindex will expect output on <tt class="docutils literal">stdout</tt> (prior
to 1.3.3, output had to be on <tt class="docutils literal">stdout</tt>).</p>
</li>
<li><p class="first"><tt class="docutils literal">%%</tt> can be used should you need a literal <tt class="docutils literal">%</tt> in the command.</p>
</li>
</ul>
<p>For example, if you'd prefer to use Abiword to extract text from word documents
(by default, omindex uses antiword), then you can pass the option
<tt class="docutils literal"><span class="pre">--filter=application/msword:'abiword</span> <span class="pre">--to=txt</span> <span class="pre">--to-name=fd://1'</span></tt> to
omindex.</p>
<p>Another example - if you wanted to handle files of MIME type
<tt class="docutils literal"><span class="pre">application/octet-stream</span></tt> by running them through <tt class="docutils literal">strings <span class="pre">-n8</span></tt>, you can
pass the option <tt class="docutils literal"><span class="pre">--filter=application/octet-stream:'strings</span> <span class="pre">-n8'</span></tt>.</p>
<p>A more complex example: to process <tt class="docutils literal">.foo</tt> files with the (fictional)
<tt class="docutils literal">foo2utf16</tt> utility which produces UTF-16 text but doesn't support writing
output to stdout, run omindex with <tt class="docutils literal"><span class="pre">-Mfoo:text/x-foo</span>
<span class="pre">-Ftext/x-foo,,utf-16:'foo2utf16</span> %f %t'</tt>.</p>
<p>A less contrived example of the use of <tt class="docutils literal"><span class="pre">--filter</span></tt> makes use of LibreOffice,
via the unoconv script, to extract text from various formats.  First you
need to start a listening instance (if you don't, unoconv will start up
LibreOffice for every file, which is rather inefficient) - the <tt class="docutils literal">&amp;</tt> tells
the shell to run it in the background:</p>
<pre class="literal-block">
unoconv --listener &amp;
</pre>
<p>Then run omindex with options such as
<tt class="docutils literal"><span class="pre">--filter=application/msword,html:'unoconv</span> <span class="pre">--stdout</span> <span class="pre">-f</span> html'</tt> (you'll want
to repeat this for each format which you want to use LibreOffice on).</p>
<p>If you specify <tt class="docutils literal">false</tt> as the command in <tt class="docutils literal"><span class="pre">--filter</span></tt>, omindex will skip
files with the specified MIME type.  (As of 1.2.20 and 1.3.3 <tt class="docutils literal">false</tt> is
explicitly checked for; in earlier versions this will also work, at least
on Unix where <tt class="docutils literal">false</tt> is a command which ignores its arguments and exits with
a non-zero status).</p>
<p>If you specify <tt class="docutils literal">true</tt> as the command in <tt class="docutils literal"><span class="pre">--filter</span></tt>, omindex won't try
to extract text from the file, but will index it such that it can be searched
for via metadata which comes from the filing system (filename, extension, mime
content-type, last modified time, size).  (As of 1.2.22 and 1.3.4 <tt class="docutils literal">true</tt> is
explicitly checked for; in earlier versions this will also work, at least
on Unix where <tt class="docutils literal">true</tt> is a command which ignores its arguments and exits with
a status zero).</p>
<p>If you know of a reliable filter which can extract text from a file format
which might be of interest to others, please let us know so we can consider
including it as a standard filter.</p>
<p>The <tt class="docutils literal"><span class="pre">--duplicates</span></tt> option controls how omindex handles documents which map
to a URL which is already in the database.  The default (which can be
explicitly set with <tt class="docutils literal"><span class="pre">--duplicates=replace</span></tt>) is to reindex if the last
modified time of the file is newer than that recorded in the database.
The alternative is <tt class="docutils literal"><span class="pre">--duplicates=ignore</span></tt>, which will never reindex an
existing document.  If you only add documents, this avoids the overhead
of checking the last modified time.  It also allows you to prioritise
adding completely new documents to the database over updating existing ones.</p>
<p>By default, omindex will remove any document in the database which has a URL
that doesn't correspond to a file seen on disk - in other words, it will clear
out everything that doesn't exist any more.  However if you are building up
an omega database with several runs of omindex, this is not
appropriate (as each run would delete the data from the previous run),
so you should use the <tt class="docutils literal"><span class="pre">--no-delete</span></tt> option.  Note that if you
choose to work like this, it is impossible to prune old documents from
the database using omindex. If this is a problem for you, an
alternative is to index each subsite into a different database, and
merge all the databases together when searching.</p>
<p><tt class="docutils literal"><span class="pre">--depth-limit</span></tt> allows you to prevent omindex from descending more than
a certain number of directories.  Specifying <tt class="docutils literal"><span class="pre">--depth-limit=0</span></tt> means no limit
is imposed on recursion; <tt class="docutils literal"><span class="pre">--depth-limit=1</span></tt> means don't descend into any
subdirectories of the start directory.</p>
<div class="section" id="tracking-files-which-couldn-t-be-indexed">
<h2>Tracking files which couldn't be indexed</h2>
<p>In older versions, omindex only tracked files which it successfully indexed -
if a file couldn't be read, or a filter program failed on it, or it was marked
not to be indexed (e.g. with an HTML meta tag) then it would be retried on
subsequent runs.  Starting from version 1.3.4, omindex now tracks failed
files in the user metadata of the database, along with their sizes and last
modified times, and uses this data to skip files which previously failed and
haven't changed since.</p>
<p>You can force omindex to retry such files using the <tt class="docutils literal"><span class="pre">--retry-failed</span></tt> option.
One situation in which this is useful is if you've upgraded a filter program
to a newer version which you suspect will index some files which previously
failed.</p>
<p>Currently there's no mechanism for automatically removing failure entries
when the file they refer to is removed or renamed.  These lingering entries are
harmless, except they bloat the database a little.  A simple way to clear them
out is to run periodically with <tt class="docutils literal"><span class="pre">--retry-failed</span></tt> as this removes any existing
failure entries before indexing starts.</p>
</div>
</div>
<div class="section" id="html-parsing">
<h1>HTML Parsing</h1>
<p>The document <tt class="docutils literal">&lt;title&gt;</tt> tag is used as the document title.  Metadata in various
<tt class="docutils literal">&lt;meta&gt;</tt> tags is also understood - these values of the <tt class="docutils literal">name</tt> parameter are
currently handled when found:</p>
<blockquote>
<ul class="simple">
<li><tt class="docutils literal">author</tt>, <tt class="docutils literal">dcterms.creator</tt>, <tt class="docutils literal">dcterms.contributor</tt>: author(s)</li>
<li><tt class="docutils literal">created</tt>, <tt class="docutils literal">dcterms.issued</tt>: document creation date</li>
<li><tt class="docutils literal">classification</tt>: document topic</li>
<li><tt class="docutils literal">keywords</tt>, <tt class="docutils literal">dcterms.subject</tt>, <tt class="docutils literal">dcterms.description</tt>: indexed as extra
document text (but not stored in the sample)</li>
<li><tt class="docutils literal">description</tt>: by default, handled as <tt class="docutils literal">keywords</tt>.  If <tt class="docutils literal">omindex</tt> is run
with <tt class="docutils literal"><span class="pre">--sample=description</span></tt>, then this is used as the preferred source for
the stored sample of document text (HTML documents with no <tt class="docutils literal">description</tt>
fall back to a sample from the body; if <tt class="docutils literal">description</tt> occurs multiple
times then second and subsequent are handled as <tt class="docutils literal">keywords</tt>).  In Omega
1.4.2 and earlier, <tt class="docutils literal"><span class="pre">--sample</span></tt> wasn't supported and the behaviour was as
if <tt class="docutils literal"><span class="pre">--sample=description</span></tt> had been specified.</li>
</ul>
</blockquote>
<p>The HTML parser will look for the 'robots' META tag, and won't index pages
which are marked as <tt class="docutils literal">noindex</tt> or <tt class="docutils literal">none</tt>, for example any of the following:</p>
<pre class="literal-block">
&lt;meta name=&quot;robots&quot; content=&quot;noindex,nofollow&quot;&gt;
&lt;meta name=&quot;robots&quot; content=&quot;noindex&quot;&gt;
&lt;meta name=&quot;robots&quot; content=&quot;none&quot;&gt;
</pre>
<p>The <tt class="docutils literal">omindex</tt> option <tt class="docutils literal"><span class="pre">--ignore-exclusions</span></tt> disables this behaviour, so
the files with the above will be indexed anyway.</p>
<p>Sometimes it is useful to be able to exclude just part of a page from being
indexed (for example you may not want to index navigation links, or a footer
which appears on every page).  To allow this, the parser supports &quot;magic&quot;
comments to mark sections of the document to not index.  Two formats are
supported - htdig_noindex (used by ht://Dig) and UdmComment (used by
mnoGoSearch):</p>
<pre class="literal-block">
Index this bit &lt;!--htdig_noindex--&gt;but &lt;b&gt;not&lt;/b&gt; this&lt;!--/htdig_noindex--&gt;
</pre>
<pre class="literal-block">
&lt;!--UdmComment--&gt;&lt;div&gt;Boring copyright notice&lt;/div&gt;&lt;!--/UdmComment--&gt;
</pre>
</div>
<div class="section" id="boolean-terms">
<h1>Boolean terms</h1>
<p>omindex will create the following boolean terms when it indexes a
document:</p>
<dl class="docutils">
<dt>E</dt>
<dd>Extension of the file (e.g. <cite>Epdf</cite>) [since Omega 1.2.5]</dd>
<dt>T</dt>
<dd>MIME type</dd>
<dt>J</dt>
<dd>The base URL, omitting any trailing slash (so if the base URL was just
<cite>/</cite>, the term is just <cite>J</cite>).  If the resulting term would be &gt; 240
bytes, it's hashed in the same way an <cite>U</cite> prefix terms are.  Mnemonic: the
Jumping-off point. [since Omega 1.3.4]</dd>
<dt>H</dt>
<dd>hostname of site (if supplied - this term won't exist if you index a
site with base URL '/press', for instance).  Since Omega 1.3.4, if the
resulting term would be &gt; 240 bytes, it's hashed in the same way as <cite>U</cite>
prefix terms are.</dd>
<dt>P</dt>
<dd>path terms - one term for the directory which the document is in, and for
each parent directories, with no trailing slashes [since Omega 1.3.4 -
in earlier versions, there was just one <cite>P</cite> term for the path of site (i.e.
the rest of the site base URL) - this will be amongst the terms Omega 1.3.4
adds].  Since Omega 1.3.4, if the resulting term would be &gt; 240 bytes, it's
hashed in the same way as <cite>U</cite> prefix terms are.</dd>
<dt>U</dt>
<dd>full URL of indexed document - if the resulting term would be &gt; 240 bytes,
a hashing scheme is used to avoid overflowing Xapian's term length limit.</dd>
<dt>D</dt>
<dd><p class="first">date (numeric format: YYYYMMDD)</p>
<p class="last">date can also have the magical form &quot;latest&quot; - a document indexed
by the term Dlatest matches any date-range without an end date.
You can index dynamic documents which are always up to date
with Dlatest and they'll match as expected.  (If you use sort by date,
you'll probably also want to set the value containing the timestamp to
a &quot;max&quot; value so dynamic documents match a date in the far future).</p>
</dd>
<dt>M</dt>
<dd>month (numeric format: YYYYMM)</dd>
<dt>Y</dt>
<dd>year (four digits)</dd>
</dl>
</div>
<div class="section" id="omega-configuration">
<h1>omega configuration</h1>
<p>Most of the omega CGI configuration is dynamic, by setting CGI
parameters. However some things must be configured using a
configuration file.  The configuration file is searched for in
various locations:</p>
<ul class="simple">
<li>Firstly, if the &quot;OMEGA_CONFIG_FILE&quot; environment variable is
set, its value is used as the full path to a configuration file
to read.</li>
<li>Next (if the environment variable is not set, or the file pointed
to is not present), the file &quot;omega.conf&quot; in the same directory as
the Omega CGI is used.</li>
<li>Next (if neither of the previous steps found a file), the file
&quot;${sysconfdir}/omega.conf&quot; (e.g. /etc/omega.conf on Linux systems)
is used.</li>
<li>Finally, if no configuration file is found, default values are used.</li>
</ul>
<p>The format of the file is very simple: a line per option, with the
option name followed by its value, separated by a whitespace.  Blank
lines are ignored.  If the first non-whitespace character on a line
is a '#', omega treats the line as a comment and ignores it.</p>
<p>The current options are:</p>
<ul class="simple">
<li><cite>database_dir</cite>: the directory containing all the Omega databases</li>
<li><cite>template_dir</cite>: the directory containing the OmegaScript templates</li>
<li><cite>log_dir</cite>: the directory which the OmegaScript <cite>$log</cite> command writes log
files to</li>
<li><cite>cdb_dir</cite>: the directory which the OmegaScript <cite>$lookup</cite> command
looks for CDB files in</li>
</ul>
<p>The default values (used if no configuration file is found) are:</p>
<pre class="literal-block">
database_dir /var/lib/omega/data
template_dir /var/lib/omega/templates
log_dir /var/log/omega
cdb_dir /var/lib/omega/cdb
</pre>
<p>Note that, with apache, environment variables may be set using mod_env, and
with apache 1.3.7 or later this may be used inside a .htaccess file.  This
makes it reasonably easy to share a single system installed copy of Omega
between multiple users.</p>
</div>
<div class="section" id="supplied-templates">
<h1>Supplied Templates</h1>
<p>The OmegaScript templates supplied with Omega are:</p>
<ul class="simple">
<li>query - This is the default template, providing a typical Web search
interface.</li>
<li>topterms - This is just like query, but provides a &quot;top terms&quot; feature
which suggests terms the user might want to add to their query to
obtain better results.</li>
<li>godmode - Allows you to inspect a database showing which terms index
each document, and which documents are indexed by each term.</li>
<li>opensearch - Provides results in OpenSearch format (for more details
see <a class="reference external" href="http://www.opensearch.org/">http://www.opensearch.org/</a>).</li>
<li>xml - Provides results in a custom XML format.</li>
<li>emptydocs - Shows a list of documents with zero length.  If CGI parameter
TERM is set to a non-empty value, then only documents indexed by that given
term are shown (e.g. TERM=Tapplication/pdf to show PDF files with no text);
otherwise all zero length documents are shown.</li>
</ul>
<p>There are also &quot;helper fragments&quot; used by the templates above:</p>
<ul class="simple">
<li>inc/anyalldropbox - Provides a choice of matching &quot;any&quot; or &quot;all&quot; terms
by default as a drop down box.</li>
<li>inc/anyallradio - Provides a choice of matching &quot;any&quot; or &quot;all&quot; terms
by default as radio buttons.</li>
<li>toptermsjs - Provides some JavaScript used by the topterms template.</li>
</ul>
</div>
<div class="section" id="document-data-construction">
<h1>Document data construction</h1>
<p>This is only useful if you need to inject your own documents into the
database independently of omindex, such as if you are indexing
dynamically-generated documents that are served using a server-side
system such as PHP or ASP, but which you can determine the contents of
in some way, such as documents generated from reasonably static
database contents.</p>
<p>The document data field stores some summary information about the
document, in the following (sample) format:</p>
<pre class="literal-block">
url=&lt;baseurl&gt;
sample=&lt;sample&gt;
caption=&lt;title&gt;
type=&lt;mimetype&gt;
</pre>
<p>Further fields may be added (although omindex doesn't currently add any
others), and may be looked up from OmegaScript using the $field{}
command.</p>
<p>As of Omega 0.9.3, you can alternatively add something like this near the
start of your OmegaScript template:</p>
<pre class="literal-block">
$set{fieldnames,$split{caption sample url}}
</pre>
<p>Then you need only give the field values in the document data, which can
save a lot of space in a large database.  With the setting of fieldnames
above, the first line of document data can be accessed with $field{caption},
the second with $field{sample}, and the third with $field{url}.</p>
</div>
<div class="section" id="stopword-list">
<h1>Stopword List</h1>
<p>At search time, Omega uses a built-in list of stopwords, which are:</p>
<pre class="literal-block">
a about an and are as at be by en for from how i in is it of on or that the
this to was what when where which who why will with you your
</pre>
</div>
</div>
</body>
</html>