File: msearch-indexer-configuration.html

package info (click to toggle)
mnogosearch 3.3.7-3
links: PTS, VCS
area: main
in suites: lenny
size: 17,484 kB
ctags: 4,565
sloc: ansic: 94,097; xml: 16,864; sh: 8,915; makefile: 1,727; perl: 801; php: 561; sql: 15
file content (1843 lines) | stat: -rw-r--r-- 41,193 bytes
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML
><HEAD
><TITLE
>indexer configuration</TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
REL="HOME"
TITLE="mnoGoSearch 3.3.7 reference manual"
HREF="index.html"><LINK
REL="UP"
TITLE="Indexing"
HREF="msearch-indexing.html"><LINK
REL="PREVIOUS"
TITLE="Content-Encoding support
		
	"
HREF="msearch-content-enc.html"><LINK
REL="NEXT"
TITLE="Extended indexing features"
HREF="msearch-extended-indexing.html"><LINK
REL="STYLESHEET"
TYPE="text/css"
HREF="mnogo.css"><META
NAME="Description"
CONTENT="mnoGoSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META
NAME="Keywords"
CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, mnoGoSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD
><BODY
CLASS="sect1"
BGCOLOR="#EEEEEE"
TEXT="#000000"
LINK="#000080"
VLINK="#800080"
ALINK="#FF0000"
><!--#include virtual="body-before.html"--><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
>mnoGoSearch 3.3.7 reference manual: Full-featured search engine software</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="msearch-content-enc.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
>Chapter 3. Indexing</TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="msearch-extended-indexing.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><DIV
CLASS="sect1"
><H1
CLASS="sect1"
><A
NAME="indexer-configuration"
>indexer configuration</A
></H1
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow"
>Specifying WEB space to be indexed</A
></H2
><P
>When indexer tries to insert a new URL into the database or is trying to index an existing one, it first of all checks whether this URL has corresponding <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
> or <B
CLASS="command"
>Subnet</B
> command given in <TT
CLASS="filename"
>indexer.conf</TT
>. URLs without a corresponding <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
> or <B
CLASS="command"
>Subnet</B
> command are not indexed. By default those URLs which are already in the database and have no Server/Realm/Subnet commands will be deleted from the database. It may happen, for example, after removing some Server/Realm/Subnet commands from <TT
CLASS="filename"
>indexer.conf</TT
>.</P
><P
>These commands have the following format: 
<PRE
CLASS="programlisting"
>&#13;Server [Method] [SubSection] &#60;pattern&#62; [alias]
Realm [Method] [CaseType] [MatchType] [CmpType] &#60;pattern&#62; [alias]
Subnet [Method] [MatchType] &#60;pattern&#62;
</PRE
>
	</P
><P
>Mandatory parameter <CODE
CLASS="option"
>pattern</CODE
> specifies an URL, or its part, or pattern to compare.</P
><P
>Optional parameter <CODE
CLASS="option"
>method</CODE
> specifies a document action for this command. It may take any of these values:
		<TT
CLASS="literal"
>Allow</TT
>, <TT
CLASS="literal"
>Disallow</TT
>, <TT
CLASS="literal"
>HrefOnly</TT
>, 
		<TT
CLASS="literal"
>CheckOnly</TT
>, <TT
CLASS="literal"
>Skip</TT
>, 	<TT
CLASS="literal"
>CheckMP3</TT
>,
		<TT
CLASS="literal"
>CheckMP3Only</TT
>. By default, the value <TT
CLASS="literal"
>Allow</TT
> is used.
		<P
></P
><OL
TYPE="1"
><LI
><P
>&#13;					<B
CLASS="command"
>Allow</B
>
					<A
NAME="AEN1046"
></A
>
				</P
><P
>&#13;					Value <TT
CLASS="literal"
>Allow</TT
> specifies that all corresponding documents will be indexed and scanned for new links. Depending on <TT
CLASS="literal"
>Content-Type</TT
>, the appropriate external parser is executed if needed.
				</P
></LI
><LI
><P
>&#13;					<B
CLASS="command"
>Disallow</B
>
					<A
NAME="AEN1056"
></A
>
				</P
><P
>&#13;					Value <TT
CLASS="literal"
>Disallow</TT
> specifies that all corresponding documents will be ignored and deleted from the database, if its was inserted there in the first place.
				</P
></LI
><LI
><P
>&#13;					<B
CLASS="command"
>HrefOnly</B
>
					<A
NAME="AEN1065"
></A
>
				</P
><P
>&#13;					Value <TT
CLASS="literal"
>HrefOnly</TT
> specifies that all corresponding documents will only be scanned for new links (not indexed). This is useful, for example, for mail archives indexing, when index pages is only scanned to detect new messages for indexing.
<PRE
CLASS="programlisting"
>&#13;Server HrefOnly Page http://www.mail-archive.com/general%40mnogosearch.org/
Server Allow    Path http://www.mail-archive.com/general%40mnogosearch.org/
</PRE
>
				</P
></LI
><LI
><P
>&#13;					<B
CLASS="command"
>CheckOnly</B
>
					<A
NAME="AEN1075"
></A
>
				</P
><P
>&#13;					Value <TT
CLASS="literal"
>CheckOnly</TT
> specifies that all corresponding documents will be requested by the HTTP HEAD request, not HTTP GET, i.e. only brief info about documents (size, last modified, content type) will be fetched. This allows, for example, to check links on your site:
<PRE
CLASS="programlisting"
>&#13;Server HrefOnly  http://www.mnogosearch.org/
Realm  CheckOnly *
</PRE
>
				</P
><P
>These commands instruct <B
CLASS="command"
>indexer</B
> to scan all documents on <TT
CLASS="literal"
>www.mnogosearch.org</TT
> site and collect all links. Brief info about every document outside <TT
CLASS="literal"
>www.mnogosearch.org</TT
> will be requested by the HEAD method. After indexing is done, <B
CLASS="command"
>indexer -S</B
> command will show status for all documents from this site.</P
></LI
><LI
><P
>&#13;					<B
CLASS="command"
>Skip</B
>
					<A
NAME="AEN1090"
></A
>
				</P
><P
>&#13;					Value <TT
CLASS="literal"
>Skip</TT
> specifies that all corresponding documents will be skipped while indexing. This is useful when you need to disable temporarily the reindexing of several sites, but to still be able to search them. These documents will be marked as expired.
				</P
></LI
><LI
><P
>&#13;					<B
CLASS="command"
>CheckMP3</B
>
					<A
NAME="AEN1099"
></A
>
				</P
><P
>&#13;					Value <TT
CLASS="literal"
>CheckMP3</TT
> specifies that the corresponding documents will be checked for MP3 tags along if its <TT
CLASS="literal"
>Content-Type</TT
> is equal to <TT
CLASS="literal"
>audio/mpeg</TT
>. This is useful, for example, if the remote server supplies <TT
CLASS="literal"
>application/octet-stream</TT
> as <TT
CLASS="literal"
>Content-Type</TT
> for MP3 files. If this tag is present, these files will be indexed as MP3 files, otherwise they will be processed according to <TT
CLASS="literal"
>Content-Type</TT
>.
				</P
></LI
><LI
><P
>&#13;					<B
CLASS="command"
>CheckMP3Only</B
>
					<A
NAME="AEN1113"
></A
>
				</P
><P
>&#13;					This value is equal to <TT
CLASS="literal"
>CheckMP3</TT
>, but if MP3 tag is not present, processing on <TT
CLASS="literal"
>Content-Type</TT
> will not be taken.
				</P
></LI
></OL
>
	</P
><P
>&#13;  		Use the optional <CODE
CLASS="option"
>SubSection</CODE
> parameter to specify the server's checking behavior. The values for <CODE
CLASS="option"
>SubSection</CODE
> are the same than the "Follow" command arguments. The <CODE
CLASS="option"
>SubSection</CODE
>'s value must be one of the following: page, path, site, world and has "path" value by default. If <CODE
CLASS="option"
>SubSection</CODE
> is not specified, current "Follow" value will be used. So, the only <TT
CLASS="literal"
>Server site http://localhost/</TT
> command and combination of <TT
CLASS="literal"
>Follow site</TT
> and <TT
CLASS="literal"
>Server http://localhost/</TT
> have the same effect.
		<P
></P
><OL
TYPE="1"
><LI
><P
><TT
CLASS="literal"
>path</TT
> subsection</P
><P
>When indexer seeks for a "Server" command corresponding to an URL, it checks that the discovered URL starts with URL given in the Server command's argument but without trailing file name. For example, if <TT
CLASS="literal"
>Server path http://localhost/path/to/index.html</TT
> is given, all URLs which have <TT
CLASS="literal"
>http://localhost/path/to/</TT
> at the beginning correspond to this Server command.</P
><P
>The following commands have the same effect except that they insert different URLs into the database:</P
><P
>&#13;					<PRE
CLASS="programlisting"
>&#13;Server path http://localhost/path/to/index.html
Server path http://localhost/path/to/index
Server path http://localhost/path/to/index.cgi?q=bla
Server path http://localhost/path/to/index?q=bla
</PRE
>
				</P
></LI
><LI
><P
> <TT
CLASS="literal"
>site</TT
> subsection</P
><P
>indexer checks that the discovered URL have the same hostname than the URL given in Server command. For example, <TT
CLASS="literal"
>Server site http://localhost/path/to/a.html</TT
> will allow the whole <TT
CLASS="literal"
>http://localhost/</TT
> server to be indexed. </P
></LI
><LI
><P
> <TT
CLASS="literal"
>world</TT
> subsection</P
><P
>If world subsection is specified in Server command, it has the same effect that URL is considered to
match this Server command. See explanation below.</P
></LI
><LI
><P
> <TT
CLASS="literal"
>page</TT
> subsection</P
><P
>This subsection describes the only one URL given in Server argument.</P
></LI
><LI
><P
>subsection in <TT
CLASS="literal"
>news://</TT
> schema</P
><P
>Subsection is always considered as "site" for news:// URL schema. This is because news:// schema has no nested paths like ftp:// or http:// . Use  <TT
CLASS="literal"
>Server news://news.server.com/</TT
> to index the whole news server or, for example, <TT
CLASS="literal"
>Server news://news.server.com/udm</TT
> to index all messages from "udm" hierarchy.</P
></LI
></OL
>
	</P
><P
> The optional parameter <TT
CLASS="literal"
>CaseType</TT
> specifies the case sensitivity for string comparison, it can take one of the following values:  <TT
CLASS="literal"
>case</TT
> - case insensitive comparison, or <TT
CLASS="literal"
>nocase</TT
> - case sensitive comparison. </P
><P
> The optional parameter <TT
CLASS="literal"
>CmpType</TT
> specifies the comparison type and can take two values: <TT
CLASS="literal"
>Regex</TT
> and <TT
CLASS="literal"
>String</TT
>. <TT
CLASS="literal"
>String</TT
> wildcards are the default match type. You can use ? and * signs in URLMask parameters, they mean "one character" and "any number of characters" respectively. For example, if you want to index all HTTP sites in the .ru domain, use this command: 
<PRE
CLASS="programlisting"
>Realm http://*.ru/*</PRE
>
	</P
><P
>Regex comparison type takes a regular expression as argument. Activate regex comparison type using <CODE
CLASS="option"
>Regex</CODE
> keyword. For example, you can describe everything in the .ru domain using the regex comparison type: <PRE
CLASS="programlisting"
>Realm Regex ^http://.*\.ru/</PRE
>
	</P
><P
>Optional parameter <TT
CLASS="literal"
>MatchType</TT
> means match type. The possible values are <TT
CLASS="literal"
>Match</TT
> and <TT
CLASS="literal"
>NoMatch</TT
> with <TT
CLASS="literal"
>Match</TT
> as default. <TT
CLASS="literal"
>Realm NoMatch</TT
> has reverse effect. It means the URL that does not match a given <CODE
CLASS="option"
>pattern</CODE
> will correspond to this <B
CLASS="command"
>Realm</B
> command. For example, use this command to index everything without .com domain:<PRE
CLASS="programlisting"
>Realm NoMatch http://*.com/*</PRE
>
	</P
><P
>Optional <CODE
CLASS="option"
>alias</CODE
> argument provides very complicated URL rewrite, more powerful than other aliasing mechanism. Take a look <A
HREF="msearch-indexer-configuration.html#aliases"
>the Section called <I
>Aliases</I
></A
> at the <CODE
CLASS="option"
>alias</CODE
> argument usage explanation. <CODE
CLASS="option"
>Alias</CODE
> works only with the <CODE
CLASS="option"
>Regex</CODE
> comparison type and has no effect with the <CODE
CLASS="option"
>String</CODE
> type.</P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-server"
>Server command
			<A
NAME="AEN1189"
></A
></A
></H3
><P
>This is the main command of the <TT
CLASS="filename"
>indexer.conf</TT
> file. It is used to add servers or their parts to be indexed. 

This command also says indexer to insert given URL into database at startup.</P
><P
>E.g. command <TT
CLASS="literal"
>Server http://localhost/</TT
>  allows indexing the whole <TT
CLASS="literal"
>http://localhost/</TT
> server. You can also specify some path to index the server subsection: <TT
CLASS="literal"
>Server http://localhost/subsection/</TT
>.</P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>You can suppress indexer behavior to add an URL given in Server command by using the -q indexer command-line argument. It is useful when you have hundreds or thousands Server commands and their URLs are already in database. This gives a faster indexer startup.</P
></BLOCKQUOTE
></DIV
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-realm"
>Realm command
<A
NAME="AEN1202"
></A
></A
></H3
><P
>Realm command is a more powerful mean of describing a web area to be indexed.

It works almost like <B
CLASS="command"
>Server</B
> command but takes a regular expression or string wildcards as <CODE
CLASS="option"
>pattern</CODE
> parameter and do not insert any URL into the database for indexing.
		</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-subnet"
>Subnet command 
			<A
NAME="AEN1210"
></A
></A
></H3
><P
>Subnet command is another way to describe a web area to be indexed.

			It works almost like <B
CLASS="command"
>Server</B
> command but takes string wildcards as <CODE
CLASS="option"
>pattern</CODE
> arguments, which are compared against IP addresses instead of URLs. Argument may have * and ? signs, they mean "one character" and "any number of characters" respectively. For example, if you want to index all HTTP sites in your local subnet, use this command:<PRE
CLASS="programlisting"
>Subnet 192.168.*.*</PRE
></P
><P
>You may use the "NoMatch" optional argument. For example, if you want to index everything without <TT
CLASS="literal"
>195.x.x.x</TT
> subnet, use:<PRE
CLASS="programlisting"
>Subnet NoMatch 195.*.*.*</PRE
>
		</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-difparam"
>Using different parameter for server and it's subsections</A
></H3
><P
>Indexer seeks for "Server" and "Realm" commands in order of their appearance. Thus, if you want to give different parameters to, e.g. whole server and its subsection, you should add the subsection line before the whole server's. Imagine that you have a server's subdirectory which contains news articles. Surely those articles are to be reindexed more often than the rest of the server. The following combination may be useful in such cases:</P
><P
>&#13;			<PRE
CLASS="programlisting"
>&#13;# Add subsection
Period 200000
Server http://servername/news/

# Add server
Period 600000
Server http://servername/
</PRE
>
		</P
><P
>These commands give different reindexing periods for <TT
CLASS="filename"
>/news/</TT
> subdirectory comparing with the period of server as a whole. indexer will choose the first "Server" record for the <TT
CLASS="filename"
>http://servername/news/page1.html</TT
> as far as it matches and was given first.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-default"
>Default indexer behavior</A
></H3
><P
>The default behavior of indexer is to follow through links having correspondent Server/Realm command in the <TT
CLASS="filename"
>indexer.conf</TT
> file. It also jumps between servers if both of them are present in <TT
CLASS="filename"
>indexer.conf</TT
> either directly in Server command or indirectly in Realm command. For example, there are two Server commands:</P
><P
>&#13;			<PRE
CLASS="programlisting"
>&#13;Server http://www/
Server http://web/
</PRE
>
		</P
><P
>When indexing <TT
CLASS="filename"
>http://www/page1.html</TT
> indexer WILL follow the link <TT
CLASS="filename"
>http://web/page2.html</TT
> if the last one has been found. Note that these pages are on different servers, but BOTH of them have the correspondent Server record.</P
><P
>If one of the Server command is deleted, indexer will remove all expired URLs from this server during the next reindexing.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-f"
>Using <KBD
CLASS="userinput"
>indexer -f &#60;filename&#62;</KBD
></A
></H3
><P
>The third scheme is very useful when running <TT
CLASS="literal"
>indexer -i -f url.txt</TT
>. You may maintain required servers in the <TT
CLASS="filename"
>url.txt</TT
>. When a new URL is added into <TT
CLASS="filename"
>url.txt</TT
> indexer will index the server of this URL during next startup.</P
></DIV
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="aliases"
>Aliases</A
></H2
><P
>mnoGoSearch has an alias support, making it possible to index sites taking information from another location. For example, if you index your local web server, it is possible to take pages directly from disk without involving your web server in the indexing process. Another example is the building of a search engine for the primary site and using its mirror while indexing. There are several ways of using aliases.</P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-conf"
>Alias indexer.conf command
			<A
NAME="AEN1251"
></A
></A
></H3
><P
>Format of "Alias" <TT
CLASS="filename"
>indexer.conf</TT
> command:
		<PRE
CLASS="programlisting"
>&#13;Alias &#60;masterURL&#62; &#60;mirrorURL&#62;
</PRE
>
		</P
><P
>E.g. you wish to index <TT
CLASS="literal"
>http://search.mnogo.ru/</TT
> using the nearest German mirror <TT
CLASS="literal"
>http://www.gstammw.de/mirrors/mnoGoSearch/</TT
>. Add these lines in your indexer.conf:

		<PRE
CLASS="programlisting"
>&#13;Server http://search.mnogo.ru/
Alias  http://search.mnogo.ru/  http://www.gstammw.de/mirrors/mnoGoSearch/
</PRE
>
		</P
><P
>&#13;			<TT
CLASS="filename"
>search.cgi</TT
> will display URLs from the master site <TT
CLASS="literal"
>http://search.mnogo.ru/</TT
> but indexer will take corresponding page from mirror site <TT
CLASS="literal"
>http://www.gstammw.de/mirrors/mnoGoSearch/</TT
>.</P
><P
>Another example: If you want to index everything in udm.net domain and one of their servers, for example <TT
CLASS="literal"
>http://home.udm.net/</TT
>, is stored on local machine in <TT
CLASS="literal"
>/home/httpd/htdocs/</TT
> directory. These commands will be useful:
		<PRE
CLASS="programlisting"
>&#13;Realm http://*.udm.net/
Alias http://home.udm.net/ file:/home/httpd/htdocs/
		</PRE
>
		</P
><P
>Indexer will take <TT
CLASS="literal"
>home.udm.net</TT
> from local disk and index other sites using HTTP.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="aliases-diff"
>Different aliases for server parts</A
></H3
><P
>Aliases are searched in the order of their appearance in <TT
CLASS="filename"
>indexer.conf</TT
>. So, you can create different aliases for a server and its parts:

		<PRE
CLASS="programlisting"
>&#13;# First, create alias for example for /stat/ directory which
# is not under common location:
Alias http://home.udm.net/stat/  file:/usr/local/stat/htdocs/

# Then create alias for the rest of the server:
Alias http://home.udm.net/ file:/usr/local/apache/htdocs/
</PRE
>
		</P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>If you change the order of these commands, the alias for the <TT
CLASS="filename"
>/stat/</TT
> directory will never be found.</P
></BLOCKQUOTE
></DIV
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-server"
>Using alias in Server command</A
></H3
><P
>You may specify the location used by indexer as an optional argument for Server command:
		<PRE
CLASS="programlisting"
>&#13;Server  http://home.udm.net/  file:/home/httpd/htdocs/
</PRE
>
		</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-realm"
>Using alias in Realm command</A
></H3
><P
>Aliases in Realm command are a very powerful feature based on regular expressions. This feature implementation is similar to how PHP <CODE
CLASS="function"
>preg_replace()</CODE
> function works. Aliases in Realm command work only if "regex" match type is used. It DOES NOT work with "string" match type.</P
><P
>Use this syntax for Realm aliases:
		<PRE
CLASS="programlisting"
>&#13;Realm regex &#60;URL_pattern&#62; &#60;alias_pattern&#62;
</PRE
>
		</P
><P
>Indexer searches URLs for matches to URL_pattern and builds an URL alias using alias_pattern. alias_pattern may contain references of the form $n, where n is a number in the range of 0-9. Every such reference will be replaced by text captured by the n'th parenthesized pattern. $0 refers to text matched by the whole pattern. Opening parentheses are counted from left to right (startingfrom 1) to obtain the number of the capturing subpattern.</P
><P
>Example: your company hosts several hundreds users with their domains in the form of <TT
CLASS="literal"
>www.username.yourname.com</TT
>. Every user's site is stored on disk in "htdocs" under the user's home directory: <TT
CLASS="literal"
>/home/username/htdocs/</TT
>.</P
><P
>You may write this command into indexer.conf (note that the dot '.' character has a special meaning in regular expressions and must be escaped with a '\' sign when dot is used in its usual meaning):
		<PRE
CLASS="programlisting"
>&#13;Realm regex (http://www\.)(.*)(\.yourname\.com/)(.*)  file:/home/$2/htdocs/$4
</PRE
>
		</P
><P
>Imagine that indexer processes the <TT
CLASS="filename"
>http://www.john.yourname.com/news/index.html</TT
> page. It will build patterns from $0 to $4:</P
><P
CLASS="literallayout"
><br>
&nbsp;&nbsp;&nbsp;$0&nbsp;=&nbsp;'http://www.john.yourname.com/news/index.htm'&nbsp;(whole&nbsp;patter&nbsp;match)<br>
&nbsp;&nbsp;&nbsp;$1&nbsp;=&nbsp;'http://www.'&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subpattern&nbsp;matches&nbsp;'(http://www\.)'<br>
&nbsp;&nbsp;&nbsp;$2&nbsp;=&nbsp;'john'&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;subpattern&nbsp;matches&nbsp;'(.*)'<br>
&nbsp;&nbsp;&nbsp;$3&nbsp;=&nbsp;'.yourname.com/'&nbsp;&nbsp;&nbsp;subpattern&nbsp;matches&nbsp;'(\.yourname\.com/)'<br>
&nbsp;&nbsp;&nbsp;$4&nbsp;=&nbsp;'/news/index.html'&nbsp;subpattern&nbsp;matches&nbsp;'(.*)'<br>
</P
><P
>Then indexer will compose alias using $2 and $4 patterns:
		<PRE
CLASS="programlisting"
>&#13;file:/home/john/htdocs/news/index.html
</PRE
>
		</P
><P
>and will use the result as document location to fetch it.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-prog"
>AliasProg command
			<A
NAME="AEN1303"
></A
></A
></H3
><P
>You may also specify "AliasProg" command for aliasing purposes. AliasProg is useful for major web hosting companies which want to index their web space taking documents directly from a disk without having to involve their web server in the indexing process. Documents layout may be very complex to describe using alias in Realm command. AliasProg is an external program that can be called, that takes a URL and returns one string with the appropriate alias to stdout. Use $1 to pass a URL to the command line.</P
><P
>For example this AliasProg command uses the 'replace' command from MySQL distribution and replaces URL substring <TT
CLASS="literal"
>http://www.apache.org/</TT
> with <TT
CLASS="literal"
>file:/usr/local/apache/htdocs/</TT
>:
		<PRE
CLASS="programlisting"
>&#13;AliasProg  "echo $1 | /usr/local/mysql/bin/mysql/replace http://www.apache.org/ file:/usr/local/apache/htdocs/"
		</PRE
>
		</P
><P
>You may also write your own very complex program to process URLs.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-reverse"
>ReverseAlias command
<A
NAME="AEN1314"
></A
></A
></H3
><P
>The <TT
CLASS="filename"
>indexer.conf</TT
> ReverseAlias command allows URL mapping before a URL is inserted into the database. Unlike the Alias command, that triggers mapping right before a document is downloaded, ReverseAlias command triggers mapping after the link is found. 
		<PRE
CLASS="programlisting"
>&#13;ReverseAlias http://name2/   http://name2.yourname.com/
Server       http://name2.yourname.com/
		</PRE
>
		</P
><P
>All links with a short server name will be mapped to links with a full server name before they are inserted into the database.</P
><P
>One possible use is cutting various unnecessary strings like <TT
CLASS="literal"
>PHPSESSID=XXXX</TT
></P
><P
>E.g. cutting from URL like <TT
CLASS="literal"
>http://www/a.php?PHPSESSID=XXX</TT
>, when PHPSESSID is the only parameter. The question sign is deleted as well:

		<PRE
CLASS="programlisting"
>&#13;ReverseAlias regex  (http://[^?]*)[?]PHPSESSID=[^&#38;]*$          $1$2
		</PRE
>
		</P
><P
>Cutting from URL like <TT
CLASS="literal"
>w/a.php?PHPSESSID=xxx&#38;..</TT
>, i.e. when PHPSESSID is the first parameter, but there are other parameters following it. The '&#38;' sign after PHPSESSID is deleted as well. The question mark (?) character is not deleted:

		<PRE
CLASS="programlisting"
>&#13;ReverseAlias regex  (http://[^?]*[?])PHPSESSID=[^&#38;]*&#38;(.*)      $1$2
</PRE
>
		</P
><P
>Cutting from URL like <TT
CLASS="literal"
>http://www/a.php?a=b&#38;PHPSESSID=xxx</TT
> or <TT
CLASS="literal"
>http://www/a.php?a=b&#38;PHPSESSID=xxx&#38;c=d</TT
>, where PHPSESSID is not the first parameter. The '&#38;' sign before PHPSESSID is deleted:

		<PRE
CLASS="programlisting"
>&#13;ReverseAlias regex  (http://.*)&#38;PHPSESSION=[^&#38;]*(.*)         $1$2
		</PRE
>
		</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-search"
>Alias in search.htm search template
			<A
NAME="AEN1335"
></A
></A
></H3
><P
>It is also possible to define aliases in the search template (<TT
CLASS="filename"
>search.htm</TT
>). The Alias command in <TT
CLASS="filename"
>search.htm</TT
> is identical to the one in <TT
CLASS="filename"
>indexer.conf</TT
>, however it is active during searching, not indexation.</P
><P
>The syntax of the <TT
CLASS="filename"
>search.htm</TT
> Alias command is the same as in <TT
CLASS="filename"
>indexer.conf</TT
>:
		<PRE
CLASS="programlisting"
>&#13;Alias &#60;find-prefix&#62; &#60;replace-prefix&#62;
		</PRE
>
		</P
><P
>For example, there is the following command in <TT
CLASS="filename"
>search.htm</TT
>:
		<PRE
CLASS="programlisting"
>&#13;Alias http://localhost/ http://www.mnogo.ru/
		</PRE
>
		</P
><P
>Search returns a page with the following URL:
		<PRE
CLASS="programlisting"
>&#13;http://localhost/news/article10.html
		</PRE
>
		</P
><P
>As a result, the $(DU) variable will be replace NOT with this URL:
		<PRE
CLASS="programlisting"
>&#13;http://localhost/news/article10.html
		</PRE
>
		</P
><P
>but with the following URL (that results in processing with Alias):
		<PRE
CLASS="programlisting"
>&#13;http://www.mnogo.ru/news/article10.html
		</PRE
>
		</P
></DIV
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="srvtable"
>ServerTable
	<A
NAME="AEN1357"
></A
></A
></H2
><P
>Since version 3.2.7 <SPAN
CLASS="application"
>mnoGoSearch</SPAN
> has "ServerTable" <TT
CLASS="filename"
>indexer.conf</TT
> command. </P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="srvtable-load"
>Loading servers table</A
></H3
><P
>When <B
CLASS="command"
>ServerTable mysql://user:pass@host/dbname/tablename[?srvinfo=infotablename]</B
>
is specified, indexer will load
the servers information from the given <TT
CLASS="literal"
>tablename</TT
> SQL table, and will load the servers parameters
from the given <TT
CLASS="literal"
>infotablename</TT
> SQL table. If the <TT
CLASS="literal"
>srvinfo</TT
> parameter is not specified,
the parameters will be loaded from the <TT
CLASS="literal"
>srvinfo</TT
> table. Check the structure of <TT
CLASS="literal"
>server</TT
> and
<TT
CLASS="literal"
>srvinfo</TT
> tables in <TT
CLASS="filename"
>create/mysql/create.txt</TT
> file. If there is no
structure example for your database, take it as an example. Please send us the structure of your database at
<CODE
CLASS="email"
>&#60;<A
HREF="mailto:devel@mnogosearch.org"
>devel@mnogosearch.org</A
>&#62;</CODE
>!</P
><P
>You may use several <B
CLASS="command"
>ServerTable</B
> commands to load the server's information from different tables. </P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="srvtable-structure"
>Server table structure</A
></H3
><P
>Server table consists of all the necessary fields which 
describe the servers parameters. Field names have correspondent
<TT
CLASS="filename"
>indexer.conf</TT
> commands. For example, to the "period"
field corresponds the <TT
CLASS="filename"
>indexer.conf</TT
> "Period"
command. Default field values are the same than default
<TT
CLASS="filename"
>indexer.conf</TT
> parameters.</P
><P
>The "gindex" field corresponds to the "Index" command. The name has been slightly changed to avoid SQL reserved word usage.</P
><P
>For the description of several fields see <A
HREF="msearch-dbschema.html"
>the Section called <I
>Database schema</I
> in Chapter 9</A
>.</P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>The servers are only read from the table where "active"
field has a 1 value and "parent" field has a 0 value. This is useful to allow users to submit new URLs
into servers table and give administrator a possibility to approve
added URLs.</P
></BLOCKQUOTE
></DIV
></DIV
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="flushsrvtable"
>FlushServerTable
<A
NAME="AEN1390"
></A
></A
></H2
><P
>&#13; Flush server sets active field to inactive for all ServerTable records. 
Use this command to deactivate all commands in ServerTable before loading new commands from <TT
CLASS="filename"
>indexer.conf</TT
>
or from other ServerTable.
</P
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="pars"
>External parsers
    <A
NAME="AEN1397"
></A
></A
></H2
><P
>mnoGoSearch indexer can use external parsers to index various file types (mime types).</P
><P
>A parser is an executable program which converts one of
    the mime types to <TT
CLASS="literal"
>text/plain</TT
> or
    <TT
CLASS="literal"
>text/html</TT
>. For example, if you have some postscript
    files, you can use the ps2ascii parser (filter), which reads postscript
    file from stdin and produces ascii to stdout, to be able to index their contents.
  </P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-sup"
>Supported parser types</A
></H3
><P
>Indexer supports four types of parsers that can:</P
><P
></P
><UL
><LI
><P
>read data from stdin and send the result to stdout</P
></LI
><LI
><P
>read data from file  and send the result to stdout</P
></LI
><LI
><P
>read data from file  and send the result to file</P
></LI
><LI
><P
>read data from stdin and send the result to file</P
></LI
></UL
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-setup"
>Setting up parsers</A
></H3
><P
></P
><OL
TYPE="1"
><LI
><P
>Configure mime types</P
><P
>Configure your web server to
send appropriate "Content-Type" header. For Apache, have a look at
<TT
CLASS="filename"
>mime.types</TT
> file, most mime types are already
defined there.
        </P
><P
>&#13;<A
NAME="AEN1423"
></A
>
If you want to index local files or via <TT
CLASS="literal"
>ftp</TT
>
use "AddType" command in <TT
CLASS="filename"
>indexer.conf</TT
> to
associate file name extensions with their mime types. For
example:
        <PRE
CLASS="programlisting"
>&#13;AddType text/html *.html
</PRE
>
        </P
></LI
><LI
><P
>&#13;          <A
NAME="AEN1431"
></A
>
          Add parsers
        </P
><P
>Add lines with parsers definitions. Lines have the following format with three arguments:
        <PRE
CLASS="programlisting"
>&#13;Mime &#60;from_mime&#62; &#60;to_mime&#62; &#60;command line&#62;
</PRE
>
        </P
><P
>For example, the following line defines the parser for man pages:
        <PRE
CLASS="programlisting"
>&#13;# Use deroff for parsing man pages ( *.man )
Mime  application/x-troff-man   text/plain   deroff
        </PRE
>
        </P
><P
>This parser will take data from stdin and output result to stdout.</P
><P
>Many parsers can not operate on
stdin and require a file to read from. In this case indexer creates a
temporary file in /tmp and removes it when the parser is done. Use $1
macro in parser command line to substitute file name. For example,
Mime command for "catdoc" MS Word to ASCII converters may look like
this:

        <PRE
CLASS="programlisting"
>&#13;Mime application/msword text/plain "/usr/bin/catdoc -a $1"
</PRE
>
        </P
><P
>If your parser writes the result
into an output file, use $2 macro. indexer will replace $2 by a temporary
file name, start the parser, read the result from this temporary file, then
remove it. For example:
        <PRE
CLASS="programlisting"
>&#13;Mime application/msword text/plain "/usr/bin/catdoc -a $1 &#62;$2"
</PRE
>
        </P
><P
>The parser above will read data
from first temporary file and write result to second one. Both
temporary files will be removed when parser exits. Note that the result
of using this parser will be absolutely the same than the previous
one, but they use different execution modes: file-&#62;stdout and
file-&#62;file correspondingly.
        </P
></LI
></OL
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="ParserTimeOut"
>Avoid indexer hang on parser execution</A
></H3
><P
>&#13;<A
NAME="AEN1447"
></A
>
To prevent the indexer to hang on parser's execution, you may specify 
an amount of time in seconds for parser's execution in your <TT
CLASS="filename"
>indexer.conf</TT
> by using the <TT
CLASS="literal"
>ParserTimeOut</TT
> command. For example:
    <PRE
CLASS="programlisting"
>&#13;ParserTimeOut 600
</PRE
>
</P
><P
>&#13;Default value is 300 seconds, i.e. 5 minutes.
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-pipes"
>Pipes in parser's command line</A
></H3
><P
>You can use pipes in parser's command line. For
example, these lines will be useful to index gzipped man pages from
local disk:
    <PRE
CLASS="programlisting"
>&#13;AddType  application/x-gzipped-man  *.1.gz *.2.gz *.3.gz *.4.gz
Mime     application/x-gzipped-man  text/plain  "zcat | deroff"
</PRE
>
    </P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-char"
>Charsets and parsers
<A
NAME="AEN1460"
></A
></A
></H3
><P
>Some parsers can produce output in any other charset
than given in the LocalCharset command. Specify a charset to make the indexer
convert parser's output to a proper one. For example, if your catdoc is
configured to produce output in windows-1251 charset but LocalCharset
is koi8-r, use this command for parsing MS Word documents:
    <PRE
CLASS="programlisting"
>&#13;Mime  application/msword  "text/plain; charset=windows-1251" "catdoc -a $1"
</PRE
>
    </P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-udmurl"
>UDM_URL variable</A
></H3
><P
>When executing a parser, the indexer creates the UDM_URL
environment variable with an URL being processed as a value. You can
use this variable in parser scripts.
    </P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>
When running several threads, don't relay on the UDM_URL variable,
use ${URL} variable in the parser
command line instead. See <A
HREF="msearch-cmdref-mime.html"
>Mime</A
> for more details.
      </P
></BLOCKQUOTE
></DIV
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-links"
>Some third-party parsers
<A
NAME="AEN1473"
></A
></A
></H3
><P
></P
><UL
><LI
><P
>RPM parser by Mario Lang <CODE
CLASS="email"
>&#60;<A
HREF="mailto:lang@zid.tu-graz.ac.at"
>lang@zid.tu-graz.ac.at</A
>&#62;</CODE
></P
><P
>        /usr/local/bin/rpminfo:

<PRE
CLASS="programlisting"
>&#13;#!/bin/bash
/usr/bin/rpm -q --queryformat="&#60;html&#62;&#60;head&#62;&#60;title&#62;RPM: %{NAME} %{VERSION}-%{RELEASE}
(%{GROUP})&#60;/title&#62;&#60;meta name=\"description\" content=\"%{SUMMARY}\"&#62;&#60;/head&#62;&#60;body&#62;
%{DESCRIPTION}\n&#60;/body&#62;&#60;/html&#62;" -p $1
</PRE
>
</P
><P
>indexer.conf:

<PRE
CLASS="programlisting"
>&#13;Mime application/x-rpm text/html "/usr/local/bin/rpminfo $1"
</PRE
>
</P
><P
>It renders to such nice RPM information:

<PRE
CLASS="programlisting"
>&#13;3. RPM: mysql 3.20.32a-3 (Applications/Databases) [4]
       Mysql is a SQL (Structured Query Language) database server.
       Mysql was written by Michael (Monty) Widenius. See the CREDITS
       file in the distribution for more credits for mysql and related
       things....
       (application/x-rpm) 2088855 bytes
</PRE
>
</P
></LI
><LI
><P
>catdoc MS Word to text converter</P
><P
>&#13;          <A
HREF="http://freshmeat.net/redir/catdoc/1055/url_homepage/"
TARGET="_top"
>Home page</A
>, also listed on <A
HREF="http://freshmeat.net/"
TARGET="_top"
>Freshmeat</A
>.
        <PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime application/msword "text/plain; charset=utf-8"  "catdoc -d utf-8 $1"
</PRE
>
</P
></LI
><LI
><P
>wvWare MS Word to HTML converter</P
><P
>&#13;           <A
HREF="http://wvware.sourceforge.net/"
TARGET="_top"
>Home page</A
>,
           also listed on <A
HREF="http://freshmeat.net/"
TARGET="_top"
>Freshmeat</A
>.
<PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime application/msword    "text/html; charset=utf-8"    "wvHtml --charset=utf-8 $1 -"
</PRE
>
</P
></LI
><LI
><P
>xls2csv MS Excel to text converter</P
><P
>It is supplied with catdoc.
        <PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime application/vnd.ms-excel   text/plain      "xls2csv $1"
</PRE
>
</P
></LI
><LI
><P
>xlhtml Excel XLS to HTML converter</P
><P
>Available from the project
        <A
HREF="http://www.xlhtml.org/"
TARGET="_top"
>homepage</A
>,
        also listed on <A
HREF="http://freshmeat.net/projects/xlhtml/"
TARGET="_top"
>Freshmeat</A
>.
        and  <A
HREF="http://chicago.sourceforge.net/xlhtml/"
TARGET="_top"
>SourceForge</A
>.
        Download page includes <A
HREF="http://chicago.sourceforge.net/xlhtml/binarys.html"
TARGET="_top"
>binaries for Windows</A
>.
        </P
><P
>&#13;          <PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime application/vnd.ms-excel  text/html  "xltohtml $1"
          </PRE
>
        </P
></LI
><LI
><P
>pptohtml PowerPoint PPT to HTML converter</P
><P
>A part of xlhtml distribution. Available from the project
        <A
HREF="http://www.xlhtml.org/"
TARGET="_top"
>homepage</A
>,
        also listed on <A
HREF="http://freshmeat.net/projects/xlhtml/"
TARGET="_top"
>Freshmeat</A
>.
        and  <A
HREF="http://chicago.sourceforge.net/xlhtml/"
TARGET="_top"
>SourceForge</A
>.
        Download page includes <A
HREF="http://chicago.sourceforge.net/xlhtml/binarys.html"
TARGET="_top"
>binaries for Windows</A
>.
        </P
><P
>&#13;          <PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime application/vnd.ms-powerpoint   text/html   "pptohtml $1"
          </PRE
>
        </P
></LI
><LI
><P
>pdftotext Adobe PDF converter</P
><P
>Supplied with xpdf project.</P
><P
>&#13;          <A
HREF="http://freshmeat.net/redir/xpdf/12080/url_homepage/"
TARGET="_top"
>Homepage</A
>, also listed on <A
HREF="http://freshmeat.net/"
TARGET="_top"
>Freshmeat</A
>.
        <PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime application/pdf            text/plain      "pdftotext $1 -"
</PRE
>
</P
></LI
><LI
><P
>unrtf RTF to html converter</P
><P
>&#13;          <A
HREF="ftp://ftp.gnu.org/pub/gnu/unrtf/"
TARGET="_top"
>Homepage</A
>
        <PRE
CLASS="programlisting"
>&#13;indexer.conf:

Mime text/rtf*        text/html                 "/usr/local/mnogosearch/sbin/unrtf --html $1
Mime application/rtf        text/html           "/usr/local/mnogosearch/sbin/unrtf --html $1
</PRE
>
</P
></LI
></UL
><P
>&#13;      Please feel free to contribute your scripts and
      parsers configuration to <CODE
CLASS="email"
>&#60;<A
HREF="mailto:general@mnogosearch.org"
>general@mnogosearch.org</A
>&#62;</CODE
>.
    </P
></DIV
></DIV
></DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="msearch-content-enc.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="index.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="msearch-extended-indexing.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>Content-Encoding support
		<A
NAME="AEN994"
></A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="msearch-indexing.html"
ACCESSKEY="U"
>Up</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>Extended indexing features</TD
></TR
></TABLE
></DIV
><!--#include virtual="body-after.html"--></BODY
></HTML
>