File: archive_search.html

package info (click to toggle)
hypermail 2.2.0.dfsg-2
links: PTS
area: main
in suites: lenny
size: 5,636 kB
ctags: 3,069
sloc: ansic: 34,221; sh: 15,510; yacc: 844; perl: 744; makefile: 737; python: 292
file content (552 lines) | stat: -rw-r--r-- 18,409 bytes
parent folder | download | duplicates (3)
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
   <title>Adding a search engine to your Hypermail archives</title>
   <meta http-equiv="Content-Type" content="text/html;CHARSET=iso-8859-1" />
   <meta name="description" content="hypermail documentation: adding search engine" />
   <meta name="keywords" content="search, hypermail" />
   <meta name="Author" content="Bob Crispen" />
   <style type="text/css">
      body, html {
         color: #000;
         background: #fff;
         padding: 0.2em 2em;
      }
      pre {
         color: #000;
         background: #eee;
         border: 1px solid #888;
         padding-top: 1em;
         margin-top: 0em;
         white-space: pre;
      }
      h1 {
         text-align: center;
         font-family: sans-serif;
         font-weight: bold;
         background: #deedf8;
      }
      h4 {
         border-top: 1px solid #888;
         border-left: 1px solid #888;
         border-right: 1px solid #888;
         text-align: center;
         font-family: sans-serif;
         font-weight: bold;
         background: #deedf8;
         padding-bottom: 0em;
         margin-bottom: 0em;
      }
   </style>
</head>
<body>
<h1>Adding a search engine<br />to your Hypermail archives</h1>

<p>One of the feature requests we hear most often is to incorporate
a search engine in Hypermail.  But because everyone has his or her
own favorite search engine, and since it's hard to imagine a search engine
that may not turn out to be unusable
for some archives, we haven't built a search engine into hypermail, and
we probably won't.</p>

<p>But hypermail's page
customizations make it easy to integrate your own search engine into your
hypermail archives.</p>

<p>For our example, we're going to put a form box on the top and bottom of
every index page, and we'll use the <a href="http://swish-e.org/">swish-e</a>
search engine.  We'll show a typical <a href="http://www.php.net/">PHP</a>
script and a typical <a href="http://www.cpan.org/">Perl</a> script that can
function as a glue layer between the web and the search engine.
There ought to be enough information here to get you
started regardless of what search engine and scripting language you choose
for your site.</p>

<p>Let's begin by modifying the header and footer hypermail puts on each
index file.</p>

<p>(1) Create a file called indexheader.hyp containing the following:</p>
<h4>indexheader.hyp</h4>
<pre>
    &lt;!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"&gt;
    &lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"&gt;
    &lt;head&gt;
    &lt;title&gt;%l: %s&lt;/title&gt;
    &lt;meta http-equiv="Content-Type" content="text/html;CHARSET=iso-8859-1" /&gt;
    &lt;meta name="generator" content="%p %v, see %h" /&gt;
    &lt;meta name="Subject" content="%s" /&gt;
    &lt;meta name="Date" content="(nil)" /&gt;
    &lt;link rev="made" href="mailto:%m" /&gt;
    &lt;style type="text/css"&gt;
        body {color: black; background: #ffffff}
        h1.center {text-align: center}
        div.center {text-align: center}
        .quotelev1 {color : #990099}
        .quotelev2 {color : #ff7700}
        .quotelev3 {color : #007799}
        .quotelev4 {color : #95c500}
        .headers {background : #e0e0d0}
        .links {background : #f8f8e0}
    &lt;/style&gt;
    &lt;/head&gt;
    &lt;body&gt;
    &lt;h1 class="center"&gt;%l&lt;br /&gt;%s&lt;/h1&gt;

    &lt;!-- Your custom code goes below this line --&gt;

    &lt;div class="center"&gt;
    &lt;form action="http://url/of/search.php" method="post"
        enctype="application/x-www-form-urlencoded"&gt;
    &lt;div&gt;
    &lt;input type="hidden" name="db" value="name.of.index" /&gt;
    &lt;p class="headers"&gt;
    &lt;input type="text" name="str" size="20" /&gt;
    &lt;input type="submit" value="Search" /&gt;
    &lt;/p&gt;
    &lt;/div&gt;
    &lt;/form&gt;
    &lt;/div&gt;

</pre>

<p>Notice the HTML comment about 2/3 of the way down.  Everything
above that line is standard, and ought to be very similar to the default
header.  The custom code (below the comment) adds the features you
want to add.  In this case, it's a form for a search.</p>

<p>You will, of course, substitute your own URL for search.php for the form's ACTION
and the name of your own search index for "name.of.index".  See the bottom of this page for
an example of creating a search index file.</p>

<p>If you like the general appearance of the web pages Hypermail creates and
don't want to spend a lot of time playing with the HTML, you might want to
look at the source of "index.html" in the directory for which you're installing
a search engine and just copy the appropriate lines (<i>e.g.</i>, the lines
above the first &lt;hr /&gt;) to indexheader.hyp.</p>

<p>(2) Create a file called indexfooter.hyp containing the following:</p>

<h4>indexfooter.hyp</h4>
<pre>
    &lt;div class="center"&gt;
    &lt;form action="/url/of/search.php" method="post"
        enctype="application/x-www-form-urlencoded"&gt;
    &lt;div&gt;
    &lt;input type="hidden" name="db" value="name.of.index" /&gt;
    &lt;p class="headers"&gt;
    &lt;input type="text" name="str" size="20" /&gt;
    &lt;input type"submit" value="Search" /&gt;
    &lt;/p&gt;
    &lt;/div&gt;
    &lt;/form&gt;
    &lt;/div&gt;

    &lt;!-- Your custom code goes above this line --&gt;

    &lt;hr /&gt;
    &lt;p&gt;&lt;small&gt;&lt;em&gt;
    This archive was generated by &lt;a href="%h"&gt;%p %v&lt;/a&gt; : %g
    &lt;/em&gt;&lt;/small&gt;&lt;/p&gt;
    &lt;/body&gt;
    &lt;/html&gt;

</pre>
<p>Since this is a footer file, the common code will be at the end, and
you'll put your custom code above the comment.</p>

<p>You may have noted what seem to be superfluous &lt;div&gt; elements in
the HTML in the header and footer file.
The <a href="http://validator.w3.org/">W3C validator</a> likes them, and
they do no harm.</p>

<p>(3) Modify the .hmrc file to point to your custom header and footer:</p>

<h4>.hmrc (excerpt)</h4>
<pre>
    # ihtmlheaderfile = [ path to index header template file | NONE ]
    #
    # Set this to the path to the Index header template file containing
    # valid HTML statements and substitution cookies for runtime expansion.
    # This will be included at the top of every index page.
    
    ihtmlheaderfile = /path/to/indexheader.hyp
    
    # ihtmlfooterfile = [ path to index footer template file | NONE ]
    #
    # Set this to the path to the Index footer template file containing
    # valid HTML statements and substitution cookies for runtime expansion.
    # This will be included at the bottom of every index page.
    
    ihtmlfooterfile = /path/to/indexfooter.hyp

</pre>
<p>You'll call hypermail using this .hmrc file to create your archive
or add a message:</p>

<pre>
    hypermail -c /path/to/.hmrc [...]

</pre>
<p>Here's a sample PHP script that performs the minimum functionality
required for "search.php":</p>

<h4>search.php</h4>
<pre>
    &lt;!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"&gt;
    &lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"&gt;
    &lt;head&gt;
    &lt;title&gt;Search results&lt;/title&gt;
    &lt;meta http-equiv="Content-Type" content="text/html;CHARSET=iso-8859-1" /&gt;
    &lt;/head&gt;
    &lt;body&gt;
    &lt;div&gt;
    &lt;?
        // Very simple search using swish-e (see http://swish-e.org/)
        // copyright 2003 by Bob Crispen &lt;http://www.crispen.org/&gt;
        // May be distributed without restriction for any purpose.
    
        // This program is distributed in the hope that it will be useful,
        // but WITHOUT ANY WARRANTY; without even the implied warranty of
        // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
        // GNU General Public License for more details.
    
        // Set these values to the appropriate ones for your system
        $db_dir = "/path/to/swish-e-databases/";          // Directory where
            // you keep the databases and indexes that swish-e generates
        $swishe = "/usr/local/bin/swish-e";               // swish-e executable
    
        // Get arguments from call
        //  "str" -- String to search for
        //  "db"  -- swish-e Database name (e.g., "site.index")
        $str = $_POST["str"];
        $db  = $_POST["db"];
    
        // This script is probably called from a form similar to the one
        // below.  If you call it without arguments, all it'll do is print
        // the form.
    
        if (($str != "") &amp;&amp; ($db != "")) {
            // Prevent shell execution exploit
            $searchfor = escapeshellarg($str);
            $database  = escapeshellarg($db_dir . $db);
    
            // Do the search
            $result = `$swishe -H 0 -d '\t' -w $searchfor -f $database`;
    
            // Turn the results into an array
            $results = split("\n", $result);
    
            // See how many results we have.  Ignore the final blank line.
            $count = count($results)-1;
            if ($count &gt; 0) {
                print("Search results for &lt;strong&gt;$searchfor&lt;/strong&gt;:\n");
                print("&lt;ul&gt;\n");
                for ($i=0; $i&lt;$count; $i++) {
                    list($score, $url, $title, $len) = split("\t", $results[$i], 4);
                    print("&lt;li&gt; &lt;a href=\"$url\"&gt;$title&lt;/a&gt; [$score]&lt;/li&gt;\n");
                }
                print("&lt;/ul&gt;\n");
            } else {
                print("Sorry, &lt;strong&gt;$searchfor&lt;/strong&gt; not found\n");
            }
        }
    
        // Print a new form so they can continue searching
        print &lt;&lt;&lt;EOT
    &lt;form action="$PHP_SELF" method="post"
        enctype="application/x-www-form-urlencoded"&gt;
    &lt;div&gt;
    &lt;input type="hidden" name="db" value="$db" /&gt;
    &lt;input type="text" name="str" size="20" /&gt;
    &lt;input type="submit" value="Search" /&gt;
    &lt;/div&gt;
    &lt;/form&gt;
    &lt;/div&gt;

    EOT;
    ?&gt;
    &lt;/body&gt;
    &lt;/html&gt;

</pre>

<p>If you prefer Perl, or if your web server doesn't offer PHP, then you could
modify these lines in indexheader.hyp and indexfooter.hyp:</p>

<pre>
    &lt;form action="/url/of/search.php" method="post"
        enctype="application/x-www-form-urlencoded"&gt;

</pre>

<p>to</p>

<pre>
    &lt;form action="/url/of/search.pl" method="post"
        enctype="application/x-www-form-urlencoded"&gt;

</pre>

<p>Here's a sample script contributed by Perl maven Greg Bacon that
performs the minimum functionality required for "search.pl":</p>

<h4>search.pl</h4>
<pre>

    #! /usr/local/bin/perl -T
    
    # Perl implementation of Bob Crispen's PHP swish-e search available
    # at &lt;URL:http://www.crispen.org/doc/hypermail/archive_search.html&gt;
    # Copyright 2003 Greg Bacon.
    
    # Very simple search using swish-e (see http://swish-e.org/)
    # copyright 2003 by Bob Crispen &lt;http://www.crispen.org/&gt;
    # May be distributed without restriction for any purpose.
        
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    # GNU General Public License for more details.
    
    use warnings;
    use strict;
    
    my @warnings;
    
    BEGIN {
        $SIG{__WARN__} = sub { push @warnings, @_ };
    
        print &lt;&lt;'EOHeader';
    Content-type: text/html
    
    &lt;!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"&gt;
    &lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"&gt;
    &lt;head&gt;
    &lt;title&gt;Search results&lt;/title&gt;
    &lt;meta http-equiv="Content-Type" content="text/html;CHARSET=iso-8859-1" /&gt;
    &lt;/head&gt;
    &lt;body&gt;
    &lt;div&gt;
    EOHeader
    
        $SIG{__DIE__} = sub {
            print "&lt;h1&gt;Error!&lt;/h1&gt;\n",
                  map { "&lt;pre&gt;$_&lt;/pre&gt;\n" } @_;
        };
    }
    
    use CGI qw/ :standard /;
    use HTML::Entities;
    
    # keep the taint checker happy
    $ENV{PATH} = "/bin:/usr/bin:/usr/local/bin";
    
    # ----------------------- Configuration ------------------------
    
    # Directory where you keep the databases and
    # indexes that swish-e generates
    my $db_dir = "/path/to/index/";
    
    # swish-e executable
    my $swishe = "/path/to/bin/swish-e";
    
    # -------------------- End of Configuration --------------------
    
    sub drop_privs {
        my @temp = ($&gt;, $));
    
        my $orig_uid = $&lt;;
        my $orig_gid = $(;
    
        # set effective user and group id to real 
        $&gt; = $&lt;;
        $) = $(;
    
        # Drop privileges
        $&lt; = $orig_uid;
        $( = $orig_gid;
    
        # Make sure privs are really gone
        ($&gt;, $)) = @temp;
        die "FATAL: can't drop privileges" unless $&lt; == $&gt; &amp;&amp; $( eq $);
    }
    
    sub search {
        my $str = shift;
        my $db  = shift;
    
        return unless $str &amp;&amp; $db;
    
        my $database = $db_dir . $db;
        
        my $pid = open SWISHE, "-|";
        unless (defined $pid) {
            warn "failed fork: $!";
            return;
        }
    
        my @results;
    
        if ($pid) {
            # parent
    
            while (&lt;SWISHE&gt;) {
                chomp;
    
                # swish-e output is four TAB-separated fields, so
                # we assume lines with three TABs are from swish-e
                # and all other non-blank lines are warnings
                if (/\t.*\t.*\t/) {
                    push @results, [split /\t/, $_, 4];
                }
                else {
                    push @warnings, $_ if /\S/;
                }
            }
    
            close SWISHE
                or warn $! ? "Error closing $swishe pipe: $!"
                           : "Exit status $? from $swishe";
    
        }
        else {
            # child
    
            local $SIG{__WARN__} = sub { print @_ };
            local $SIG{__DIE__}  = sub { print @_; exit 1 };
    
            # 2&gt;&amp;1
            open STDERR, "&gt;&amp;STDOUT" or warn "WARNING: dup STDOUT: $!";
    
            drop_privs;
    
            # Do the search
            no warnings;
            exec $swishe, '-H', 0, '-d', '\t', '-w', $str, '-f', $database
                or die "FATAL: exec $swishe: $!";
    
            exit 1;
        }
    
        my $searchfor = encode_entities $str;
        if (@results) {
            print "Search results for &lt;strong&gt;$searchfor&lt;/strong&gt;:\n",
                  "&lt;ul&gt;\n";
    
            for (@results) {
                my($score,$url,$title,$len) = @$_;
    
                print qq{  &lt;li&gt; &lt;a href="$url"&gt;$title&lt;/a&gt; [$score]&lt;/li&gt;\n};
            }
    
            print "&lt;/ul&gt;\n";
        }
        else {
            print("Sorry, &lt;strong&gt;$searchfor&lt;/strong&gt; not found\n");
        }
    }
    
    ## main
    
    # URL of this page
    my $me = url -full =&gt; 1;
    
    # Get arguments from call
    #  "str" -- String to search for
    #  "db"  -- swish-e Database name (e.g., "site.index")
    my $str = param "str";
    my $db  = param "db";
    
    # This script is probably called from a form similar to the one
    # below.  If you call it without arguments, all it'll do is print
    # the form.
    
    my @results = search $str, $db;
    
    # Print a new form so they can continue searching
    print startform,
          "&lt;div&gt;\n",
          hidden(db =&gt; $db),
          textfield(-name =&gt; 'str', -size =&gt; 20),
          submit('Search'),
          "&lt;/div&gt;\n",
          end_form, "\n";
    
    if (@warnings) {
        my $messages = @warnings == 1 ? "message" : "messages";
    
        print &lt;&lt;EOWarningsHead;
    &lt;p&gt;&lt;hr&gt;
    &lt;h1&gt;Warnings&lt;/h1&gt;
    &lt;em&gt;Warning $messages:&lt;/em&gt;
    &lt;ul&gt;
    EOWarningsHead
    
        for (@warnings) {
            print "  &lt;li&gt;&lt;code&gt;$_&lt;/code&gt;&lt;/li&gt;\n";
        }
    
        print "&lt;/ul&gt;\n";
    }
    
    print &lt;&lt;EOFooter;
    
    &lt;/div&gt;
    &lt;/body&gt;
    
    &lt;/html&gt;
    EOFooter

</pre>

<p>Swish-e, like many other search engines, requires you to generate
a search index before you perform any searches.  Here's a .conf file for
swish-e that might generate a reasonable
search index for a hypermail archive of messages about model railroading:</p>

<h4>model-rr.conf</h4>
<pre>
    IndexDir /path/to/model-rr-archive
    IndexFile /path/to/model-rr.index
    IndexReport 3
    IndexOnly .html
    ReplaceRules replace "/path/to/document_root/" "http://www.yoursite.com/"
    FileRules filename is attachment.html
    FileRules filename is author.html
    FileRules filename is date.html
    FileRules filename is index.html
    FileRules filename is subject.html
    FileRules filename is thread.html

</pre>

<p>This tells swish-e to collect data from all the HTML files in your
model railroading archive <em>except</em> the index pages that hypermail
generates.  That way, all the search results will point directly to the
messages themselves.</p>

<p>When you want to build your search index, you'll call swish-e something like this:</p>

<pre>
    swish-e -v 3 -c /path/to/model-rr.conf &gt; /path/to/model-rr.report 2&gt;&amp;1

</pre>
<p>Many people do this either in a cron job during off-peak hours or
through a CGI script that they call whenever they update their archive.</p>

<p>You might want to add search logging, page control (so you only
print, say, 20 results at a time), some nice CSS, and all sorts of other
things to your script.  And if you'd prefer to write your script in
Python, sh, or Ada, you can do that too.  From here on, it's up to you.</p>

<div>
-- 
<address>Bob Crispen</address>
<address>Thursday, June 26, 2003</address>
</div>
</body>
</html>