File: sitescooper.html

package info (click to toggle)
sitescooper 3.1.2-1
links: PTS
area: main
in suites: sarge, woody
size: 3,000 kB
ctags: 662
sloc: perl: 8,677; makefile: 105
file content (729 lines) | stat: -rw-r--r-- 26,817 bytes
<HTML>
<HEAD>
<TITLE>sitescooper - download news from web sites and convert it automatically
into one of several formats suitable for viewing on a Palm handheld.</TITLE>
<LINK REV="made" HREF="mailto:root@porky.devel.redhat.com">
</HEAD>

<BODY>

<!-- INDEX BEGIN -->

<UL>

	<LI><A HREF="#NAME">NAME</A>
	<LI><A HREF="#SYNOPSIS">SYNOPSIS</A>
	<LI><A HREF="#DESCRIPTION">DESCRIPTION</A>
	<LI><A HREF="#OPTIONS">OPTIONS</A>
	<LI><A HREF="#OUTPUT_TEMPLATES">OUTPUT TEMPLATES</A>
	<LI><A HREF="#INSTALLATION">INSTALLATION</A>
	<LI><A HREF="#EXAMPLES">EXAMPLES</A>
	<LI><A HREF="#ENVIRONMENT">ENVIRONMENT</A>
	<LI><A HREF="#AUTHOR">AUTHOR</A>
	<LI><A HREF="#COPYRIGHT">COPYRIGHT</A>
	<LI><A HREF="#SCRIPT_CATEGORIES">SCRIPT CATEGORIES</A>
	<LI><A HREF="#PREREQUISITES">PREREQUISITES</A>
	<LI><A HREF="#COREQUISITES">COREQUISITES</A>
	<LI><A HREF="#README">README</A>
</UL>
<!-- INDEX END -->

<HR>
<P>
<H1><A NAME="NAME">NAME</A></H1>
<P>
sitescooper - download news from web sites and convert it automatically
into one of several formats suitable for viewing on a Palm handheld.

<P>
<HR>
<H1><A NAME="SYNOPSIS">SYNOPSIS</A></H1>
<P>
sitescooper [options] [ [-site sitename] ...]

<P>
sitescooper [options] [-sites sitename ...]

<P>
sitescooper [options] [-name nm] [-levels n] [-storyurl regexp] [-set
sitefileparam value] url [...]

<P>
Options: [-debug] [-refresh] [-fullrefresh] [-config file] [-install dir]
[-instapp app] [-dump] [-dumpprc] [-nowrite] [-nodates] [-quiet] [-admin
cmd] [-nolinkrewrite] [-stdout-to file] [-badcache] [-keep-tmps]
[-fromcache] [-noheaders] [-nofooters] [-outputtemplate file.tmpl] [-grep]
[-profile file.nhp] [-profiles file.nhp file2.nhp ...] [-filename template]
[-prctitle template] [-parallel] [-disc] [-limit numkbytes] [-maxlinks
numlinks] [-maxstories numstories]

<P>
[-text | -html | -mhtml | -doc | -plucker | -mplucker | -isilo | -misilo |
-richreader | -pipe fmt command] [-bw | -color] [-cvtargs
args_for_converter]

<P>
<HR>
<H1><A NAME="DESCRIPTION">DESCRIPTION</A></H1>
<P>
This script, in conjunction with its configuration file and its set of
<STRONG>site</STRONG> files, will download news stories from several top news sites into text
format and/or onto your Palm handheld (with the aid of the
<STRONG>makedoc</STRONG>/<STRONG>MakeDocW</STRONG> or <STRONG>iSilo</STRONG> utilities).

<P>
Alternatively URLs can be supplied on the command line, in which case those
URLs will be downloaded and converted using a reasonable set of default
settings.

<P>
HTTP and local files, using the <CODE>file:///</CODE> protocol, are both supported.

<P>
Multiple types of sites are supported:

<P>
1-level sites, where the text to be converted is all present on one page
(such as Slashdot, Linux Weekly News, BluesNews, NTKnow, Ars Technica);

<P>
2-level sites, where the text to be converted is linked to from a Table of
Contents page (such as Wired News, BBC News, and I, Cringely);

<P>
3-level sites, where the text to be converted is linked to from a Table of
Contents page, which in turned is linked to from a list of issues page
(such as PalmPower).


<P>
In addition sites that post news as items on one big page, such as
Slashdot, Ars Technica, and BluesNews, are supported using diff.

<P>
Note that at this moment in time, the URLs-on-the-command-line invocation
format does not support 2- or 3-level sites.

<P>
The script is portable to most UNIX variants that support perl, as well as
the Win32 platform (tested with ActivePerl 5.00502 build 509).

<P>
sitescooper maintains a cache in its temporary directory; files are kept in
this cache for a week at most. Ditto for the text output directory (set
with <STRONG>TextSaveDir</STRONG> in the built-in configuration).

<P>
If a password is required for the site, and the current sitescooper session
is interactive, the user will be prompted for the username and password.
This authentication token will be saved for later use. This way a site that
requires login can be set up as a .site -- just log in once, and your
password is saved for future non-interactive runs.

<P>
Note however that the encryption used to hide the password in the
sitescooper configuration is pretty transparent; I recommend that rather
than using your own username and password to log in to passworded sites, a
dedicated, sitescooper account is used instead.

<P>
<HR>
<H1><A NAME="OPTIONS">OPTIONS</A></H1>
<DL>
<DT><STRONG><A NAME="item__refresh">-refresh</A></STRONG><DD>
<P>
Refresh all links -- ignore the <EM>already_seen</EM> file, do not diff pages, and always fetch links. If a cached page is
available, it will be used.

<DT><STRONG><A NAME="item__fullrefresh">-fullrefresh</A></STRONG><DD>
<P>
Refresh all links -- ignore the <EM>already_seen</EM> file, do not diff pages, and always fetch links, even if they are available
in the cache.

<DT><STRONG><A NAME="item__config">-config file</A></STRONG><DD>
<P>
Read the configuration from <STRONG>file</STRONG> instead of using the built-in one.

<DT><STRONG><A NAME="item__limit">-limit numkbytes</A></STRONG><DD>
<P>
Set the limit for output file size to <STRONG>numkbytes</STRONG> kilobytes, instead of the default 200K. A limit of 0 means unlimited, any
amount of output.

<DT><STRONG><A NAME="item__maxlinks">-maxlinks numlinks</A></STRONG><DD>
<P>
Stop retrieving web pages after <STRONG>numlinks</STRONG> have been traversed. This is not used to specify how ``deep'' a site should
be scooped -- it is the number of links followed in total.

<DT><STRONG><A NAME="item__maxstories">-maxstories numstories</A></STRONG><DD>
<P>
Stop retrieving web pages after <STRONG>numstories</STRONG> stories have been retrieved.

<DT><STRONG><A NAME="item__install">-install dir</A></STRONG><DD>
<P>
The directory to save PDB files to once they've been converted, in order to
have them installed to your Palm handheld.

<DT><STRONG><A NAME="item__instapp">-instapp app</A></STRONG><DD>
<P>
The application to run to install PDB files onto your Palm, once they've
been converted.

<DT><STRONG><A NAME="item__site">-site sitename</A></STRONG><DD>
<P>
Limit the run to the site named in the <STRONG>sitename</STRONG> argument. Normally all available sites will be downloaded. To limit the run
to 2 or more sites, provide multiple <STRONG>-site</STRONG> arguments like so:

<P>
<PRE>        -site ntk.site -site tbtf.site
</PRE>
<DT><STRONG><A NAME="item__sites">-sites sitename [...]</A></STRONG><DD>
<P>
Limit the run to multiple sites; an easier way to specify multiple sites
than using the -site argument for each file.

<DT><STRONG><A NAME="item__grep">-grep</A></STRONG><DD>
<P>
Use James Brown's <STRONG>NewsHound</STRONG> profile searching code. Any sites that do not contain <STRONG>IgnoreProfiles: 1</STRONG> will then be searched for the active profiles. Active profiles are loaded
from the <STRONG>ProfileDir</STRONG> specified in the sitescooper configuration file, or specified using the <STRONG>-profile</STRONG> or <STRONG>-profiles</STRONG>
arguments.

<DT><STRONG><A NAME="item__profile">-profile file.nhp</A></STRONG><DD>
<P>
Limit the run to the site named in the <STRONG>file.nhp</STRONG> argument. Normally all available sites will be downloaded. To limit the run
to 2 or more sites, provide multiple <STRONG>-profile</STRONG> arguments like so:

<P>
<PRE>        -profile ntk.site -profile tbtf.site
</PRE>
<DT><STRONG><A NAME="item__profiles">-profiles file.nhp [...]</A></STRONG><DD>
<P>
Limit the run to multiple sites; an easier way to specify multiple sites
than using the -profile argument for each file.

<DT><STRONG><A NAME="item__name">-name name</A></STRONG><DD>
<P>
When specifying a URL on the command-line, this provides the name that
should be used when installing the site to the Pilot. It acts exactly the
same way as the Name: field in a site file.

<DT><STRONG><A NAME="item__levels">-levels n</A></STRONG><DD>
<P>
When specifying a URL on the command-line, this indicates how many levels a
site has. Not needed when using .site files.

<DT><STRONG><A NAME="item__storyurl">-storyurl regexp</A></STRONG><DD>
<P>
When specifying a URL on the command-line, this indicates the regular
expression which links to stories should conform to. Not needed when using
.site files.

<DT><STRONG><A NAME="item__doc">-doc</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into DOC format, with all the
articles listed in full, one after the other.

<DT><STRONG><A NAME="item__text">-text</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into plain text format, with
all the articles listed in full, one after the other.

<DT><STRONG><A NAME="item__html">-html</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into HTML format, on one big
page, with a table of contents (taken from the site if possible), followed
by all the articles one after another.

<DT><STRONG><A NAME="item__mhtml">-mhtml</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into HTML format, but retain
the multiple-page format. This will create the output in a directory called <STRONG>site_name</STRONG>; in conjunction with the <STRONG>-dump</STRONG> argument, it will output the path of this directory on standard output
before exiting.

<DT><STRONG><A NAME="item__plucker">-plucker</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into Plucker format (see <A
HREF="http://plucker.gnu-designs.com/">http://plucker.gnu-designs.com/</A>
), on one big page. The <CODE>page(s)</CODE> will be displayed with a table
of contents (taken from the site if possible), followed by all the articles
one after another.

<DT><STRONG><A NAME="item__isilo">-isilo</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into iSilo format (see <A
HREF="http://www.isilo.com/">http://www.isilo.com/</A> ), on one big page.
This is the default. The <CODE>page(s)</CODE> will be displayed with a
table of contents (taken from the site if possible), followed by all the
articles one after another.

<DT><STRONG><A NAME="item__misilo">-misilo</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into iSilo format (see <A
HREF="http://www.isilo.com/">http://www.isilo.com/</A> ), with one iSilo
document per site, with each story on a separate page. The iSilo document
will have a table-of-contents page, taken from the site if possible, with
each article on a separate page.

<DT><STRONG><A NAME="item__richreader">-richreader</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into RichReader format using
HTML2Doc.exe (see <A
HREF="http://users.erols.com/arenakm/palm/RichReader.html">http://users.erols.com/arenakm/palm/RichReader.html</A>
). The <CODE>page(s)</CODE> will be displayed with a table of contents
(taken from the site if possible), followed by all the articles one after
another.

<DT><STRONG><A NAME="item__pipe">-pipe fmt command</A></STRONG><DD>
<P>
Convert the <CODE>page(s)</CODE> downloaded into an arbitrary format, using
the command provided. Sitescooper will still rewrite the
<CODE>page(s)</CODE> according to the
<STRONG>fmt</STRONG> argument, which should be one of:

<DL>
<DT><STRONG><A NAME="item_text">text</A></STRONG><DD>
<P>
Plain text format.

<DT><STRONG><A NAME="item_html">html</A></STRONG><DD>
<P>
HTML in one big page.

<DT><STRONG><A NAME="item_mhtml">mhtml</A></STRONG><DD>
<P>
HTML in multiple pages.

</DL>
<P>
The command argument can contain <CODE>__SCOOPFILE__</CODE>, which will be replaced with the filename of the file containing the
rewritten pages in the above format, <CODE>__SYNCFILE__</CODE>, which will be replaced with a suitable filename in the Palm
synchronization folder, and <CODE>__TITLE__</CODE>, which will be replaced by the title of the file (generally a string
containing the date and site name).

<P>
Note that for the <STRONG>-mhtml</STRONG> switch, <CODE>__SCOOPFILE__</CODE> will be replaced with the name of the file containing the table-of-contents
page. It's up to the conversion utility to follow the href links to the
other files in that directory.

<DT><STRONG><A NAME="item__cvtargs">-cvtargs</A></STRONG><DD>
<P>
Arguments for the conversion utility. For example, Plucker will display
images better on some Palms using ``-cvtargs --bpp=2'' or ``-cvtargs
--bpp=4''.

<DT><STRONG><A NAME="item__bw">-bw</A></STRONG><DD>
<P>
Indicate that the target can display only 2-bit images, black and white
only. This is generally the default for iSilo and Plucker.

<DT><STRONG><A NAME="item__color">-color</A></STRONG><DD>
<P>
Indicate that the target can display colour images.

<DT><STRONG><A NAME="item__fixlinks">-fixlinks</A></STRONG><DD>
<P>
Rewrite links to external sites or unscooped pages as underlined text, to
differentiate them from links to scooped pages. This is the default
behaviour for most formats apart from <STRONG>-plucker</STRONG> or <STRONG>-mplucker</STRONG>.

<DT><STRONG><A NAME="item__keeplinks">-keeplinks</A></STRONG><DD>
<P>
Do not rewrite links to external sites or unscooped pages; leave them
pointing outside the current scoop. However, links to other pages that are
included in the current scoop, are rewritten to point to the scooped pages
instead of the source URL. This is the default for Plucker (<STRONG>-plucker</STRONG> or <STRONG>-mplucker</STRONG>
arguments).

<DT><STRONG><A NAME="item__nolinkrewrite">-nolinkrewrite</A></STRONG><DD>
<P>
Do not rewrite links on scooped documents -- leave them exactly as they
are. This includes even links to other scooped pages. See also <STRONG>-keeplinks</STRONG>).

<DT><STRONG><A NAME="item__dump">-dump</A></STRONG><DD>
<P>
Output the <CODE>page(s)</CODE> downloaded directly to stdout in text or
HTML format, instead of writing them to files and converting each one. This
option NO LONGER implies <STRONG>-text</STRONG>, like it used to, so to dump text, use <STRONG>-dump -text</STRONG>.

<DT><STRONG><A NAME="item__dumpprc">-dumpprc</A></STRONG><DD>
<P>
Output the <CODE>page(s)</CODE> downloaded directly to stdout, in converted
format as a PDB file (note: not PRC format!), suitable for installation to
a Palm handheld.

<DT><STRONG><A NAME="item__nowrite">-nowrite</A></STRONG><DD>
<P>
Test mode -- do not write to the cache or already_seen file, instead write
what would be written normally to a directory called new_cache and a
new_already_seen file. This is very handy when writing a new site file.

<DT><STRONG><A NAME="item__badcache">-badcache</A></STRONG><DD>
<P>
Send some HTTP headers to bypass web caching proxy servers. This is
generally useful if a web caching proxy server somewhere between
sitescooper and the target site is returning out-of-date files.

<DT><STRONG><A NAME="item__debug">-debug</A></STRONG><DD>
<P>
Enable debugging output. This output is in addition to the usual progress
messages.

<DT><STRONG><A NAME="item__quiet">-quiet</A></STRONG><DD>
<P>
Process sites quietly, without printing the usual progress messages to
STDERR. Warnings about incorrect site files and system errors will still be
output, however.

<DT><STRONG><A NAME="item__admin">-admin cmd</A></STRONG><DD>
<P>
Perform an administrative command. This is intended to ease the task of
writing scripts which use sitescooper output. The following admin commands
are available:

<DL>
<DT><STRONG><A NAME="item_dump">dump-sites</A></STRONG><DD>
<P>
List the sites which would be scooped on a scooping run, and their URLs.
Instead of scooping any sites, sitescooper will exit after performing this
task. The format is one site per line, with the site file name first, a
tab, the site's URL, a tab, the site name, a tab, and the output filename
that would be generated without path or extension. For example:

<P>
foobar.site	http://www.foobar.com/	Foo&nbsp;Bar	1999_01_01_Foo_Bar



<DT><STRONG><A NAME="item_journal">journal</A></STRONG><DD>
<P>
Write a journal with dumps of the documents as they pass through the
formatting and stripping steps of the scooping process. This is written to
a file called <STRONG>journal</STRONG> in the sitescooper temporary directory.

<DT><STRONG><A NAME="item_import">import-cookies file</A></STRONG><DD>
<P>
Import a Netscape <STRONG>cookies</STRONG> file into sitescooper, so that certain sites which require them, can use
them. For example, the site
<STRONG>economist_full.site</STRONG> requires this. Here's how to import cookies on a UNIX machine:

<P>
sitescooper.pl&nbsp;-admin&nbsp;import-cookies&nbsp;~/.netscape/cookies



<P>
and on Windows:

<P>
perl&nbsp;sitescooper.pl&nbsp;-admin&nbsp;import-cookies
&nbsp;&nbsp;"C:\Program&nbsp;Files\Netscape\Users\Default\cookies.txt"



<P>
Unfortunately, MS Internet Explorer cookies are currently unsupported. If
you wish to write a patch to support them, that'd be great.

</DL>
<DT><STRONG><A NAME="item__noheaders">-noheaders</A></STRONG><DD>
<P>
Do not attach the sitescooper header (URL, site name, and navigation links)
to each page.

<DT><STRONG><A NAME="item__nofooters">-nofooters</A></STRONG><DD>
<P>
Do not attach the sitescooper footer (``copyright retained by original
authors'' blurb) to each page.

<DT><STRONG><A NAME="item__outputtemplate">-outputtemplate file.tmpl</A></STRONG><DD>
<P>
Read the output formatting template from the file <STRONG>file.tmpl</STRONG>. This overrides the settings of the <STRONG>-noheaders</STRONG> and <STRONG>-nofooters</STRONG> flags. See the OUTPUT TEMPLATES section below for details on this.

<DT><STRONG><A NAME="item__fromcache">-fromcache</A></STRONG><DD>
<P>
Do not perform any network access, retrieve everything from the cache or
the shared cache.

<DT><STRONG><A NAME="item__filename">-filename template</A></STRONG><DD>
<P>
Change the format of output filenames. <STRONG>template</STRONG> contains the following keyword strings, which are substituted as follows:

<DL>
<DT><STRONG><A NAME="item_YYYY">YYYY</A></STRONG><DD>
<P>
The current year, in 4-digit format.

<DT><STRONG><A NAME="item_MM">MM</A></STRONG><DD>
<P>
The current month number (from 01 to 12), in 2-digit format.

<DT><STRONG><A NAME="item_Mon">Mon</A></STRONG><DD>
<P>
The current month name (from Jan to Dec), in 3-letter format.

<DT><STRONG><A NAME="item_DD">DD</A></STRONG><DD>
<P>
The current day of the month (from 01 to 31), in 2-digit format.

<DT><STRONG><A NAME="item_Day">Day</A></STRONG><DD>
<P>
The current day of the week (from Sun to Sat), in 3-letter format.

<DT><STRONG><A NAME="item_hh">hh</A></STRONG><DD>
<P>
The current hour (from 00 to 23), in 2-digit format.

<DT><STRONG><A NAME="item_mm">mm</A></STRONG><DD>
<P>
The current minute (from 00 to 59), in 2-digit format.

<DT><STRONG><A NAME="item_Site">Site</A></STRONG><DD>
<P>
The current site's name.

<DT><STRONG><A NAME="item_Section">Section</A></STRONG><DD>
<P>
The section of the current site (now obsolete).

</DL>
<P>
The default filename template is <STRONG>YYYY_MM_DD_Site</STRONG>.

<DT><STRONG><A NAME="item__prctitle">-prctitle template</A></STRONG><DD>
<P>
Change the format of the titles of the resulting PDB files. <STRONG>template</STRONG>
may contain the same keyword strings as <STRONG>-filename</STRONG>.

<P>
The default PDB title template is <STRONG>YYYY-Mon-DD: Site</STRONG>.

<DT><STRONG><A NAME="item__nodates">-nodates</A></STRONG><DD>
<P>
Do not put the date in the installable file's filename. This allows you to
automatically overwrite old files with new ones when you HotSync. It's a
compatibility shortcut for <STRONG>-filename Site -prctitle "Site"</STRONG>.

<DT><STRONG><A NAME="item__preload">-preload preload_method</A></STRONG><DD>
<P>
Preload pages using the given preload method. Currently supported preload
methods are:

<DL>
<DT><STRONG><A NAME="item_lwp">lwp</A></STRONG><DD>
<P>
Use the Perl LWP module to load pages. This is the default, and is
single-threaded; in other words, each page needs to load fully before the
next page can be requested.

<DT><STRONG><A NAME="item_fork">fork[n]</A></STRONG><DD>
<P>
Use a number of subprocesses running LWP requests to load pages. This is
multi-threaded, and several pages can be loaded at once; however you pay in
costs of network bandwidth, CPU time and memory used. The optional <STRONG>n</STRONG>
argument instructs sitescooper to use that number of processes; the default
<STRONG>n</STRONG> is 4. This is only available on UNIX at the moment.

</DL>
<DT><STRONG><A NAME="item__disc">-disc</A></STRONG><DD>
<P>
Disconnect a PPP connection once the scooping has finished. Currently this
code is experimental, and will probably only work on Macintoshes. This is
off by default.

<DT><STRONG><A NAME="item__stdout_to">-stdout-to file</A></STRONG><DD>
<P>
Redirect the output of sitescooper into the named file. This is needed on
Windows NT and 95, where certain combinations of perl and Windows do not
seem to support the &amp;gt; operator.

<DT><STRONG><A NAME="item__keep_tmps">-keep-tmps</A></STRONG><DD>
<P>
Keep temporary files after conversion. Normally the .txt or .html rendition
of a site is deleted after conversion; this option keeps it around.

</DL>
<P>
<HR>
<H1><A NAME="OUTPUT_TEMPLATES">OUTPUT TEMPLATES</A></H1>
<P>
You can control exactly what HTML or text is written to the output file
using the <STRONG>-outputtemplate</STRONG> argument. This argument takes the name of a file, which is read and parsed
to provide replacement templates for sitescooper.

<P>
The file is read as a HTML- or XML-style tagged format; so for example the
template for the main page in HTML format is read from between the
<STRONG>&lt;htmlmainpage&gt;</STRONG> and <STRONG>&lt;/htmlmainpage&gt;</STRONG> tags. The templates that can be defined are as follows:

<DL>
<DT><STRONG><A NAME="item_htmlmainpage">htmlmainpage</A></STRONG><DD>
<P>
The main page, in HTML format; this is used when the -html output format,
or one based on it (such as -plucker or -isilo), is used. It is also used
for the -mhtml format's main (top-level) page.

<DT><STRONG><A NAME="item_htmlsubpage">htmlsubpage</A></STRONG><DD>
<P>
Sub-page, in HTML format; this is used for the -mhtml output format's sub
pages, ie. pages other than the top-level one.

<DT><STRONG><A NAME="item_htmlstory">htmlstory</A></STRONG><DD>
<P>
The snippet of HTML encapsulating each story. This is included for each
piece of snarfed text, in all HTML files.

<DT><STRONG><A NAME="item_textmainpage">textmainpage</A></STRONG><DD>
<P>
The main page, in text format; this is used when the -text output format,
or one based on it (such as -doc), is used.

<DT><STRONG><A NAME="item_textsubpage">textsubpage</A></STRONG><DD>
<P>
Sub-page, in text format; this is currently unused.

<DT><STRONG><A NAME="item_textstory">textstory</A></STRONG><DD>
<P>
The snippet of text encapsulating each story. This is included for each
piece of snarfed text, in all text-format or DOC-format files.

</DL>
<P>
A sample template file is provided in the file <CODE>default_templates.html</CODE>; this may have been installed in the sitescooper install directory,
/usr/share/sitescooper, or /usr/local/share/sitescooper. Note that the
actual templates used are not loaded from this file; instead they are
incorporated inside the sitescooper script, so changing this file will have
no effect.

<P>
<HR>
<H1><A NAME="INSTALLATION">INSTALLATION</A></H1>
<P>
To install, edit the script and change the #! line. You may also need to
(a) change the Pilot install dir if you plan to use the pilot installation
functionality, and (b) edit the other parameters marked with CUSTOMISE in
case they need to be customised for your site. They should be set to
acceptable defaults (unless I forgot to comment out the proxy server lines
I use ;).

<P>
<HR>
<H1><A NAME="EXAMPLES">EXAMPLES</A></H1>
<P>
<PRE>        sitescooper.pl <A HREF="http://www.ntk.net/">http://www.ntk.net/</A>
</PRE>
<P>
To snarf the ever-cutting NTKnow newsletter.

<P>
<PRE>        sitescooper.pl -refresh -html <A HREF="http://www.ntk.net/">http://www.ntk.net/</A>
</PRE>
<P>
To snarf NTKnow, ignoring any previously-read text, and producing HTML
output.

<P>
<PRE>        sitescooper.pl -refresh -html -site site_samples/tech/ntk.site
</PRE>
<P>
To snarf NTKnow using the site file provided with the main distribution,
producing HTML output.

<P>
<HR>
<H1><A NAME="ENVIRONMENT">ENVIRONMENT</A></H1>
<P>
<STRONG>sitescooper</STRONG> makes use of the <CODE>$http_proxy</CODE> environment variable, if it is set.

<P>
<HR>
<H1><A NAME="AUTHOR">AUTHOR</A></H1>
<P>
Justin Mason &lt;jm /at/ jmason.org&gt;



<P>
<HR>
<H1><A NAME="COPYRIGHT">COPYRIGHT</A></H1>
<P>
Copyright (C) 1999-2000 Justin Mason

<P>
This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your option)
any later version.

<P>
This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
more details.

<P>
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc., 59
Temple Place - Suite 330, Boston, MA 02111-1307, USA, or read it on the web
at <A
HREF="http://www.gnu.org/copyleft/gpl.html">http://www.gnu.org/copyleft/gpl.html</A>
.

<P>
<HR>
<H1><A NAME="SCRIPT_CATEGORIES">SCRIPT CATEGORIES</A></H1>
<P>
The CPAN script category for this script is <CODE>Web</CODE>. See <A
HREF="http://www.cpan.org/scripts/">http://www.cpan.org/scripts/</A> .

<P>
<HR>
<H1><A NAME="PREREQUISITES">PREREQUISITES</A></H1>
<P>
<CODE>File::Find</CODE>

<CODE>File::Copy</CODE>

<CODE>File::Path</CODE>

<CODE>FindBin</CODE>

<CODE>Carp</CODE>

<CODE>Cwd</CODE>

<CODE>URI::URL</CODE>

<CODE>LWP::UserAgent</CODE>

<CODE>HTTP::Request::Common</CODE>

<CODE>HTTP::Date</CODE>

<CODE>HTML::Entities</CODE>



<P>
All these can be picked up from CPAN at <A
HREF="http://www.cpan.org/">http://www.cpan.org/</A> . Note that
<CODE>HTML::Entities</CODE> is actually included in one of the previous packages, so you do not need to
install it separately.

<P>
<HR>
<H1><A NAME="COREQUISITES">COREQUISITES</A></H1>
<P>
<CODE>Win32::TieRegistry</CODE> will be used, if running on a Win32 platform, to find the Pilot Desktop
software's installation directory. <CODE>Algorithm::Diff</CODE> to support diffing sites without running an external <STRONG>diff</STRONG> application (this is required on Mac systems).

<P>
<HR>
<H1><A NAME="README">README</A></H1>
<P>
Sitescooper downloads news stories from the web and converts them to Palm
handheld iSilo, DOC or text format for later reading on-the-move. Site
files and full documentation can be found at <A
HREF="http://sitescooper.org/">http://sitescooper.org/</A> .

</BODY>

</HTML>