1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
|
This program is (C) copyright 1995 by Christian Blum. It is published
under the terms of the GNU general public license.
Newsflash retrieves news articles via NNTP from one server and
delivers them to another. It requires at least read-only permissions
on the remote server, and needs to have peer permissions on the local
server. It works well with INN, but should also work with any other
RFC977 compliant news software. Newsflash's highly parallel design
is optimized for throughput, which makes quite a difference to INN's
nntpget and the like.
How to compile
==============
Check config.h. Set BASENAME to whatever directory you'd like to put
newsflash's config files in. If the receiving server is not
"localhost" or its port is not 119, set LOCALHOST and LOCALPORT
appropriately.
Have a short glance at the Makefile; it should be allright for most
systems.
Now type 'make dep' and 'make'.
How to install
==============
ATTENTION! The "serverlist" file format has changed since version 0.97!
Make sure you're allowed to create files in the BINPATH directory set
in the makefile; do 'su' or 'su news' if you have to. Then type 'make
install'. Now create the lib directory given in config.h (BASENAME).
Create a file "serverlist" in it. Have it contain one or more server
entries like this:
server-nickname server-fqdn port grouplist timestamp
where "server-nickname" is some random string to be prepended to any
output related to this spool job, "server-fqdn" is the name or IP of a
news server, "port" is the port it listens on (usually 119 - ignored
for now), "grouplist" ist the absolute pathname (!) of a file
containing newsgroup patterns (see below), and "timestamp" is the time
this server has been checked last (seconds since Jan 1st, 1970,
midnight, GMT), or 0 if it has never been checked before. Lines
starting with whitespace or '#' are ignored. Though you can safely
set the timestamp to zero it is probably not a good idea to do so,
because it will make your first spool from this server very slow and
possibly pretty bulky, depending on the remote server's news spool
size. Use the included printdate program to get the current time in
seconds since the epoch. If you subtract a few days (or multiples of
86400), you get a good initial value. Make sure the serverlist file
is writable when newsflash is started (eg. set the file owner to news,
permissions to 644, and make newsflash setuid news).
Now you need to create the group file(s) referenced to in the
serverlist; it/they must be readable for newsflash, but need not be
writable. Have them contain lines like
misc.misc
comp.os.linux.*
alt.sex*,!alt.sex.watersports,!alt.sex.fetish.*
and the like. Note that you _can_ use lists of wildmat expressions,
and you _should_ wherever possible, because this speeds up the NEWNEWS
command on the remote host, but let the first line or two be single
groups; this makes newsflash start the transfer process more quickly.
Whitespace terminates lines; everything behind a blank or tab
character is treated as a comment, so you can use INN's active file
directly if you like. 'junk' and 'control' are ignored if you haven't
disabled this feature in the config.h file, but these groups are
processed if they match any of the wildmat lists instead of a single
line. See RFC977 for details on the NEWNEWS command, and the wildmat
man page on wildmat expressions. See the 'newsfeeds' man page for
further description of newsgroup patterns.
Make sure your local news server carries all the groups you're
interested in, or retrieved articles will end in junk or /dev/null.
INN: don't edit the active file directly if you don't have to; use
ctlinnd instead (see manpage).
Now go for it!
==============
Fire up your IP connection if you have to, then run 'newsflash -v'.
If it does not seem to do anything useful, you're probably running it
on SunOS (or something similarly bozotic :-). Some SunOS's libraries
appear to have a bug which prevents stdio'ish read requests to sockets
from working; re-compile newsflash with -DSCUMOS added to the OFLAGS
list in the Makefile ('make clean dep all'). Using this parameter
will probably slow down newsflash a slight little bit, so don't use it
if you don't have to, because we want it to be the fastest news
retriever in the west, right? :-)
You should now get some output describing what newsflash is doing.
Note that it is somewhat 'interleaved'; this is a feature, not a bug.
Newsflash uses several processes and several connections in parallel
to speed up the lookup and retrieve process. While newsflash
retrieves articles for one line in the newsgroup list, it already
probes the remote host for the next one, so that the data connection
will hopefully be running continuously, thus increasing its
throughput.
It is a good idea to use more than one line in the serverlist, even if
they all point to the same server, because they will not be processed
sequentially, but in parallel. Be sure to use different group list
files with each line though, so that different connections are probing
for different newsgroups (first). Sorting the lines in another order is
what you should do if you're harvesting articles on several servers;
if all lines refer to the same server, make the group lists disjunct
for best efficiency.
If you set up newsflash with all these hints in mind, it should run
considerably faster than any other non-batching news retriever. The
backdraw is that it also imposes a heavier load on the remote server,
but that's likely to be somebody else's problem. :-)
Once everything is properly set up, you might want to start newsflash
without the -v parameter to make it less wordy. Try 'newsflash -h'
for a list of other useful parameters.
Known bugs
==========
As I've said, it's still beta... or even alpha, how do you know? :-)
Don't over-interpret newsflash's output in verbose mode; it is still
in the beta state, and the texts are partly misleading.
Newsflash does not contain any locking mechanism for access to the
serverlist. Since the files is only shortly opened for write access,
this probably doesn't hurt too much for now.
The program's speed is highly dependent on the user's configuration
skills; newsflash does not do much sanity checking.
A man page for newsflash is underway.
Chris Blum <chris@phil.uni-sb.de>
|
