1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
|
puf 0.9 (beta)
==============
Copyright (C) 2000,2001 by Oswald Buddenhagen <puf@ossi.cjb.net>
based on puf 0.1.x (C) 1999,2000 by Anders Gavare <gavare@hotmail.com>
This program is FREE software in the sense of the GPL. See COPYING for
details.
Project homepage: http://ossi.cjb.net/sw/puf.html
What is puf?
------------
puf is a "parallel url fetcher" for UN*X systems. It is has some
similarities to GNU wget. The most notable difference from wget
is that puf downloads files in parallel.
NOTE: If you are planing on using puf to do massive downloads to a
system where multiple users are working, you might want to tell people
what you are doing since puf can use up a lot of resources (mostly
network bandwidth, but also memory if left running for too long).
How to compile and install:
---------------------------
First run "./configure", then "make". Then run "make install" as root.
Tested plattforms include Linux, Solaris, OpenBSD, Digital UNIX 4.0
and even CygWin.
Ultrix is known not to work.
If you manage to compile puf on a platform which is not specified
herein, then I'd appreciate if you email me about it.
Usage:
------
Just run puf without any parameters and you should get the pretty
straight forward syntax printed to stdout. In general, the syntax
looks like this:
puf [options] url [...]
I will not list all the options here. To get the list of options,
simply run "puf -h".
urls may be "real" urls, like this: http://some.host.org/path/file
or partial, like: www.blah.com (http:// is automatically prepended)
(At the time of writing, only the http protocol is recognized.)
There are options available for recusive fetching and for fetching
images and frames associated with the specified url.
When running puf, you'll see a status line which looks something like
the following example:
URLs Connections Bytes Time Kbyte/s
done+ fail/ total errs cur/max done/total pass left cur/avg
1+ 0/ 1 0 0/20 7466/7466 00:00 00:00 364/364
The first numbers are the number of files downloaded, the number of
files which cannot be retrieved and total number of files to download.
Errs is the total number of network and file errors encountered.
Next comes the number of current active connections. puf tries to use
the maximum number as much as possible.
Number of bytes downloaded and total bytes go a bit up and down, and
you shouldn't trust them too much. :-) This is because puf doesn't
know beforehand how large the files are. Another problem is that some
servers don't send the total size of documents. The size of dynamically
created documents (CGI etc.) are obviously also always of unknown size.
The elapsed time should be correct, but the time left is calculated
using a weird speed calculation and the number of bytes left, which
might be unknown. Therefore the time left cannot be trusted unless you
have a very stable connection (in terms of speed) to the server(s) to
which you are connected and all downloads are already running (if
there are still urls in the queue, then the numbers will grow later).
Special features:
-----------------
Parallel fetching:
This is the main point with puf. This is also the feature which
might make it a bit unstable. Bringing a unix-system down by
using up memory resources is usually refered to as "thrashing",
but I don't know what this is called (using up the network
resources). Don't set the number of open network connections
too high if you don't want to risk bringing your system down.
Recursion:
This makes puf act pretty much like the famous "wget" utility.
Combined with parallelism, this is a very powerful feature.
File handle deficiency management:
On systems where the kernel hasn't been compiled to allow a
high number of open file handles (or when harsh per-user
limits are set), this will allow more files to be written to
in parallel. (This is not good performance-wise, though.)
Bugs: (text by Anders Gavare; maybe obsolete by now)
----------------------------------------------------
I have experienced segmentation faults both under Linux and OpenBSD
(they are much more common under Linux, though). I've managed to lock
up both Linux and OpenBSD. The OpenBSD kernel was maybe modified a bit
too much, since the `GENERIC' OpenBSD kernel didn't hang. Some people
told me that this can be because my RAM is damaged... too bad since I
don't want to spend money buying new RAM... :-(
The Linux problems may have been resolved in kernel version 2.2.3, but
I'm not sure, since I haven't had any opportunity to verify this
myself.
If you find any bugs (even if you think I already know about them),
don't hesitate to email me.
|
