1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
|
.TH HTCP 1 "October 2005" "htcp" "GridSite Manual"
.SH NAME
.B htcp, htmv, htrm, htls, htll, htmkdir, htfind, htping
\- file transfers and queries via HTTP/HTTPS/SiteCast
.SH SYNOPSIS
.B htcp, htmv
[options] Source-URL[s] Destination-URL
.B htrm, htls, htll, htmkir, htfind
[options] Target-URL[s]
.B htping
[options]
.SH DESCRIPTION
.B htcp
is a client to fetch files or directory listings from remote servers using
HTTP or HTTPS, or to put or delete files or directories onto remote servers
using HTTPS. htcp is similar to scp(1), but uses HTTP/HTTPS rather than ssh
as its transfer protocol. htcp can also use the HTCP protocol to query
HTTP(S) fileservers via SiteCast.
When talking to a fileserver with HTTPS, htcp can run "anonymously", with a
standard X.509 user certificate and key, or with a GSI Proxy. This makes
htcp very useful in Grid environments where many users have certificates
and where jobs and users have access to GSI proxies.
.SH URLs
htcp supports the file:, http: and https: URL schemes as sources and
destinations. If no scheme is given, the URL scheme is assumed to be file:
and relative to the current directory if not an absolute path.
If multiple sources are given during a copy, they will be used in turn and
the destination must be a directory (directories are indicated by a trailing
/) However, source and destination cannot both refer to remote servers.
.SH OPTIONS
.IP "-v/--verbose"
Turn on debugging information. Used once, this option will enable htcp's
messages to stderr. Used twice, will also enable the underlying libcurl
messages.
.IP "--delete"
Instead of copying files, delete all the URLs given on the command line.
Calling the program as htrm has the same effect.
.IP "--list"
Instead of copying files, output lists of files located in the URL-directories
given on the command line. Calling the program as htls has the same effect.
.IP "--long-list"
Instead of copying files, output long listings of files located in the
URL-directories given on the command line. If available, the size in bytes
and modification time of each file is given. Calling the program as
htll has the same effect.
.IP "--mkdir"
Instead of copying files, attempt to create a directory on a remote server
with HTTP PUT. The server must support the convention that PUT to a URL with
a trailing slash means create a directory. No file body is sent. Calling the
program as htmkdir has the same effect.
.IP "--move"
Move/rename files on a single remote server, given the two, absolute URLs
of the remote file names. Server must support HTTP/WebDAV MOVE. Calling the
program as htmv has the same effect.
.IP "--ping"
Query specified multicast groups with the HTCP NOP ("No Operation") code.
SiteCast enabled servers will respond immediately with a NOP reply, and all
of the responses will be listed, with the round trip time in milliseconds.
Any waiting times specified in the --groups option will be ignored. Calling
the program as htping has the same effect.
(--groups must be used for this option to work.)
.IP "--find"
Query specified multicast groups with the HTCP TST code. SiteCast enabled
servers will respond with TST replies if they have the files corresponding
to the given SiteCast target URL(s). All of the transfer URLs returned
will be listed. Waiting times specified in the --groups option will be used
to space out the multicast queries, but the program listens for responses
continuously. Calling the program as htfind has the same effect.
(--groups must be used for this option to work.)
.IP "--groups <IP Groups>"
IP multicast groups to use for SiteCast queries. IP Groups is a comma
separated list of groups, in the format: nnn.nnn.nnn.nnn:port[:ttl[:seconds]]
The IP number and port must be specified. The IP time-to-live, ttl, controls
how many networks the multicast packets may pass through - the default, 1,
limits packets to the local network. Multiple groups may be specified,
separated by commas. If multiple groups are specified, then seconds is the
time to wait before making the next multicast - 1 second is the default.
.IP "--timeout <seconds>"
A request timeout used for multicast ping.
.IP "--anon"
Do not attempt to use X.509 user certificates or GSI proxies to authenticate
to the remote HTTPS server. This means you are "anonymous", but the server's
identity may still be verified and the connection is still encrypted.
.IP "--cert <X.509 cert path> and --key <X.509 key path>"
Path to the PEM-encoded
X.509 or GSI Proxy user certificate and key to use for HTTPS
connections, instead of "anonymous mode." If only one of --key or --cert
is given, then that will be tried for both. If neither is given, then the
following order of precedence is used:
the file name held by the variable X509_USER_PROXY; the file
/tmp/x509up_uID (with Unix UID equal to ID); the file names held by
X509_USER_CERT / X509_USER_KEY; the files ~/.globus/usercert.pem and
~/.globus/userkey.pem (where ~/ is the home directory of the user.)
.IP "--capath <X.509 CA root certs directory or file>"
Path to the PEM-encoded CA root certificates to use when
verifying remote servers' host certificates in HTTPS connections. Ideally
this should be a directory of hash.0 files as described in the OpenSSL
verify(1) man page, but a file may be used instead. If --capath is not
given, the value of the environment variable X509_CERT_DIR will be tried.
If this is not valid, then /etc/grid-security/certificates will be used.
.IP "--no-verify"
Do not use CA root certificates to verify remote servers' host certificates.
This is useful for testing sites before their certificate is set up properly,
but leaves you vulnerable to "man in the middle" attacks by hostile servers
masquerading as your target.
.IP "--grid-http"
Try to use GridHTTP redirection for HTTPS URLs. Compatible servers will perform
authentication and authorization on the HTTPS connection and then redirect
to HTTP for the GET or PUT file transfer. htcp makes the HTTP request using
the GRID_AUTH_PASSCODE single-use passcode obtained via HTTPS. The --grid-http
option will be ignored for directory operations or HTTP URLs. If a redirected
transfer isn't possible, a normal HTTPS data transfer will be attempted.
.IP "--sitecast"
Try to use SiteCast to locate remote files which are to be copied (currently
only for the
.BR fetching
of remote files.) If no location is found via SiteCast, then a direct request
for the given URL is tried. (--groups must be used for this option to work.)
.IP "--domain <SiteCast domain>"
Try to use SiteCast to locate remote files which are to be copied (currently
only for the
.BR fetching
of remote files)
.BR "if the domain component of the URL matches"
the SiteCast domain given.
If no location is found via SiteCast, then a direct request
for the given URL is tried. (--groups must be used for this option to work.)
.SH FILES
.IP /tmp/x509up_uID
Default GSI Proxy file for Unix UID equal to ID.
.IP /etc/grid-security/certificates
Default location for trusted Certification Authority root certificates to use
when checking server certificates.
.IP /tmp/.ca-roots-XXXXXX
Prior to 7.9.8, the underlying curl library did not support the CA root
certificates directory.
If built with an old version of libcurl, htcp will concatenate the
certificates in the CA roots directory into a unique temporary file and use
that.
.SH ENVIRONMENT
.IP X509_CERT_DIR
Holds directory to search for Certification Authority root certificates when
verifying server certificates. (Tried if --capath is not given on the
command line.)
.IP X509_USER_PROXY
Holds file name of a GSI Proxy to use as user certificate. (Tried if --cert or
--key are not given on the command line.)
.IP "X509_USER_CERT and X509_USER_KEY"
Holds file name of X.509 user certificate and key. (Tried if X509_USER_PROXY
is not valid.)
.SH EXIT CODES
0 is returned on complete success. Curl error codes are returned when
reported by the underlying curl library, and CURLE_HTTP_RETURNED_ERROR (22)
is returned when the HTTP(S) server returns a code outside the range 200-299.
The manpage libcurl-errors(3) lists all the curl error codes.
.SH TO DO
Recursive copying. Server-side wildcards. Parallel streams. Better error
recovery.
.SH AUTHOR
Andrew McNab <Andrew.McNab@manchester.ac.uk>
htcp is part of GridSite: http://www.gridsite.org/
.SH "SEE ALSO"
.BR scp(1),
.BR curl(1),
.BR wget(1),
.BR verify(1),
.BR libcurl-errors(3)
|
