1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
|
.TH htslib-s3-plugin 7 "12 September 2024" "htslib-1.21" "Bioinformatics tools"
.SH NAME
htslib-s3-plugin \- htslib AWS S3 plugin
.\"
.\" Copyright (C) 2021-2022 Genome Research Ltd.
.\"
.\" Author: Andrew Whitwham <aw7@sanger.ac.uk>
.\"
.\" Permission is hereby granted, free of charge, to any person obtaining a
.\" copy of this software and associated documentation files (the "Software"),
.\" to deal in the Software without restriction, including without limitation
.\" the rights to use, copy, modify, merge, publish, distribute, sublicense,
.\" and/or sell copies of the Software, and to permit persons to whom the
.\" Software is furnished to do so, subject to the following conditions:
.\"
.\" The above copyright notice and this permission notice shall be included in
.\" all copies or substantial portions of the Software.
.\"
.\" THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
.\" IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
.\" FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
.\" THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
.\" LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
.\" FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
.\" DEALINGS IN THE SOFTWARE.
.\"
.
.\" For code blocks and examples (cf groff's Ultrix-specific man macros)
.de EX
. in +\\$1
. nf
. ft CR
..
.de EE
. ft
. fi
. in
..
.SH DESCRIPTION
The S3 plugin allows htslib file functions to communicate with servers that use
the AWS S3 protocol. Files are identified by their bucket and object key in a
URL format e.g.
.B s3://mybucket/path/to/file
With \fIpath/to/file\fR being the object key.
Necessary security information can be provided in as part of the URL, in
environment variables or from configuration files.
The full URL format is:
.B s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH
The elements are:
.TP
.I SCHEME
The protocol used. Defaults to \fIhttps\fR.
.TP
.I ID
The user AWS access key.
.TP
.I SECRET
The secret key for use with the access key.
.TP
.I TOKEN
Token used for temporary security credentials.
.TP
.I BUCKET
AWS S3 bucket.
.TP
.I PATH
Path to the object under the bucket.
.LP
The environment variables below will be used if the user ID is not set.
.TP
.B AWS_ACCESS_KEY_ID
The user AWS access key.
.TP
.B AWS_SECRET_ACCESS_KEY
The secret key for use with the access key.
.TP
.B AWS_DEFAULT_REGION
The region to use. Defaults to
.IR us-east-1 .
.TP
.B AWS_SESSION_TOKEN
Token used for temporary security credentials.
.TP
.B AWS_DEFAULT_PROFILE
The profile to use in \fIcredentials\fR, \fIconfig\fR or \fIs3cfg\fR files.
Defaults to
.IR default .
.TP
.B AWS_PROFILE
Same as above.
.TP
.B AWS_SHARED_CREDENTIALS_FILE
Location of the credentials file. Defaults to
.IR ~/.aws/credentials .
.TP
.B HTS_S3_S3CFG
Location of the s3cfg file. Defaults to
.IR ~/.s3cfg .
.TP
.B HTS_S3_HOST
Sets the host. Defaults to
.IR s3.amazonaws.com .
.TP
.B HTS_S3_V2
If set use signature v2 rather the default v4. This will limit the plugin to
reading only.
.TP
.B HTS_S3_PART_SIZE
Sets the upload part size in Mb, the minimum being 5Mb.
By default the part size starts at 5Mb and expands at regular intervals to
accommodate bigger files (up to 2.5 Tbytes with the current rate).
Using this setting disables the automatic part size expansion.
.TP
.B HTS_S3_ADDRESS_STYLE
Sets the URL style. Options are auto (default), virtual or path.
.LP
In the absence of an ID from the previous two methods the credential/config
files will be used. The default file locations are either
\fI~/.aws/credentials\fR or \fI~/.s3cfg\fR (in that order).
Entries used in aws style credentials file are aws_access_key_id,
aws_secret_access_key, aws_session_token, region, addressing_style and
expiry_time (unofficial, see SHORT-LIVED CREDENTIALS below).
Only the first two are usually needed.
Entries used in s3cmd style config files are access_key, secret_key,
access_token, host_base, bucket_location and host_bucket. Again only the first
two are usually needed. The host_bucket option is only used to set a path-style
URL, see below.
.SH SHORT-LIVED CREDENTIALS
Some cloud identity and access management (IAM) systems can make short-lived
credentials that allow access to resources.
These credentials will expire after a time and need to be renewed to
give continued access.
To enable this, the S3 plugin allows an \fIexpiry_time\fR entry to be set in the
\fI.aws/credentials\fR file.
The value for this entry should be the time when the token expires,
following the format in RFC3339 section 5.6, which takes the form:
2012-04-29T05:20:48Z
That is, year - month - day, the letter "T", hour : minute : second.
The time can be followed by the letter "Z", indicating the UTC timezone,
or an offset from UTC which is a "+" or "-" sign followed by two digits for
the hours offset, ":", and two digits for the minutes.
The S3 plugin will attempt to re-read the credentials file up to 1 minute
before the given expiry time, which means the file needs to be updated with
new credentials before then.
As the exact way of doing this can vary between services and IAM providers,
the S3 plugin expects this to be done by an external user-supplied process.
This may be achieved by running a program that replaces the file as new
credentials become available.
The following script shows how it might be done for AWS instance credentials:
.EX 2
#!/bin/sh
instance='http://169.254.169.254'
tok_url="$instance/latest/api/token"
ttl_hdr='X-aws-ec2-metadata-token-ttl-seconds: 10'
creds_url="$instance/latest/meta-data/iam/security-credentials"
key1='aws_access_key_id = \(rs(.AccessKeyId)\(rsn'
key2='aws_secret_access_key = \(rs(.SecretAccessKey)\(rsn'
key3='aws_session_token = \(rs(.Token)\(rsn'
key4='expiry_time = \(rs(.Expiration)\(rsn'
while true; do
token=`curl -X PUT -H "$ttl_hdr" "$tok_url"`
tok_hdr="X-aws-ec2-metadata-token: $token"
role=`curl -H "$tok_hdr" "$creds_url/"`
expires='now'
( curl -H "$tok_hdr" "$creds_url/$role" \(rs
| jq -r "\(rs"${key1}${key2}${key3}${key4}\(rs"" > credentials.new ) \(rs
&& mv -f credentials.new credentials \(rs
&& expires=`grep expiry_time credentials | cut -d ' ' -f 3-`
if test $? -ne 0 ; then break ; fi
expiry=`date -d "$expires - 3 minutes" '+%s'`
now=`date '+%s'`
test "$expiry" -gt "$now" && sleep $((($expiry - $now) / 2))
sleep 30
done
.EE
Note that the \fIexpiry_time\fR key is currently only supported for the
\fI.aws/credentials\fR file (or the file referred to in the
.B AWS_SHARED_CREDENTIALS_FILE
environment variable).
.SH NOTES
In most cases this plugin transforms the given URL into a virtual host-style
format e.g. \fIhttps://bucket.host/path/to/file\fR. A path-style format is used
where the URL is not DNS compliant or the bucket name contains a dot e.g.
\fIhttps://host/bu.cket/path/to/file\fR.
Path-style can be forced by setting one either HTS_S3_ADDRESS_STYLE,
addressing_style or host_bucket. The first two can be set to \fBpath\fR while
host_bucket must \fBnot\fR include the \fB%(bucket).s\fR string.
.SH "SEE ALSO"
.IR htsfile (1)
.IR samtools (1)
.PP
RFC 3339: <https://www.rfc-editor.org/rfc/rfc3339#section-5.6>
.PP
htslib website: <http://www.htslib.org/>
|
