1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
|
=head1 NAME
nbdkit-gcs-plugin - expose data in Google Cloud Storage (GCS) buckets
as block device
=head1 SYNOPSIS
nbdkit gcs [json-credentials=FILE]
[size=NN object-size=NN]
bucket=BUCKET key=STRING
=head1 DESCRIPTION
C<nbdkit-gcs-plugin> is a plugin for L<nbdkit(1)> which lets you open
objects stored in Google Cloud Storage (GCS) as disk images.
This plugin uses the Python Google Cloud Storage client library
(L<google-cloud-storage|https://pypi.org/project/google-cloud-storage/>).
It requires Python E<ge> 3.7.
=head1 EXAMPLES
nbdkit gcs json-credentials=/path/to-my-json-credentials.json \
bucket=MY-BUCKET key=disk.img
Provides a read only block device holding the data contained in the
F<disk.img> object.
nbdkit gcs json-credentials=/path/to-my-json-credentials.json \
size=50G object-size=128k \
bucket=MY-BUCKET key=disk
Provides a read-write block device with size 50G, whose contents are
stored across multiple in objects of size 128k, prefixed with F<disk/>
=head1 PARAMETERS
=over 4
=item B<json-credentials=>FILE
Path to a JSON file with Google Service account keys. Application Default
Credentials will be used if C<json-credentials> is not specified (see
L</CREDENTIALS> below).
=item B<bucket=>BUCKET
The bucket containing the object(s). This parameter is required.
=item B<key=>STRING
The object name (if C<size> is not specified) or object prefix (if
C<size> is specified) to use within the bucket. This parameter is
required.
=item B<size=>SIZE
=item B<object-size=>SIZE
These two parameters must always be specified together. If set, data
will be split into blocks of C<object-size> and stored as separate
objects. The block device will report a total size of C<size> and be
writeable and trim-able.
Object names will have the form F<key/%16x>, where C<%16x> is the
16-digit hexadecimal block number. If there are existing objects under
the prefix that do not have the expected size, the plugin will crash.
=back
=head1 PERFORMANCE CONSIDERATIONS
It is highly recommended that clients do their utmost to issue
requests that exactly match the object size: Smaller write requests
will incur a performance penalty due to the need for read-modify-write
cycles (thus also incurring latency from two network
round-trips). Larger read and write requests will incur a performance
penalty because of sequential execution.
The L<nbdkit-blocksize-filter(1)> can be used to alleviate the impact
of requests larger than the object size, but does not help if the
client issues requests smaller than the block size.
The L<nbdkit-stats-filter(1)> can be used to investigate what block
sizes and alignments are used by the client.
When connecting through the Linux kernel's NBD module, consider
setting C</sys/block/nbdE<lt>XE<gt>/queue/max_sectors_kb> to match the
object size.
=head1 CREDENTIALS
Refer to the following web pages:
=over 4
=item Google Service account keys
L<https://developers.google.com/workspace/guides/create-credentials#service-account>
=item Application Default Credentials
L<https://cloud.google.com/docs/authentication/application-default-credentials>
=back
=head1 FILES
=over 4
=item F<$plugindir/nbdkit-gcs-plugin>
The plugin.
Use C<nbdkit --dump-config> to find the location of C<$plugindir>.
=item F<$HOME/.config/gcloud/application_default_credentials.json>
Application Default Credentials.
=back
=head1 ENVIRONMENT VARIABLES
=over 4
=item C<GOOGLE_APPLICATION_CREDENTIALS>
You can use the C<GOOGLE_APPLICATION_CREDENTIALS> environment variable
to provide the location of a credential JSON file.
=back
=head1 COMPARISON TO S3 PLUGIN
You can use the C<S3> plugin to access Google Cloud Storage in
interoperability mode. C<read> and C<write> work perfectly fine,
while C<trim> and C<zero> operations don't work in the general case:
trimmed or zeroed-out blocks cannot be deleted from the storage using
the C<S3> plugin.
=head1 VERSION
C<nbdkit-gcs-plugin> first appeared in nbdkit 1.38.
=head1 SEE ALSO
L<nbdkit(1)>,
L<nbdkit-plugin(3)>,
L<nbdkit-python-plugin(3)>,
L<nbdkit-S3-plugin(3)>,
L<https://pypi.org/project/google-cloud-storage/>,
L<https://developers.google.com/workspace/guides/create-credentials#service-account>,
L<https://cloud.google.com/docs/authentication/application-default-credentials>.
=head1 AUTHORS
Mykola Ivanets
=head1 COPYRIGHT
Copyright Infrascale Inc.
|
