File: SETUP.en

package info (click to toggle)
gfarm 2.3.2-1
  • links: PTS
  • area: main
  • in suites: squeeze
  • size: 10,972 kB
  • ctags: 10,894
  • sloc: ansic: 84,510; sh: 13,707; java: 6,866; makefile: 2,286; python: 771; perl: 325; sql: 130; xml: 50; asm: 37; csh: 2
file content (443 lines) | stat: -rw-r--r-- 14,462 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
		Gfarm Filesystem Setup Manual

About this document
===================

This document describes how to configure the Gfarm filesystem.

For users who would like to configure a Gfarm filesystem by a non
privileged user, refer to SETUP.private.en.

Please install the Gfarm software before using this manual,
By using either INSTALL.en, for installation from the source code,
or INSTALL.RPM.en, for installation from the RPM binary packages.

If problems occur, please refer to the Trouble Shooting section
in Gfarm-FAQ.en.

Overview
========

To introduce the Gfarm filesystem, configure a metadata server, and
filesystem nodes have to be configured, and register users,
administrators and root users.

Quick Start
===========

Configuration of a Gfarm metadata server (MDS)
 # config-gfarm -A <admin_name>
 # useradd -c "Gfarm gfsd" _gfarmfs
 # su _gfarmfs
 $ gfkey -f -p 31536000

Configuration of a Gfarm filesystem node
 # useradd -c "Gfarm gfsd" _gfarmfs
 Copy ~_gfarmfs/.gfarm_shared_key from the MDS
 Copy /etc/gfarm2.conf from the MDS
 # config-gfsd <spool_directory>
 Ask administrators to register the filesystem node
 # /etc/init.d/gfsd start

Configuration of a Gfarm client node
 Copy /etc/gfarm2.conf from the MDS
 $ gfkey -f -p 31536000
 Copy a shared secret key ~/.gfarm_shared_key to the MDS, all file
 system nodes

Registration of Users
 $ gfuser -c username realname homedir gsi_dn
 $ gfmkdir homedir
 $ gfchown username homedir

Registration of Administrators
 $ gfgroup -m gfarmadm user1 user2 ...

Registration of Root users
 $ gfgroup -m gfarmroot user1 user2 ...

Configuration of a Gfarm metadata server (MDS)
==============================================

To set up a Gfarm file system, the root privilege is required.

To begin with, the following information is required:

- a global username of an administrator	[-A]
- a subject DN of an administrator	[-D]	(for GSI auth only)

  The global username is a unique user name used in Gfarm file system
  to share files among several administrative domains having different
  account names.

  When you log on as a normal user and change to the root privilege by
  'su', the normal user account name is used as a global username of
  the administrator by default.

  In case of the shared secret authentication, the default mapping
  from a local account to a global username is identical.  A user
  having the same account name as the global username is an
  administrator by default.  This mapping can be managed by a user
  mapfile that is specified by 'local_user_map' directive in
  gfarm2.conf.

  In case of GSI, a user having the specified subject DN is an
  administrator for a Gfarm file system.

Run 'config-gfarm' to configure a Gfarm filesystem.  First, make
sure of the default setting with the -t option.  With the -t option,
nothing is really executed except for the display of the settings for
configuration.

	# config-gfarm -t -A <global_admin_name> -a gsi_auth -D <Subject DN>

The -a and -D options are only required for GSI authentication.  The
-a option specifies the authentication method in Gfarm file system.
The default is shared secret authentication (sharedsecret).  For
details, refer to a manual page of gfarm2.conf(5) and Gfarm-FAQ.en.

You can modify any default parameter with the option shown in [ ].

When you have confirmed the parameter, run 'config-gfarm' by the root
user without the -t option.

config-gfarm sets up the backend database, creates a Gfarm
configuration file, /etc/gfarm2.conf and /etc/gfmd.conf, and executes
the Gfarm metadata server gfmd.

  'prefix' is used to configure several Gfarm filesystems. It
  generates all configuration files under the specified directory.

When you use the shared secret authentication between gfsd and gfmd,
it is necessary to create a '_gfarmfs' user and a shared secret key.

	# useradd -c "Gfarm gfsd" _gfarmfs

  We recommend to create a home directory in NFS if available.

Create a shared secret key '~/.gfarm_shared_key' for the _gfarmfs user
in the home directory.

	# su _gfarmfs
	$ gfkey -f -p 31536000

  This example sets up an expiration date of about one year (31536000
  seconds).

You can check whether the metadata server gfmd is running or not by
the following command execution;

	# /etc/init.d/gfmd status

For the automatic startup during the boot process, chkconfig command
is useful in Red Hat Linux.

	# chkconfig --add gfmd
	# chkconfig --add gfarm-pgsql
	(or # chkconfig --add gfarm-slapd)

For debian, use update-rc.d instead of chkconfig.

	# update-rc.d gfmd defaults
	# update-rc.d gfarm-pgsql defaults
	(or # update-rc.d gfarm-slapd defaults)

If you would like to remove all files created by config-gfarm, first
stop gfmd and backend database server;

	# /etc/init.d/gfmd stop
	(or # killall gfmd)
	# /etc/init.d/gfarm-pgsql stop
	(or # /etc/init.d/gfarm-slapd stop)

Then, remove files and directories as follows;

	# rm -f /etc/gfarm2.conf /etc/gfmd.conf
	# rm -f /etc/init.d/gfarm-* /etc/init.d/gfmd
	# rm -rf /var/gfarm-*

o Firewall configuration

The Gfarm metadata server should be able to accept TCP connections at
a port that are specified by the -m option of the config-gfarm
command.

Configuration of a Gfarm filesystem node
========================================

To set up a Gfarm filesystem node, the root privilege is required.

Create a '_gfarmfs' user if not exist.  This user is a system account
for a gfsd I/O daemon.

	# useradd -c "Gfarm gfsd" _gfarmfs

In case of the shared secret authentication, copy the shared secret
key .gfarm_shared_key in the _gfarmfs's directory from the metadata
server if the home directory is not shared.

  Note that the permission of .gfarm_shared_key should be '0600'
  and the owner should be '_gfarmfs'.

In case of the GSI authentication, obtain a service certificate for
the gfsd.

  Copy the certificate to /etc/grid-security/gfsd/gfsd{cert,key}.pem
  and change the owner to '_gfarmfs'.  The permission of gfsdkey.pem
  should be 0400.

Copy /etc/gfarm2.conf from the metadata server that is created by
'config-gfarm'.

Run 'config-gfsd' to set up a spool directory in a local filesystem
on the filesystem node, and register it in the metadata server.
First, make sure of the default setting with the -t option.  With the -t
option, nothing is really executed except for display of the settings for
configuration.

	# config-gfsd -t /var/gfarm

  You can modify any default parameter by using the option shown in [ ].

  The last argument of config-gfsd is a spool directory, which stores
  physical files in Gfarm file system.  Note that the spool directory
  should be a non-shared area among filesystem nodes.

When you have confirmed the parameter, run 'config-gfsd' by the root
user without the -t option.

	# config-gfsd /var/gfarm

config-gfsd displays a command line of 'gfhost' that would be executed
by an administrator in Gfarm file system.

Ask the administrator to execute the displayed text.

  The administrator should execute the gfhost command as a global user
  specified by config-gfarm command on the metadata server host.  In
  case of the shared secret authentication, it is a local account of
  the same name or mapped by a user mapfile.  In case of GSI, use a
  user certificate specified by the -D option of config-gfarm.

	$ /usr/bin/gfhost -c -a i386-fedora5-linux -p 600 -n 2 node1.example.org

  In case of GSI, add a service certificate of the gfsd in
  /etc/grid-security/grid-mapfile of a metadata server in the
  following format.  '@host@' is a fixed string.  It cannot be
  changed.

	"Subject DN of gfsd server cert" @host@ FQDN

Start gfsd by the root user.

	# /etc/init.d/gfsd start

You can check whether the gfsd is running or not by the following
command execution;

	# /etc/init.d/gfmd status

For the automatic startup during the boot process, chkconfig command
is useful in Red Hat Linux.

	# chkconfig --add gfsd

For debian, use update-rc.d instead of chkconfig.

	# update-rc.d gfsd defaults

If you would like to remove all files created by config-gfsd, first
stop gfsd;

	# /etc/init.d/gfsd stop

Then, remove files and directories as follows;

	# rm -f /etc/gfarm2.conf /etc/init.d/gfsd
	# rm -rf <spool_directory>

o Firewall configuration

Filesystem nodes should be able to accept TCP connections and
UDP packet reception and transmission at the port that is
specified by the -p option of the config-gfsd command.

Also, it requires the same settings as those of the client nodes.

Configuration of a Gfarm client node
====================================

This chapter describes configuration for a client node.

o Configuration of a client node

Copy /etc/gfarm2.conf from the metadata server to the client node.

  This setting can be substituted for copying to ~/.gfarm2rc.

In case of the shared secret authentication, if user's home directory
is shared among metadata server and file system nodes, there is
nothing to do.  If not, .gfarm_shared_key needs to be create and
copied among every nodes.

In case of GSI, obtain the user certificate, and ask to add your entry
into grid-mapfile of every file system node and a metadata server.

o Firewall configuration

Client nodes should be able to initiate TCP connections to the metadata
server, at the port specified by the -m option of the config-gfarm command.
Furthermore, they should be able to initiate TCP connections and
should be able to send/receive UDP packets to filesystem nodes, from the
port specified by the -p option of the config-gfsd command.

Testing of the Gfarm filesystem
===============================

You can check whether the Gfarm filesystem works or not using any client,
since it can be accessed (or shared) by every client node.

o gfls - Directory listing

'gfls' lists the contents of a directory.

	% gfls -la
	drwxrwxr-x gfarmadm gfarmadm          4 Aug 23 06:33 .
	drwxrwxr-x gfarmadm gfarmadm          4 Aug 23 06:33 ..
	drwxr-xr-x tatebe   gfarmadm          0 Aug 22 16:08 tmp

o gfhost - Filesystem node information

'gfhost -M' displays the information for filesystem nodes registered
with the metadata server.

	% gfhost -M
	i386-fedora3-linux 2 linux-1.example.com 600 0
	i386-fedora3-linux 2 linux-2.example.com 600 0
	i386-fedora3-linux 2 linux-3.example.com 600 0
	i386-redhat8.0-linux 1 linux-4.example.com 600 0
	sparc-sun-solaris8 1 solaris-1.example.com 600 0
	sparc-sun-solaris8 1 solaris-2.example.com 600 0
	...

'gfhost -l' displays the status of the filesystem nodes.

	% gfhost -l
	0.01/0.03/0.03 s i386-fedora3-linux 2 linux-1.example.com 600 0(10.0.0.1)
	0.00/0.00/0.00 s i386-fedora3-linux 2 linux-2.example.com 600 0(10.0.0.2)
	-.--/-.--/-.-- - i386-fedora3-linux 2 linux-3.example.com 600 0(10.0.0.3)
	0.00/0.02/0.00 x i386-redhat8.0-linux 1 linux-4.example.com 600 0(10.0.0.4)
	0.10/0.00/0.00 G sparc-sun-solaris8 1 solaris-1.example.com 600 0(10.0.1.1)
	x.xx/x.xx/x.xx - sparc-sun-solaris8 1 solaris-2.example.com 600 0(10.0.1.2)
	...

The second field shows the status of authentication with the filesystem
node.  's', 'g', and 'G' show successful authentication, while 'x'
shows an authentication failure.

'-.--/-.--/-.--' in the first field shows that gfsd has not executed
correctly, and 'x.xx/x.xx/x.xx' shows the filesystem node is probably
down.

o gfdf - display storage capacity

'gfdf' displays capacity of each file system node and the total
capacity.

	% gfdf
	   1K-blocks        Used       Avail Capacity Host
	  1824815008      213068  1824601940     0%   linux-1.example.com
	  6835798016    71836400  6763961616     1%   linux-2.example.com
	  1669232308    44224088  1625008220     3%   solaris-1.example.com
	---------------------------------------------
	 10329845332   116273556 10213571776     1%

For details of the above Gfarm commands, refer to the respective man page.

User registration
=================

Every user needs to be registered to access Gfarm file system by a
Gfarm administrator.  Besides, a Gfarm root user needs to create his
home directory.

  A Gfarm administrator specified by config-gfarm is automatically
  registered during config-gfarm.

  Gfarm administrators are members of gfarmadm group.

  Gfarm root users are members of gfarmroot group.

For user registration, execute gfuser command;

	% gfuser -c global_username realname homedir gsi_dn

'global_username' is a global user name in Gfarm file system.
'realname' is a name to identify the user.  'homedir' is a home
directory in Gfarm.  'gsi_dn' is a subject DN of a user certificate
used in GSI authentication.

  'homedir' is not used for now.

When 'realname' includes a space, it is necessary to quote.  The
following example registers a global user 'foo';

	% gfuser -c foo "foo bar" /home/foo ""

Next, it is necessary to create a home directory for the user.  This
should be done by a Gfarm root user.  In the case of the above
example,

	% gfmkdir /home
	% gfmkdir /home/foo
	% gfchown foo /home/foo

Registration of Gfarm administrators and root users
===================================================

Gfarm administrators manages file system nodes, users, and groups.
Gfarm root users manages access control in Gfarm file system.

Gfarm administrators are members of the gfarmadm group.  The following
command displays the current administrators.

	% gfgroup -l gfarmadm

To change the members, use 'gfgroup -m' command.

	% gfgroup -m gfarmadm user1 user2 ...

Note that 'gfgroup -m' replaces members with specified members.  The
old members are overwritten.

Gfarm root users are members of the gfarmroot group.

	% gfgroup -m gfarmroot user1 user2 ...

Further examples of advanced functionality
==========================================

o File replica creation

Each file in the Gfarm filesystem can have several file copies that
can be stored on two and more filesystem nodes.

Multiple file copies of the same file enables access to the file even
when one of the filesystem nodes is down.  Moreover, it enables
prevention of access performance deterioration by allowing access to
different file copies.

The 'gfwhere' command displays the location of file copies, or a replica
catalog, of the specified files.

	% gfwhere .bashrc
	linux-1.example.com

The 'gfrep' command creates file copies.

	% gfrep -N 2 .bashrc
	% gfwhere .bashrc
	linux-1.example.com linux-2.example.com

In this case, '.bashrc' has two copies; one is stored on
linux-1.example.com and the other is stored on linux-2.example.com.