1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
|
<!DOCTYPE doctype PUBLIC "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR"
content="Mozilla/4.7 [en] (Win98; U) [Netscape]">
</head>
<body>
<b><font size="+2"><font color="#000000">Synchronizing</font> Disks for
HA</font></b>
<p><b><font size="+1">What is this document?</font></b> <br>
<i>This is only ONE way to keeps nodes in synchronization for a high
availability cluster. It suits our needs and may be a good</i> <br>
<i>starting block for others so I'm sharing it. Your mileage may
vary.</i> </p>
<p><i>We used this on a web server to update ~1GB worth of documents in
sync every 10 minutes. Depending on how quickly your files change,
you may be able to update more data, more frequently. I would not
recommend using this without a private fast ethernet channel, i.e. an
extra NIC in each node connected via a crossover cable.<br>
</i></p>
<p><i>To determine whether this method would be appropriate for you,
you may wish to do this:</i> </p>
<ol>
<li> <i>Set the minimum update rate you find acceptable (say, 3
minutes).</i></li>
<li> <i>Conservatively approximate the amount of data you would
expect to change in 3 minutes (200MB)</i></li>
<li> <i>Assuming you can count on 6MB/sec from your fast ethernet,
multiply this by 1/3 of your update time. Why? You need to
account for rsync (relatively slow executing) to determine what needs
updating and disk write time. In our example, this would give us
360MB worth of updates per cycle.</i></li>
</ol>
<i>Of course, you can always do what I did - set it up and see if it
updates fast enough. The scripts I've provided will notify you if
it kicks off again before the previous one <br>
finished.......</i>
<p><i>BTW, for lots of data, or rapidly changing data, you'll want a
shared disk or use <a
href="http://www.complang.tuwien.ac.at/reisner/drbd/">drbd</a>.
However, we find rsync is still very useful to keep config files,
passwd files, etc. consistent and use it in conjuction with drbd.<br>
</i> </p>
<p><i><font color="#000000">One last item: rsync with the
--delete option is </font><font color="#ff0000">dangerous</font><font
color="#000000">! Make sure your command options point to the
proper destination! Test this on non-critical data. And
then test it again.</font></i> <br>
</p>
<p><b><font size="+1">What you'll need:</font></b> </p>
<p> 1. OpenSSH <br>
You can get it at: <a href="http://www.openssh.com/portable.html">http://www.openssh.com/portable.html</a><br>
2. A copy of rsync <br>
Found at <a href="http://rsync.samba.org/ftp/rsync/binaries/">http://rsync.samba.org/ftp/rsync/binaries/</a>
-OR- <a
href="http://rsync.samba.org/ftp/rsync/rsync-2.3.1.tar.gz">http://rsync.samba.org/ftp/rsync</a><br>
3. Cron and stuff to mirror. </p>
<p> <i><font size="-1">In actuality, you don't even
need SSH. You could use rsh instead, <b><font color="#ff0000">if
your security needs permit it.</font></b></font></i> </p>
<p><b><font size="+1">Installing SSH</font></b> </p>
<p><font color="#ff6600">START HERE for source distribution:</font> <br>
Untar your openssh distribution. You might want to read the
INSTALL file or HOWTO, but most can get by with the following: <br>
<b><tt>./configure</tt></b> <br>
<b><tt> make</tt></b> <br>
<b><tt> make install</tt></b> </p>
<p><font color="#ff6600">START HERE for rpm distribution after running
"rpm -ivh openssh_<version#>.rpm":</font> <br>
Once this is done, make sure sshd is started on bootup. This
could be done via init.d scripts or by placing (if ssh is installed to
the default location) "/usr/local/sbin/sshd" in you /etc/rc.d/rc.local
file. Type this in now to start it.<br>
<br>
<span style="color: rgb(255, 102, 0);"><span style="font-weight: bold;">NOTE:
YOU SHOULD READ THE OPENSSH DOCUMENTATION.
THE FOLLOWING INSTRUCTIONS WILL INSTRUCT YOU<br>
ON ONE WAY TO SET UP AUTO-AUTHENTICATION BETWEEN YOUR CLUSTER
NODES....BUT YOU SHOULD UNDERSTAND WHAT<br>
YOU ARE DOING!!!<br>
</span></span> </p>
<p>Make sure that /usr/local/bin is in your path and type:
"ssh-keygen -d" This will create your ssh
"key". Do not enter a passphrase (hit return). </p>
In your ~/.ssh directory, there will be two files: "id_dsa" and
"id_dsa.pub". "id_dsa" is your private key, "id_dsa.pub" is your
public key (please refer to your<br>
ssh documentation for explanations). Run the following commands:<br>
<div style="margin-left: 40px;"><span style="font-weight: bold;">cp
id_dsa.pub authorized_keys2</span><br>
<span style="font-weight: bold;"></span></div>
<div style="margin-left: 40px;"><span style="font-weight: bold;">chmod
400 authorized_keys2</span><br>
</div>
Now copy the entire .ssh directory (preserving permissions) to your
home directory on the other node in the cluster. <br>
<br>
You should be all set. Try it out by typing on node A: "ssh
nodeB". You should be logged in to nodeB without having to type
anything (except "yes" to accept<br>
the host key the very first time you connect....)<br>
<p><b><font size="+1">Installing Rsync</font></b> </p>
<p>Well, I'm not really going to tell you how to do this. I just
used the rpm. If that's not possible for you, I'm sure the good
folks at samba will have a nice README.Install for you to follow.
However, the binary link in the "What you'll need" section has binaries
for just about all flavors. Here's the link for the
rpm: <a
href="http://rsync.samba.org/ftp/rsync/binaries/redhat">http://rsync.samba.org/ftp/rsync/binaries/redhat</a> </p>
<p><b><font size="+1">Determining your Rsync command</font></b> </p>
<p>For our example, let's say you have a web server cluster. As a
result, you need the directory tree "/html" to be current on both
nodes. Assuming node A is the master, I would use the command
from node A: </p>
<pre><i> <tt>rsync --rsh=/usr/local/bin/ssh -naurvl --delete /html/ localnetB:/html</tt></i></pre>
Let's note a few things. First, since this will be used with
cron, you want to be sure that you use the full path for the ssh
executable (and the rsync executable for that matter). Also, note
the "/html/" syntax. This last "/" is necessary - otherwise
you'll have the tree "/html/html" on your slave. Lastly, I used
the "-n" option. This is for a dry run. You want to do this
to make sure everything is copied/deleted as you would expect it.
When you put this in your crontab file, you'll leave off the "-n"
option. Similarly, the "-v" verbosity option is only for this
test. In your crontab entry, it will be replaced with the "-q"
option for quiet. Test the command now and make sure it does what
you want - the "--delete" option can be dangerous!
<p><b><font size="+1">Create your sync script and crontab entry</font></b> </p>
<p>You now want to create the script which will run rsync every X
minutes via cron. Our solution has the same script running on
both nodes, but it checks whether it is running on the current primary
(the one holding the services) or not. If it's running on the
secondary, it quits immediately. I use the following perl script,
called "synch_all.pl": </p>
<center>
<table border="1" bgcolor="#ffffcc" nosave="">
<tbody>
<tr nosave="">
<td nosave=""><i><tt>#!/usr/bin/perl<br>
<br>
</tt></i>use strict;<br>
use diagnostics;<br>
<br>
use lib "/root/scripts";<br>
use whohostlib;<br>
use EnvConfig;<br>
use Mail::Sendmail;<br>
<br>
#Take care of two-way syncing case during shutdown of one server.
Skips the first try to sync after failover.<br>
if ( -e "/var/lock/subsys/mirrorstop" ) {<br>
unlink "/var/lock/subsys/mirrorstop";<br>
exit;<br>
}<br>
<br>
# Only sync if serving IP<br>
my $dirname = "";<br>
my $filename = "";<br>
<br>
#Get server that this is running on...<br>
my $me = "";<br>
my $other = "";<br>
($me,$other) = whohostlib::whohost;<br>
<br>
if (&whohostlib::whostatus() == 1)<br>
{<br>
#I'm serving! You want fries with that?<br>
<br>
#Only sync if other node is running heartbeat:<br>
my $tst = `$EnvConfig::sshpath local$other
/etc/rc.d/init.d/heartbeat status`;<br>
# For 0.4.9.2 and earlier use --->
if ($tst =~ /running.../) {<br>
# For 0.4.9 "beta" versions...<br>
if ($tst =~ /heartbeat OK/) {<br>
#make sure previous mirror has
completed...<br>
<br>
#To avoid hard link probs over
partitions, created /var/lock/subsys/.DONOTREMOVE file<br>
if
(link("/var/lock/subsys/.DONOTREMOVE", "/var/lock/subsys/mirror")) { <br>
# /home/
- Home Directories (all users)<br>
#
/root/scripts/ - Root scripts/system scripts - like where we put this
script...<br>
#
/var/spool/cron/ - System crontabs<br>
my @dirlist = ( "/home/", "/root/scripts/",
"/var/spool/cron/" );<br>
foreach $dirname (@dirlist) {<br>
system "/usr/bin/rsync
--rsh=$EnvConfig::sshpath $EnvConfig::rsyncoptions $dirname
local$other:$dirname";<br>
}<br>
<br>
#
/etc/password - System Password File<br>
#
/etc/shadow - Actual Encrypted passwords.<br>
#
/etc/group - System Group File<br>
#
/etc/ld.so.conf - System Linked Libraries.<br>
#
/etc/shells - Valid Login Shells<br>
my @filelist = ( "/etc/passwd", "/etc/shadow",
"/etc/group", "/etc/ld.so.conf", "/etc/shells" );<br>
foreach $filename (@filelist) {<br>
system "/usr/bin/rsync
--rsh=$EnvConfig::sshpath $EnvConfig::rsyncoptions $filename
local$other:$filename";<br>
}<br>
<br>
# release
rsync process<br>
unlink
"/var/lock/subsys/mirror";<br>
} else {<br>
my $ddd =
`date`;<br>
my $subject =
"Next RSYNC process starting before previous has completed!\n";<br>
my $message =
$subject.$ddd;<br>
<br>
sendmail
('smtp' => $EnvConfig::smtphost,<br>
'To' => 'admin@domain.com,admin2@domain.com',<br>
'From' => 'cluster@domain.com',<br>
'Reply-To' => 'cluster@domain.com',<br>
'Subject' => $subject,<br>
'Message' => $message)<br>
or warn "Next
RSYNC process starting before previous has completed (Couldn't send
email)";<br>
<br>
#Try to get to
known state by killing all synch procs and removing link...<br>
unlink
"/var/lock/subsys/mirror";<br>
exec
"/root/scripts/kill_synchs.sh";<br>
#"kill_synchs.sh is a one
liner: ps ax | grep synch_all | awk {'print $1'} | xargs -n1 kill
-9<br>
exit;<br>
}<br>
}<br>
}<br>
</td>
</tr>
</tbody>
</table>
</center>
<br clear="all">
Mail::Sendmail is a standard Perl module you can get from <a
href="www.cpan.org">CPAN</a>, but you'll notice two homemade ones.
There's no magic here, just decided to use them for portability
and readability (yes, I know, even though it's perl ;-)). The
"EnvConfig" one lets us set the paths and options we're using external
to the script so that the<br>
actual synch_all.pl script doesn't change. You could even add the
file and directory lists to this, but ours doesn't change much cluster
to cluster. The "whohostlib" has two functions used often.
One checks which host the script is running on, and the other
(whostatus) checks whether the node running the script is holding the
services.<br>
<br>
<table border="1"
style="background-color: rgb(255, 255, 204); text-align: left; margin-left: auto; margin-right: auto;"
nosave="" cellpadding="5">
<tbody>
<tr>
<td># Configuration file for all scripts, including host
definitions, etc.<br>
<br>
package EnvConfig;<br>
<br>
#www_host = "wwwdev.domain.com";<br>
$www_host = "wwwint.domain.com";<br>
#www_host = "wwwprod.domain.com";<br>
<br>
$sshpath = "/usr/bin/ssh";<br>
<br>
$rsyncoptions = "-aurlq --delete";<br>
#$rsyncoptions = "-aurlv";<br>
<br>
$smtphost = 'mail.domain.com';<br>
<br>
$Debug = 0;<br>
1;<br>
</td>
</tr>
</tbody>
</table>
<br>
<br>
<br>
<table border="1"
style="background-color: rgb(255, 255, 204); text-align: left; margin-left: auto; margin-right: auto;"
nosave="" cellpadding="5">
<tbody>
<tr>
<td>#!/usr/bin/perl<br>
<br>
use strict;<br>
use diagnostics;<br>
<br>
package whohostlib;<br>
sub whohost {<br>
my $other;<br>
my $me;<br>
<br>
#Who am I?<br>
$me = `hostname -s`;<br>
chomp $me;<br>
$me =~ s/[a-zA-Z]//g;<br>
<br>
if ($me) {<br>
$other = 0;<br>
} else {<br>
$other = 1;<br>
}<br>
<br>
return($me,$other);<br>
}<br>
<br>
#By doing "grep mirror haresources", we peel off the line with the info
we want...the cluster IP. Mirror will be added to you haresources
later...<br>
sub whostatus {<br>
my $status = 0;<br>
my $x = `grep mirror /etc/ha.d/haresources`;<br>
my @p = split(' ',$x);<br>
my $ip = $p[1];<br>
$ip =~ s/IPaddr:://g;<br>
$x = `/etc/ha.d/resource.d/IPaddr $ip
status`; <br>
<br>
# If I am primary, return 1 else return 0<br>
if ($x =~ /running/) {<br>
$status = 1;<br>
} else {<br>
$status = 0;<br>
}<br>
<br>
return $status;<br>
}<br>
<br>
<br>
1; #Return True Value<br>
</td>
</tr>
</tbody>
</table>
<br>
<br>
<b>NOTE: </b>We use a convention for our clusters that if the
clustername is "foo" the two nodes that make up the cluster will be
named "foo0" and "foo1". The private data interfaces are given
the hostnames "local0" and "local1". This makes stuff easier with
scripting, but you don't have to do it.<br>
<b><br>
</b><b>NOTE #2:</b> You may not want a mail fired off. You
may just want a log entry. You might want both. If you're
unfamiliar with perl, to add a log entry, substitute all the commands
with "MAIL" in them above with the following:
<center>
<table border="1" bgcolor="#ffffcc" nosave="">
<tbody>
<tr>
<td><i><tt>open(LOG,">>/var/log/ha-log");</tt></i> <br>
<i><tt>$dstr = `date +%Y/%m/%d_%T`;</tt></i> <br>
<i><tt>chomp $dstr;</tt></i> <br>
<i><tt>print LOG "$dstr RSYNC: Process starting before previous
one completes!\n";</tt></i> <br>
<i><tt>close LOG;</tt></i></td>
</tr>
</tbody>
</table>
</center>
<br>
At this point you want to create your crontab entry. The hard
link prevents more than one sync process from running at the same time,
but you want to have a decent idea of how often you need to synchronize
and how long it will take to synchronize.
<p>SO, once you determine how often you'll be synchronizing, type
"crontab -e" to modify your crontab entry. If you don't like vi,
try using "setenv EDITOR /usr/local/bin/emacs" (or export for bash
users) to select emacs or a different editor. If you want to
synchronize every 10 minutes, your entry would look like this: </p>
<blockquote>
<pre><i>*/10 * * * * /script_directory/sync.pl &> /dev/null</i></pre>
</blockquote>
You could also redirect output to some logfile if you desire, but keep
in mind how often it runs.
<p><b><font size="+1">Dealing with Failover</font></b> </p>
<p>We're just about there now. The last thing you need to
consider is when you shutdown. You want to do one last
synchronization before you shut down. I accomplished this with a
heartbeat service called "mirror". On startup, we do nothing
other than prevent the very first synch_all (as a precaution) and
notify the necessary admins about the failover. </p>
<p><b>NOTE:</b> You will want to be sure that any
applications writing to the synch'ed areas stop before your last
sync. If these applications are controlled via the ipresources
configuration file, you can ensure this by listing the "mirror" script
right after your IP address. However, if they are not, you may
want to add an application exit to the beginning of your mirror script. </p>
<p><b></b></p>
<center>
<table border="1" bgcolor="#ffffcc" nosave="">
<tbody>
<tr nosave="">
<td nosave="">#!/usr/bin/perl<br>
<br>
use lib "/root/scripts";<br>
use whohostlib;<br>
<br>
# See how we were called.<br>
if ($#ARGV == 0) {<br>
$switch = $ARGV[0];<br>
} else {<br>
print "Usage: mirror {start|stop|status|restart}\n";<br>
exit -1;<br>
} <br>
<br>
# start)<br>
<br>
if ($switch eq "start") {<br>
#Make sure to skip first sync.....<br>
($me,$other) = whohostlib::whohost;<br>
system ("/usr/bin/ssh local$me /bin/touch
/var/lock/subsys/mirrorstop");<br>
#Notify admins of startup....<br>
open (XXX,"/root/scripts/.pagelist");<br>
$hh = `hostname`;<br>
chomp $hh;<br>
while (<XXX>) {<br>
chomp;<br>
system "/root/scripts/pagescript.pl $_ \"Starting up
heartbeat services on node $hh\"";<br>
}<br>
close XXX;<br>
<br>
} elsif ($switch eq "stop") {<br>
#stop)<br>
<br>
print "Mirror stop: \n";<br>
<br>
#Am I serving???<br>
#We'll consider mirror to be "running" if the IP is
on this box...<br>
$x = `grep mirror /etc/ha.d/haresources`;<br>
@p = split(' ',$x);<br>
$ip = $p[1];<br>
$x = `/etc/ha.d/resource.d/IPaddr $ip
status`; <br>
if ($x =~ /running/) {<br>
#Notify admins of shutdown....<br>
open (XXX,"/root/scripts/.pagelist");<br>
$hh = `hostname`;<br>
chomp $hh;<br>
while (<XXX>) {<br>
chomp;<br>
system
"/root/scripts/pagescript.pl $_ \"Shutting down heartbeat services on
node $hh\"";<br>
}<br>
close XXX;<br>
<br>
#You want fries with that? One last sync to
other box, if not in the middle of one now...<br>
if ( -e "/var/lock/subsys/mirror" ) {<br>
print "Not syncing, already doing
so...\n";<br>
} else {<br>
print "Synchronizing data on
standby node: \n";<br>
system
"/root/scripts/synch_all.pl";<br>
}<br>
# To prevent NEW master from syncing back to us too
soon, lock their first synch_all...<br>
($me,$other) = whohostlib::whohost;<br>
system ("/usr/bin/ssh local$other /bin/touch
/var/lock/subsys/mirrorstop");<br>
}<br>
<br>
} elsif ($switch eq "status") {<br>
#status)<br>
#We'll consider mirror to be "running" if the IP is
on this box...<br>
$y = `grep mirror /etc/ha.d/haresources`;<br>
@p = split(' ',$y);<br>
$ip = $p[1];<br>
<br>
$x = `/etc/ha.d/resource.d/IPaddr $ip
status`; <br>
print $x;<br>
} elsif ($switch eq "restart") {<br>
#restart)<br>
<br>
system "/etc/ha.d/resource.d/mirror stop";<br>
system "/etc/ha.d/resource.d/mirror start";<br>
<br>
} else {<br>
print "Usage: mirror {start|stop|status|restart}\n";<br>
}<br>
exit;<br>
</td>
</tr>
</tbody>
</table>
</center>
<center></center>
<br>
Finally, you need to install the script in your ipresources on both
nodes. You'll want mirror to be the first service listed (after
IP). For our webserver, it would read:
<pre><i> nodeA 192.168.85.1 mirror httpd</i></pre>
<p><b><font color="#ff0000"><font size="+3">Be Careful</font></font></b> </p>
<p>Please test your setup on non-critical data first. There could
be a bad typo above or whatever. The "--delete" option can be
dangerous. You've been warned. <br>
</p>
<p>Rev 1.0.0<br>
Rudy Pawul <br>
rpawul@iso-ne.com </p>
</body>
</html>
|
