1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<!-- This document was generated using DocBuilder 3.3.3 -->
<HTML>
<HEAD>
<TITLE>Distributed Applications</TITLE>
<SCRIPT type="text/javascript" src="../../doc/erlresolvelinks.js">
</SCRIPT>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000" LINK="#0000FF" VLINK="#FF00FF"
ALINK="#FF0000">
<CENTER>
<A HREF="http://www.erlang.se"><IMG BORDER=0 ALT="[Ericsson AB]" SRC="min_head.gif"></A>
</CENTER>
<A NAME="9"><!-- Empty --></A>
<H2>9 Distributed Applications</H2>
<A NAME="9.1"><!-- Empty --></A>
<H3>9.1 Definition</H3>
<P>In a distributed system with several Erlang nodes, there may be
a need to control applications in a distributed manner. If
the node, where a certain application is running, goes down,
the application should be restarted at another node.
<P>Such an application is called a <STRONG>distributed application</STRONG>.
Note that it is the control of the application which is
distributed, all applications can of course be distributed in
the sense that they, for example, use services on other nodes.
<P>Because a distributed application may move between nodes, some
addressing mechanism is required to ensure that it can be
addressed by other applications, regardless on which node it
currently executes. This issue is not addressed here, but the
Kernel module <CODE>global</CODE> or STDLIB module <CODE>pg</CODE> can be
used for this purpose.<A NAME="9.2"><!-- Empty --></A>
<H3>9.2 Specifying Distributed Applications</H3>
<P>Distributed applications are controlled by both the application
controller and a distributed application controller process,
<CODE>dist_ac</CODE>. Both these processes are part of the <CODE>kernel</CODE>
application. Therefore, distributed applications are specified by
configuring the <CODE>kernel</CODE> application, using the following
configuration parameter (see also <CODE>kernel(6)</CODE>):
<P>
<DL>
<DT>
<CODE>distributed = [{Application, [Timeout,] NodeDesc}]</CODE>
</DT>
<DD>
Specifies where the application <CODE>Application = atom()</CODE>
may execute. <CODE>NodeDesc = [Node | {Node,...,Node}]</CODE> is
a list of node names in priority order. The order between
nodes in a tuple is undefined.<BR>
<CODE>Timeout = integer()</CODE> specifies how many milliseconds to
wait before restarting the application at another node.
Defaults to 0.<BR>
</DD>
</DL>
<P>For distribution of application control to work properly,
the nodes where a distributed application may run must contact
each other and negotiate where to start the application. This is
done using the following <CODE>kernel</CODE> configuration parameters:
<P>
<DL>
<DT>
<CODE>sync_nodes_mandatory = [Node]</CODE>
</DT>
<DD>
Specifies which other nodes must be started (within
the timeout specified by <CODE>sync_nodes_timeout</CODE>.
</DD>
<DT>
<CODE>sync_nodes_optional = [Node]</CODE>
</DT>
<DD>
Specifies which other nodes can be started (within
the timeout specified by <CODE>sync_nodes_timeout</CODE>.
</DD>
<DT>
<CODE>sync_nodes_timeout = integer() | infinity</CODE>
</DT>
<DD>
Specifies how many milliseconds to wait for the other nodes
to start.
</DD>
</DL>
<P>When started, the node will wait for all nodes specified by
<CODE>sync_nodes_mandatory</CODE> and <CODE>sync_nodes_optional</CODE> to
come up. When all nodes have come up, or when all mandatory nodes
have come up and the time specified by <CODE>sync_nodes_timeout</CODE>
has elapsed, all applications will be started. If not all
mandatory nodes have come up, the node will terminate.
<P>Example: An application <CODE>myapp</CODE> should run at the node
<CODE>cp1@cave</CODE>. If this node goes down, <CODE>myapp</CODE> should
be restarted at <CODE>cp2@cave</CODE> or <CODE>cp3@cave</CODE>. A system
configuration file <CODE>cp1.config</CODE> for <CODE>cp1@cave</CODE> could
look like:
<PRE>
[{kernel,
[{distributed, [{myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}]},
{sync_nodes_mandatory, [cp2@cave, cp3@cave]},
{sync_nodes_timeout, 5000}
]
}
].
</PRE>
<P>The system configuration files for <CODE>cp2@cave</CODE> and
<CODE>cp3@cave</CODE> are identical, except for the list of mandatory
nodes which should be <CODE>[cp1@cave, cp3@cave]</CODE> for
<CODE>cp2@cave</CODE> and <CODE>[cp1@cave, cp2@cave]</CODE> for
<CODE>cp3@cave</CODE>.
<P>
<TABLE CELLPADDING=4>
<TR>
<TD VALIGN=TOP><IMG ALT="Note!" SRC="note.gif"></TD>
<TD>
<P>All involved nodes must have the same value for
<CODE>distributed</CODE> and <CODE>sync_nodes_timeout</CODE>, or
the behaviour of the system is undefined. </TD>
</TR>
</TABLE>
<A NAME="9.3"><!-- Empty --></A>
<H3>9.3 Starting and Stopping Distributed Applications</H3>
<P>When all involved (mandatory) nodes have been started,
the distributed application can be started by calling
<CODE>application:start(Application)</CODE> at <STRONG>all of these
nodes.</STRONG>
<P>It is of course also possible to use a boot script (see
<A HREF="release_structure.html">Releases</A>) which
automatically starts the application.
<P>The application will be started at the first node, specified
by the <CODE>distributed</CODE> configuration parameter, which is up
and running. The application is started as usual. That is, an
application master is created and calls the application callback
function:
<PRE>
Module:start(normal, StartArgs)
</PRE>
<P>Example: Continuing the example from the previous section,
the three nodes are started, specifying the system configuration
file:
<PRE>
> <STRONG>erl -sname cp1 -config cp1</STRONG>
> <STRONG>erl -sname cp2 -config cp2</STRONG>
> <STRONG>erl -sname cp3 -config cp3</STRONG>
</PRE>
<P>When all nodes are up and running, <CODE>myapp</CODE> can be started.
This is achieved by calling <CODE>application:start(myapp)</CODE> at
all three nodes. It is then started at <CODE>cp1</CODE>, as shown in
the figure below.
<P>
<CENTER>
<IMG ALT="dist1" SRC="dist1.gif"><BR>
<EM><A NAME="dist1"><!-- Empty --></A>Application myapp - Situation 1
</EM>
</CENTER>
<P>Similarly, the application must be stopped by calling
<CODE>application:stop(Application)</CODE> at all involved nodes.<A NAME="9.4"><!-- Empty --></A>
<H3>9.4 Failover</H3>
<P>If the node where the application is running goes down,
the application is restarted (after the specified timeout) at
the first node, specified by the <CODE>distributed</CODE> configuration
parameter, which is up and running. This is called a
<STRONG>failover</STRONG>.
<P>The application is started the normal way at the new node,
that is, by the application master calling:
<PRE>
Module:start(normal, StartArgs)
</PRE>
<P>Exception: If the application has the <CODE>start_phases</CODE> key
defined (see <A HREF="included_applications.html">Included
Applications</A>), then the application is instead started
by calling:
<PRE>
Module:start({failover, Node}, StartArgs)
</PRE>
<P>where <CODE>Node</CODE> is the terminated node.
<P>Example: If <CODE>cp1</CODE> goes down, the system checks which one of
the other nodes, <CODE>cp2</CODE> or <CODE>cp3</CODE>, has the least number of
running applications, but waits for 5 seconds for <CODE>cp1</CODE> to
restart. If <CODE>cp1</CODE> does not restart and <CODE>cp2</CODE> runs fewer
applications than <CODE>cp3,</CODE> then <CODE>myapp</CODE> is restarted on
<CODE>cp2</CODE>.
<P>
<CENTER>
<IMG ALT="dist2" SRC="dist2.gif"><BR>
<EM><A NAME="dist2"><!-- Empty --></A>Application myapp - Situation 2
</EM>
</CENTER>
<P>Suppose now that <CODE>cp2</CODE> goes down as well and does not
restart within 5 seconds. <CODE>myapp</CODE> is now restarted on
<CODE>cp3</CODE>.
<P>
<CENTER>
<IMG ALT="dist3" SRC="dist3.gif"><BR>
<EM><A NAME="dist3"><!-- Empty --></A>Application myapp - Situation 3
</EM>
</CENTER>
<A NAME="9.5"><!-- Empty --></A>
<H3>9.5 Takeover</H3>
<P>If a node is started, which has higher priority according
to <CODE>distributed</CODE>, than the node where a distributed
application is currently running, the application will be
restarted at the new node and stopped at the old node. This is
called a <STRONG>takeover</STRONG>.
<P>The application is started by the application master calling:
<PRE>
Module:start({takeover, Node}, StartArgs)
</PRE>
<P>where <CODE>Node</CODE> is the old node.
<P>Example: If <CODE>myapp</CODE> is running at <CODE>cp3</CODE>, and if
<CODE>cp2</CODE> now restarts, it will not restart <CODE>myapp</CODE>,
because the order between nodes <CODE>cp2</CODE> and <CODE>cp3</CODE> is
undefined.
<P>
<CENTER>
<IMG ALT="dist4" SRC="dist4.gif"><BR>
<EM><A NAME="dist4"><!-- Empty --></A>Application myapp - Situation 4
</EM>
</CENTER>
<P>However, if <CODE>cp1</CODE> restarts as well, the function
<CODE>application:takeover/2</CODE> moves <CODE>myapp</CODE> to <CODE>cp1</CODE>,
because <CODE>cp1</CODE> has a higher priority than <CODE>cp3</CODE> for this
application. In this case,
<CODE>Module:start({takeover, cp3@cave}, StartArgs)</CODE> is executed
at <CODE>cp1</CODE> to start the application.
<P>
<CENTER>
<IMG ALT="dist5" SRC="dist5.gif"><BR>
<EM><A NAME="dist5"><!-- Empty --></A>Application myapp - Situation 5
</EM>
</CENTER>
<CENTER>
<HR>
<SMALL>
Copyright © 1991-2006
<A HREF="http://www.erlang.se">Ericsson AB</A><BR>
</SMALL>
</CENTER>
</BODY>
</HTML>
|
