1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
|
<!--startcut ======================================================= -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<META NAME="generator" CONTENT="lgazmail v1.2N.d">
<TITLE>The Answer Guy 46: From the Dim History: EQL Revisited</TITLE>
</HEAD><BODY BGCOLOR="#FFFFFF" TEXT="#000000"
LINK="#3366FF" VLINK="#A000A0">
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<H4>"The Linux Gazette...<I>making Linux just a little more fun!</I>"</H4>
<P> <hr> <P>
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<center>
<H1><A NAME="answer">
<img src="../../gx/dennis/qbubble.gif" alt="(?)"
border="0" align="middle">
<font color="#B03060">The Answer Guy</font>
<img src="../../gx/dennis/bbubble.gif" alt="(!)"
border="0" align="middle">
</A></H1>
<BR>
<H4>By James T. Dennis,
<a href="mailto:answerguy@ssc.com">answerguy@ssc.com</a><BR>
LinuxCare,
<A HREF="http://www.linuxcare.com/">http://www.linuxcare.com/</A>
</H4>
</center>
<p><hr><p>
<!-- endcut ======================================================= -->
<!-- begin 7 -->
<H3 align="left"><img src="../../gx/dennis/qbubble.gif"
height="50" width="60" alt="(?) " border="0"
>From the Dim History: EQL Revisited</h3>
<h4 align="center">Bandwidth Load Sharing w/o ISP Support</h4>
<p><a href="../../issue13/answer.html">Issue 13</a> was the very
first issue that contained the Answer Guy's replies!</p>
<hr width="40%" align="center">
<p><strong>From Andrew Byrne on Wed, 08 Sep 1999</strong></p>
<P><STRONG>
Hi there,
</STRONG></P>
<P><STRONG>
I came across the information below on the web page
<A HREF="http://sunsite.mff.cuni.cz/lg/issue13/answer.html"
>http://sunsite.mff.cuni.cz/lg/issue13/answer.html</A> which
appears to be written by you.
</STRONG></P>
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
Well, that would probably be mine. I guess that would be a
mirror in Czechoslovakia.
</BLOCKQUOTE>
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
Since it dates back to late 1996, I was wondering if you have any new
information on EQL. I have been told by someone who works for an ISP that
EQL may still work even if it isn't supported at the ISP's end of the
connection. He noted that all incoming connections could only be directed
to either dial-up connection's IP address, but all outgoing data could be
sent via EQL.
</STRONG></P>
<P><STRONG>
If this is true, then EQL may work for what I need, that being using it
with two 33.6kbps PPP connections to provide a web server. All incoming
requests would come via one PPP connection, but web traffic sent out would
be shared across the two PPP connections.
</STRONG></P>
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
Actually, if you use DNS round robin then incoming requests
will be roughly distributed across each connection. Using
"policy-based" routing and the "equal-cost multi-path"
options in the Linux kernel can give you the load
distribution on the outbound traffic.
</BLOCKQUOTE>
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
If you do know any more about how EQL works, could you please tell me if
what I'm saying is true, or correct me if i'm wrong.
</STRONG></P>
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
I think it will be better to address the objective. You
want traffic distribution over multiple ISP links but you're
asking about traffic distribution over multiple low-level
links to a single ISP (EQL). They aren't quite the same
thing.
</BLOCKQUOTE>
<BLOCKQUOTE>
It is quite common for people to present a diagnosis and
perceived solution as though it was their question. One of
the things I learned as a tech support guy (and continually
strive to remember) is to look past the question that's
presented, and guess at the underlying objective.
</BLOCKQUOTE>
<P><STRONG><IMG SRC="../../gx/dennis/qbub.gif" ALT="(?)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
Thankyou!
<br>Andrew Byrne.
</STRONG></P>
<BLOCKQUOTE><IMG SRC="../../gx/dennis/bbub.gif" ALT="(!)"
HEIGHT="28" WIDTH="50" BORDER="0"
>
Before I answer I'll also quote something I said at the end
of my original answer:
</BLOCKQUOTE>
<font color="#000066"><strong><BlockQuote><em>
(After reading this you'll know about as much on this subject
as I do; after using any of this you'll know </EM>much<EM> more).
</em></BlockQuote></strong></font>
<BLOCKQUOTE>
This is true of many things that I say in this column. It will
be worth remembering as you read on.
</BLOCKQUOTE>
<BLOCKQUOTE>
As far as I know EQL still has the constraints that I
detailed back in 1996. Your ISP must participate with a
compatible driver on his end; both of your lines must go to
a single host at your provider's end.
</BLOCKQUOTE>
<BLOCKQUOTE>
It's also important to note that EQL and other "bonding" or
line multiplexing scheme will only increase your effective
bandwidth. This does nothing to lower your latency. Here's
a useful link to explain the importance of this observation:
</BLOCKQUOTE>
<BLOCKQUOTE>
<strong>Bandwidth and Latency: It's the Latency, Stupid</strong>
<dd>by Stuart Cheshire <<A HREF="mailto:cheshire@cs.stanford.edu"
>cheshire@cs.stanford.edu</A>>
<dl>
<dt>TidBITS#367/24-Feb-97 (Part 1)
<DD><A HREF="http://www.tidbits.com/tb-issues/TidBITS-367.html#lnk4"
>http://www.tidbits.com/tb-issues/TidBITS-367.html#lnk4</A>
</DL></BLOCKQUOTE>
<BLOCKQUOTE><DL><DT>
TidBITS#368/03-Mar-97 (Part 2)
<DD><A HREF="http://www.tidbits.com/tb-issues/TidBITS-368.html#lnk4"
>http://www.tidbits.com/tb-issues/TidBITS-368.html#lnk4</A>
</DL></BLOCKQUOTE>
<BLOCKQUOTE>
(Someone kindly pointed me to some copy of this article back
when this column was published. Now, at long last, I can
pass it along. I don't remember whether I was publishing
follow-up comments to TAG columns back then).
</BLOCKQUOTE>
<BLOCKQUOTE>
In any event EQL is not appropriate for cases where you want
to distribute your traffic across connections to different
providers. It's not even useful for distributing traffic
load to different POPs (points of presence) for one ISP.
</BLOCKQUOTE>
<BLOCKQUOTE>
However, there are a couple of options that might help.
</BLOCKQUOTE>
<BLOCKQUOTE>
First, you could simple DNS round robin. This is the
easiest technique. It is also particularly well suited to
web servers. Basically you get one IP address from one ISP,
and another from a different ISP (or two addresses from
different subnets of one ISP). You can bind each of these
addresses to a different PPP interface. If you were using
ISDN or DSL routers (connecting to your system via ethernet)
then you'd use IP aliasing, binding both IP addresses to one
ethernet interface in your Linux host. Then you create an A
record for each of these in your DNS table. Both A records
(or all of them, if you're using more than two) are under
the name: www.YOURDOMAIN.XXX).
</BLOCKQUOTE>
<BLOCKQUOTE>
DNS round robin is quite simple. It's been supported for
years. Basically it's a natural consequence of the fact
that hosts might have multiple interfaces. So I might have
eth0 and eth1 on a system known as foo.starshine.org. There
is no law that says that these interfaces have to be in the
same machine. I can create two web servers with identical
content, and refer to both of them as www.starshine.org.
</BLOCKQUOTE>
<BLOCKQUOTE>
The only change that was required for "round robin DNS" was
to patch BIND (named, the DNS daemon) to "shuffle" the order
of the records as it returned them. Clients tend to use the
first A record they find. Actually a TCP/IP client should
scan the returned addresses for any DNS query to see if any
of them are on matching subnets. Thus a client on the
<tt>192.168.2.*</tt> address should prefer the <tt>192.168.2.*</tt> address
over a <tt>10.*.*.*</tt> address for the same hostname. (For our
purposes this will not be a problem since 99.9999% of your
external web requests will not be from networks that share
any prefix to yours).
</BLOCKQUOTE>
<BLOCKQUOTE>
The load distribution mechanics of this technique are
completely blind. On average about half of the clients will
be accessing you through one of the IP addresses while the
other half will use the other address. In fact, for N
addresses in a round robin farm you'll get roughly 1/N
requests routed to each.
</BLOCKQUOTE>
<BLOCKQUOTE>
The is the important point. Since you're not "peering" with
your ISPs at the routing level (you don't have an AS number,
and you aren't running BGP4) then the links between you and
your ISPs are static. Thus the IP address selected by a
client determines which route the packets will take into
your domain.
</BLOCKQUOTE>
<BLOCKQUOTE>
Note, only the last few hops are static. You're ISP might
be using some routing protocol such as RIP or OSPF to
dynamically select routes through their network, and the
backbones and NAP (network access points) are always using
BGP4 to dynamically select the earlier portions of the
routes --- the ones that lead to your ISP.
</BLOCKQUOTE>
<BLOCKQUOTE>
I realize this is confusing without a diagram. Try to
understand this: each packet between your server and any
client can travel many different routes to get to you.
That's true even if you only have a single IP address.
However, the first few hops (from the client's system to
their ISP) are usually determined by static routes. The
last few hops (from your ISP to your server) are also
usually along static routes. So, for almost all traffic
over the Internet it's only the middle hops that are
dynamic.
</BLOCKQUOTE>
<BLOCKQUOTE>
The key point about DNS round robin load balancing (and
fault tolerance) is that the different IP addresses must be
on different networks (and therefore along different
routes).
</BLOCKQUOTE>
<BLOCKQUOTE>
So, this handles the incoming packets. They come into
different IP addresses on different networks. Therefore
they come in through different routes and thus over
different connections to your ISP(s).
</BLOCKQUOTE>
<BLOCKQUOTE>
Now, what about outgoing traffic. When we use round robin
to feed traffic to multiple servers (mirrored to one
another) there is no problem. Each of the server can have
different routes (outbound), so the traffic will return
along roughly the same route as it traversed on it way in.
</BLOCKQUOTE>
<BLOCKQUOTE>
When we use round robin to funnel packets into a single
system we have a problem.
</BLOCKQUOTE>
<BLOCKQUOTE>
Consider this: an HTTP request comes in on 192.168.2.34 from
172.17.89.10; the web server fashions a response (source:
192.168.2.34, destination: 172.17.89.10). What route will
this response take?
</BLOCKQUOTE>
<BLOCKQUOTE>
The default route.
</BLOCKQUOTE>
<BLOCKQUOTE>
There can normally only be one default route. Normally only
the destination address is considered when making routing
selections. Thus all packets that aren't destined for one
of the local networks (or one of the networks or hosts
explicitly defined in one of our routing tables) normally go
through our default.
</BLOCKQUOTE>
<BLOCKQUOTE>
However, this is Linux. Linux is not always constrained by
"normalcy."
</BLOCKQUOTE>
<BLOCKQUOTE>
In the 2.2 and later kernels we have a few options which
allow us finer control over our routing. Specifically we
"policy based routing" in the kernel, get the "<tt>iproute</tt>"
package and configure a set of routes based on "source
policy." This forces the kernel to consider the source IP
address as well as the destination when it makes its route
selection.
</BLOCKQUOTE>
<BLOCKQUOTE>
Actually it allows us to build multiple routing tables, and
a set of rules which select which table is traversed based
on source IP address, TOS (type of service flags) and other
factors.
</BLOCKQUOTE>
<BLOCKQUOTE>
I found a short "micro HOWTO" on the topic at:
</BLOCKQUOTE>
<BLOCKQUOTE><DL><DT>
Linux Policy Routing
<DD><A HREF="http://www.compendium.com.ar/policy-routing.txt"
>http://www.compendium.com.ar/policy-routing.txt</A>
</DL></BLOCKQUOTE>
<BLOCKQUOTE>
... that site was hard enough to reach that I've
tossed a copy on my own web site at:
</BLOCKQUOTE>
<BLOCKQUOTE><DL><DT>
Starshine Technical Services: The Answer Guy
<DD><A HREF="http://www.starshine.org/tag"
>http://www.starshine.org/tag</A>
</DL></BLOCKQUOTE>
<BLOCKQUOTE>
(I should do that more often!).
</BLOCKQUOTE>
<BLOCKQUOTE>
There are also some notes under
<TT>/usr/src/linux/Documentation/networking/</TT> in any of the
recent (2.2.* or later) kernel.
</BLOCKQUOTE>
<BLOCKQUOTE>
I guess it's also possible to enable the "equal cost
multi-path" option in the kernel. This is a simple (and
crude) technique that will allow the kernel to use redundant
routes. Normally if I were to define two routes to the same
destination then only the first one will be used, so long as
that route is "up." The other (redundant) route would only
be used when the kernel received specific ICMP packets to
alert it to the fact that that route was "down." With
multi-path routing we can define multiple routes to a given
destination and the kernel will distribute packets over them
in a round-robin fashion.
</BLOCKQUOTE>
<BLOCKQUOTE>
I think you could enable both of these features. Thus any
outbound traffic which matched none of your policies would
still be distributed evenly across your available default
routes.
</BLOCKQUOTE>
<BLOCKQUOTE>
I hope you understand that these techniques are ad hoc.
They accomplish "blind" distribution of your load across
your available routes/connections without any sensitivity to
load or any weighting. This is a band-aid approach which
gives some relief based on the averages.
</BLOCKQUOTE>
<BLOCKQUOTE>
Let's contrast this to the ideal networking solution. In an
ideal network you'd be able to publish all of the routes to
your server(s). Routers would then be able to select the
"best" path (based on shortest number of hops across least
loaded lines with the lowest latencies).
</BLOCKQUOTE>
<BLOCKQUOTE>
In the real world this isn't currentl feasible for several
reasons. First, you'd have to have an AS (autonomous
systems) identification. You're ISPs (all of them) would
have to agree to "peer" with you. They'd have to configure
their routers to accept routes from you. Naturally they
would also have to be "peering" with their interconnects and
so on. Finally these routes would then take up valuable
memory in the backbone and 2nd tier routers all over the
Internet. This extra entry in all of those routers is an
additional bit of overhead for them.
</BLOCKQUOTE>
<BLOCKQUOTE>
Ultimately a router's performance is limited to the number
of routes it can hold and the amount of computation that it
takes to select an interface for a given address. So it's
not feasible to store entries for every little "multi-homed"
domain on the Internet. In fact the whole point of the CIDR
"supernetting" policies was to reduce the number of routes
in the backbone routers (and consequently reduce the
latencies of routing traffic through them).
</BLOCKQUOTE>
<BLOCKQUOTE>
So we use these cruder techniques of "equal-cost multi-path"
and "policy-based" routing. They are things that can
be done at the grassroots level (which is the Linux forte,
of course).
</BLOCKQUOTE>
<!-- sig -->
<!-- end 7 -->
<!--startcut ======================================================= -->
<P> <hr> <P>
<H5 align="center"><a href="http://www.linuxgazette.com/ssc.copying.html"
>Copyright ©</a> 1999, James T. Dennis
<BR>Published in <I>The Linux Gazette</I> Issue 46 October 1999</H5>
<H6 ALIGN="center">HTML transformation by
<A HREF="mailto:star@starshine.org">Heather Stern</a> of
Starshine Technical Services,
<A HREF="http://www.starshine.org/">http://www.starshine.org/</A>
</H6>
<P> <hr> <P>
<!-- begin tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::-->
<TABLE WIDTH="96%"><TR VALIGN="center" ALIGN="center">
<TD colspan="2" align="left"><A
HREF="../lg_answer46.html"
><IMG SRC="../../gx/dennis/answernew.gif"
ALT="[ Answer Guy Current Index ]"></A></td>
<TD WIDTH="12%"><A HREF="1.html">1</A></TD>
<TD WIDTH="12%"><A HREF="2.html">2</A></TD>
<TD WIDTH="12%"><A HREF="3.html">3</A></TD>
<TD WIDTH="12%"><A HREF="4.html">5</A></TD>
<TD WIDTH="12%"><A HREF="5.html">5</A></TD>
<TD WIDTH="12%"><A HREF="6.html">6</A></TD>
</TR><TR VALIGN="center" ALIGN="center">
<TD WIDTH="12%"><A HREF="7.html">7</A></TD>
<TD WIDTH="12%"><A HREF="8.html">8</A></TD>
<TD WIDTH="12%"><A HREF="9.html">9</A></TD>
<TD WIDTH="12%"><A HREF="10.html">10</A></TD>
<TD WIDTH="12%"><A HREF="11.html">11</A></TD>
<TD WIDTH="12%"><A HREF="12.html">12</A></TD>
<TD align="right" colspan="2"><A
HREF="../../lg_index_tag.html"
><IMG SRC="../../gx/dennis/answertoc.gif"
ALT="[ Index of Past Answers ]"></A></td>
</TR></TABLE>
<!-- end tagnav ::::::::::::::::::::::::::::::::::::::::::::::::::::-->
<P> <hr> <P>
<!-- begin lgnav ::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<A HREF="../../lg_toc46.html"
><IMG SRC="../../gx/indexnew.gif" ALT="[ Table Of Contents ]"></A>
<A HREF="/index.html"
><IMG SRC="../../gx/homenew.gif" ALT="[ Front Page ]"></A>
<A HREF="../lg_bytes46.html"
><IMG SRC="../../gx/back2.gif" ALT="[ Previous Section ]"></A>
<A HREF="../../lg_faq.html"
><IMG SRC="../../gx/dennis/faq.gif"
ALT="[ Linux Gazette FAQ ]"></A>
<A HREF="../lg_tips46.html"
><IMG SRC="../../gx/fwd.gif" ALT="[ Next Section ]"></A>
<!-- end lgnav ::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
<!-- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: -->
</BODY></HTML>
<!--endcut ========================================================= -->
|
