1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619
|
Network Working Group V. Jacobson
Request for Comments: 2598 K. Nichols
Category: Standards Track Cisco Systems
K. Poduri
Bay Networks
June 1999
An Expedited Forwarding PHB
Status of this Memo
This document specifies an Internet standards track protocol for the
Internet community, and requests discussion and suggestions for
improvements. Please refer to the current edition of the "Internet
Official Protocol Standards" (STD 1) for the standardization state
and status of this protocol. Distribution of this memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (1999). All Rights Reserved.
Abstract
The definition of PHBs (per-hop forwarding behaviors) is a critical
part of the work of the Diffserv Working Group. This document
describes a PHB called Expedited Forwarding. We show the generality
of this PHB by noting that it can be produced by more than one
mechanism and give an example of its use to produce at least one
service, a Virtual Leased Line. A recommended codepoint for this PHB
is given.
A pdf version of this document is available at
ftp://ftp.ee.lbl.gov/papers/ef_phb.pdf
1. Introduction
Network nodes that implement the differentiated services enhancements
to IP use a codepoint in the IP header to select a per-hop behavior
(PHB) as the specific forwarding treatment for that packet [RFC2474,
RFC2475]. This memo describes a particular PHB called expedited
forwarding (EF). The EF PHB can be used to build a low loss, low
latency, low jitter, assured bandwidth, end-to-end service through DS
domains. Such a service appears to the endpoints like a point-to-
point connection or a "virtual leased line". This service has also
been described as Premium service [2BIT].
Jacobson, et al. Standards Track [Page 1]
RFC 2598 An Expedited Forwarding PHB June 1999
Loss, latency and jitter are all due to the queues traffic
experiences while transiting the network. Therefore providing low
loss, latency and jitter for some traffic aggregate means ensuring
that the aggregate sees no (or very small) queues. Queues arise when
(short-term) traffic arrival rate exceeds departure rate at some
node. Thus a service that ensures no queues for some aggregate is
equivalent to bounding rates such that, at every transit node, the
aggregate's maximum arrival rate is less than that aggregate's
minimum departure rate.
Creating such a service has two parts:
1) Configuring nodes so that the aggregate has a well-defined
minimum departure rate. ("Well-defined" means independent of
the dynamic state of the node. In particular, independent of
the intensity of other traffic at the node.)
2) Conditioning the aggregate (via policing and shaping) so that
its arrival rate at any node is always less than that node's
configured minimum departure rate.
The EF PHB provides the first part of the service. The network
boundary traffic conditioners described in [RFC2475] provide the
second part.
The EF PHB is not a mandatory part of the Differentiated Services
architecture, i.e., a node is not required to implement the EF PHB in
order to be considered DS-compliant. However, when a DS-compliant
node claims to implement the EF PHB, the implementation must conform
to the specification given in this document.
The next sections describe the EF PHB in detail and give examples of
how it might be implemented. The keywords "MUST", "MUST NOT",
"REQUIRED", "SHOULD", "SHOULD NOT", and "MAY" that appear in this
document are to be interpreted as described in [Bradner97].
2. Description of EF per-hop behavior
The EF PHB is defined as a forwarding treatment for a particular
diffserv aggregate where the departure rate of the aggregate's
packets from any diffserv node must equal or exceed a configurable
rate. The EF traffic SHOULD receive this rate independent of the
intensity of any other traffic attempting to transit the node. It
SHOULD average at least the configured rate when measured over any
time interval equal to or longer than the time it takes to send an
output link MTU sized packet at the configured rate. (Behavior at
time scales shorter than a packet time at the configured rate is
Jacobson, et al. Standards Track [Page 2]
RFC 2598 An Expedited Forwarding PHB June 1999
deliberately not specified.) The configured minimum rate MUST be
settable by a network administrator (using whatever mechanism the
node supports for non-volatile configuration).
If the EF PHB is implemented by a mechanism that allows unlimited
preemption of other traffic (e.g., a priority queue), the
implementation MUST include some means to limit the damage EF traffic
could inflict on other traffic (e.g., a token bucket rate limiter).
Traffic that exceeds this limit MUST be discarded. This maximum EF
rate, and burst size if appropriate, MUST be settable by a network
administrator (using whatever mechanism the node supports for non-
volatile configuration). The minimum and maximum rates may be the
same and configured by a single parameter.
The Appendix describes how this PHB can be used to construct end-to-
end services.
2.2 Example Mechanisms to Implement the EF PHB
Several types of queue scheduling mechanisms may be employed to
deliver the forwarding behavior described in section 2.1 and thus
implement the EF PHB. A simple priority queue will give the
appropriate behavior as long as there is no higher priority queue
that could preempt the EF for more than a packet time at the
configured rate. (This could be accomplished by having a rate
policer such as a token bucket associated with each priority queue to
bound how much the queue can starve other traffic.)
It's also possible to use a single queue in a group of queues
serviced by a weighted round robin scheduler where the share of the
output bandwidth assigned to the EF queue is equal to the configured
rate. This could be implemented, for example, using one PHB of a
Class Selector Compliant set of PHBs [RFC2474].
Another possible implementation is a CBQ [CBQ] scheduler that gives
the EF queue priority up to the configured rate.
All of these mechanisms have the basic properties required for the EF
PHB though different choices result in different ancillary behavior
such as jitter seen by individual microflows. See Appendix A.3 for
simulations that quantify some of these differences.
2.3 Recommended codepoint for this PHB
Codepoint 101110 is recommended for the EF PHB.
Jacobson, et al. Standards Track [Page 3]
RFC 2598 An Expedited Forwarding PHB June 1999
2.4 Mutability
Packets marked for EF PHB MAY be remarked at a DS domain boundary
only to other codepoints that satisfy the EF PHB. Packets marked for
EF PHBs SHOULD NOT be demoted or promoted to another PHB by a DS
domain.
2.5 Tunneling
When EF packets are tunneled, the tunneling packets must be marked as
EF.
2.6 Interaction with other PHBs
Other PHBs and PHB groups may be deployed in the same DS node or
domain with the EF PHB as long as the requirement of section 2.1 is
met.
3. Security Considerations
To protect itself against denial of service attacks, the edge of a DS
domain MUST strictly police all EF marked packets to a rate
negotiated with the adjacent upstream domain. (This rate must be <=
the EF PHB configured rate.) Packets in excess of the negotiated
rate MUST be dropped. If two adjacent domains have not negotiated an
EF rate, the downstream domain MUST use 0 as the rate (i.e., drop all
EF marked packets).
Since the end-to-end premium service constructed from the EF PHB
requires that the upstream domain police and shape EF marked traffic
to meet the rate negotiated with the downstream domain, the
downstream domain's policer should never have to drop packets. Thus
these drops SHOULD be noted (e.g., via SNMP traps) as possible
security violations or serious misconfiguration. Similarly, since the
aggregate EF traffic rate is constrained at every interior node, the
EF queue should never overflow so if it does the drops SHOULD be
noted as possible attacks or serious misconfiguration.
4. IANA Considerations
This document allocates one codepoint, 101110, in Pool 1 of the code
space defined by [RFC2474].
Jacobson, et al. Standards Track [Page 4]
RFC 2598 An Expedited Forwarding PHB June 1999
5. References
[Bradner97] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2474] Nichols, K., Blake, S., Baker, F. and D. Black,
"Definition of the Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers", RFC 2474, December
1998.
[RFC2475] Black, D., Blake, S., Carlson, M., Davies, E., Wang, Z.
and W. Weiss, "An Architecture for Differentiated
Services", RFC 2475, December 1998.
[2BIT] K. Nichols, V. Jacobson, and L. Zhang, "A Two-bit
Differentiated Services Architecture for the Internet",
Work in Progress, ftp://ftp.ee.lbl.gov/papers/dsarch.pdf
[CBQ] S. Floyd and V. Jacobson, "Link-sharing and Resource
Management Models for Packet Networks", IEEE/ACM
Transactions on Networking, Vol. 3 no. 4, pp. 365-386,
August 1995.
[RFC2415] Poduri, K. and K. Nichols, "Simulation Studies of
Increased Initial TCP Window Size", RFC 2415, September
1998.
[LCN] K. Nichols, "Improving Network Simulation with Feedback",
Proceedings of LCN '98, October 1998.
Jacobson, et al. Standards Track [Page 5]
RFC 2598 An Expedited Forwarding PHB June 1999
6. Authors' Addresses
Van Jacobson
Cisco Systems, Inc
170 W. Tasman Drive
San Jose, CA 95134-1706
EMail: van@cisco.com
Kathleen Nichols
Cisco Systems, Inc
170 W. Tasman Drive
San Jose, CA 95134-1706
EMail: kmn@cisco.com
Kedarnath Poduri
Bay Networks, Inc.
4401 Great America Parkway
Santa Clara, CA 95052-8185
EMail: kpoduri@baynetworks.com
Jacobson, et al. Standards Track [Page 6]
RFC 2598 An Expedited Forwarding PHB June 1999
Appendix A: Example use of and experiences with the EF PHB
A.1 Virtual Leased Line Service
A VLL Service, also known as Premium service [2BIT], is quantified by
a peak bandwidth.
A.2 Experiences with its use in ESNET
A prototype of the VLL service has been deployed on DOE's ESNet
backbone. This uses weighted-round-robin queuing features of Cisco
75xx series routers to implement the EF PHB. The early tests have
been very successful and work is in progress to make the service
available on a routine production basis (see
ftp://ftp.ee.lbl.gov/talks/vj-doeqos.pdf and
ftp://ftp.ee.lbl.gov/talks/vj-i2qos-may98.pdf for details).
A.3 Simulation Results
A.3.1 Jitter variation
In section 2.2, we pointed out that a number of mechanisms might be
used to implement the EF PHB. The simplest of these is a priority
queue (PQ) where the arrival rate of the queue is strictly less than
its service rate. As jitter comes from the queuing delay along the
path, a feature of this implementation is that EF-marked microflows
will see very little jitter at their subscribed rate since packets
spend little time in queues. The EF PHB does not have an explicit
jitter requirement but it is clear from the definition that the
expected jitter in a packet stream that uses a service based on the
EF PHB will be less with PQ than with best-effort delivery. We used
simulation to explore how weighted round-robin (WRR) compares to PQ
in jitter. We chose these two since they"re the best and worst cases,
respectively, for jitter and we wanted to supply rough guidelines for
EF implementers choosing to use WRR or similar mechanisms.
Our simulation model is implemented in a modified ns-2 described in
[RFC2415] and [LCN]. We used the CBQ modules included with ns-2 as a
basis to implement priority queuing and WRR. Our topology has six
hops with decreasing bandwidth in the direction of a single 1.5 Mbps
bottleneck link (see figure 6). Sources produce EF-marked packets at
an average bit rate equal to their subscribed packet rate. Packets
are produced with a variation of +-10% from the interpacket spacing
at the subscribed packet rate. The individual source rates were
picked aggregate to 30% of the bottleneck link or 450 Kbps. A mixture
of FTPs and HTTPs is then used to fill the link. Individual EF packet
sources produce either all 160 byte packets or all 1500 byte packets.
Jacobson, et al. Standards Track [Page 7]
RFC 2598 An Expedited Forwarding PHB June 1999
Though we present the statistics of flows with one size of packet,
all of the experiments used a mixture of short and long packet EF
sources so the EF queues had a mix of both packet lengths.
We defined jitter as the absolute value of the difference between the
arrival times of two adjacent packets minus their departure times,
|(aj-dj) - (ai-di)|. For the target flow of each experiment, we
record the median and 90th percentile values of jitter (expressed as
% of the subscribed EF rate) in a table. The pdf version of this
document contains graphs of the jitter percentiles.
Our experiments compared the jitter of WRR and PQ implementations of
the EF PHB. We assessed the effect of different choices of WRR queue
weight and number of queues on jitter. For WRR, we define the
service-to-arrival rate ratio as the service rate of the EF queue (or
the queue"s minimum share of the output link) times the output link
bandwidth divided by the peak arrival rate of EF-marked packets at
the queue. Results will not be stable if the WRR weight is chosen to
exactly balance arrival and departure rates thus we used a minimum
service-to-arrival ratio of 1.03. In our simulations this means that
the EF queue gets at least 31% of the output links. In WRR
simulations we kept the link full with other traffic as described
above, splitting the non-EF-marked traffic among the non-EF queues.
(It should be clear from the experiment description that we are
attempting to induce worst-case jitter and do not expect these
settings or traffic to represent a "normal" operating point.)
Our first set of experiments uses the minimal service-to-arrival
ratio of 1.06 and we vary the number of individual microflows
composing the EF aggregate from 2 to 36. We compare these to a PQ
implementation with 24 flows. First, we examine a microflow at a
subscribed rate of 56 Kbps sending 1500 byte packets, then one at the
same rate but sending 160 byte packets. Table 1 shows the 50th and
90th percentile jitter in percent of a packet time at the subscribed
rate. Figure 1 plots the 1500 byte flows and figure 2 the 160 byte
flows. Note that a packet-time for a 1500 byte packet at 56 Kbps is
214 ms, for a 160 byte packet 23 ms. The jitter for the large packets
rarely exceeds half a subscribed rate packet-time, though most
jitters for the small packets are at least one subscribed rate
packet-time. Keep in mind that the EF aggregate is a mixture of small
and large packets in all cases so short packets can wait for long
packets in the EF queue. PQ gives a very low jitter.
Table 1: Variation in jitter with number of EF flows: Service/arrival
ratio of 1.06 and subscription rate of 56 Kbps (all values given as %
of subscribed rate)
Jacobson, et al. Standards Track [Page 8]
RFC 2598 An Expedited Forwarding PHB June 1999
1500 byte pack. 160 byte packet
# EF flows 50th % 90th % 50th % 90th %
PQ (24) 1 5 17 43
2 11 47 96 513
4 12 35 100 278
8 10 25 96 126
24 18 47 96 143
Next we look at the effects of increasing the service-to-arrival
ratio. This means that EF packets should remain enqueued for less
time though the bandwidth available to the other queues remains the
same. In this set of experiments the number of flows in the EF
aggregate was fixed at eight and the total number of queues at five
(four non-EF queues). Table 2 shows the results for 1500 and 160 byte
flows. Figures 3 plots the 1500 byte results and figure 4 the 160
byte results. Performance gains leveled off at service-to-arrival
ratios of 1.5. Note that the higher service-to-arrival ratios do not
give the same performance as PQ, but now 90% of packets experience
less than a subscribed packet-time of jitter even for the small
packets.
Table 2: Variation in Jitter of EF flows: service/arrival ratio
varies, 8 flow aggregate, 56 Kbps subscribed rate
WRR 1500 byte pack. 160 byte packet
Ser/Arr 50th % 90th % 50th % 90th %
PQ 1 3 17 43
1.03 14 27 100 178
1.30 7 21 65 113
1.50 5 13 57 104
1.70 5 13 57 100
2.00 5 13 57 104
3.00 5 13 57 100
Increasing the number of queues at the output interfaces can lead to
more variability in the service time for EF packets so we carried out
an experiment varying the number of queues at each output port. We
fixed the number of flows in the aggregate to eight and used the
minimal 1.03 service-to-arrival ratio. Results are shown in figure 5
and table 3. Figure 5 includes PQ with 8 flows as a baseline.
Jacobson, et al. Standards Track [Page 9]
RFC 2598 An Expedited Forwarding PHB June 1999
Table 3: Variation in Jitter with Number of Queues at Output
Interface: Service-to-arrival ratio is 1.03, 8 flow aggregate
# EF 1500 byte packet
flows 50th % 90th %
PQ (8) 1 3
2 7 21
4 7 21
6 8 22
8 10 23
It appears that most jitter for WRR is low and can be reduced by a
proper choice of the EF queue's WRR share of the output link with
respect to its subscribed rate. As noted, WRR is a worst case while
PQ is the best case. Other possibilities include WFQ or CBQ with a
fixed rate limit for the EF queue but giving it priority over other
queues. We expect the latter to have performance nearly identical
with PQ though future simulations are needed to verify this. We have
not yet systematically explored effects of hop count, EF allocations
other than 30% of the link bandwidth, or more complex topologies. The
information in this section is not part of the EF PHB definition but
provided simply as background to guide implementers.
A.3.2 VLL service
We used simulation to see how well a VLL service built from the EF
PHB behaved, that is, does it look like a `leased line' at the
subscribed rate. In the simulations of the last section, none of the
EF packets were dropped in the network and the target rate was always
achieved for those CBR sources. However, we wanted to see if VLL
really looks like a `wire' to a TCP using it. So we simulated long-
lived FTPs using a VLL service. Table 4 gives the percentage of each
link allocated to EF traffic (bandwidths are lower on the links with
fewer EF microflows), the subscribed VLL rate, the average rate for
the same type of sender-receiver pair connected by a full duplex
dedicated link at the subscribed rate and the average of the VLL
flows for each simulation (all sender-receiver pairs had the same
value). Losses only occur when the input shaping buffer overflows but
not in the network. The target rate is not achieved due to the
well-known TCP behavior.
Table 4: Performance of FTPs using a VLL service
% link Average delivered rate (Kbps)
to EF Subscribed Dedicated VLL
20 100 90 90
40 150 143 143
60 225 213 215
Jacobson, et al. Standards Track [Page 10]
RFC 2598 An Expedited Forwarding PHB June 1999
Full Copyright Statement
Copyright (C) The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
Jacobson, et al. Standards Track [Page 11]
|