<pre>Internet Engineering Task Force (IETF)                       R. Krishnan
Request for Comments: 7424                        Brocade Communications
Category: Informational                                          L. Yong
ISSN: 2070-1721                                               Huawei USA
                                                             A. Ghanwani
                                                                    Dell
                                                                   N. So
                                                           Vinci Systems
                                                           B. Khasnabish
                                                         ZTE Corporation
                                                            January 2015


       <span class="h1">Mechanisms for Optimizing Link Aggregation Group (LAG) and</span>
   <span class="h1">Equal-Cost Multipath (ECMP) Component Link Utilization in Networks</span>

Abstract

   Demands on networking infrastructure are growing exponentially due to
   bandwidth-hungry applications such as rich media applications and
   inter-data-center communications.  In this context, it is important
   to optimally use the bandwidth in wired networks that extensively use
   link aggregation groups and equal-cost multipaths as techniques for
   bandwidth scaling.  This document explores some of the mechanisms
   useful for achieving this.

Status of This Memo

   This document is not an Internet Standards Track specification; it is
   published for informational purposes.

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Not all documents
   approved by the IESG are a candidate for any level of Internet
   Standard; see <a href="./rfc5741#section-2">Section&nbsp;2 of RFC 5741</a>.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   <a href="http://www.rfc-editor.org/info/rfc7424">http://www.rfc-editor.org/info/rfc7424</a>.










<span class="grey">Krishnan, et al.              Informational                     [Page 1]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to <a href="https://www.rfc-editor.org/bcp/bcp78">BCP 78</a> and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (<a href="http://trustee.ietf.org/license-info">http://trustee.ietf.org/license-info</a>) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.





































<span class="grey">Krishnan, et al.              Informational                     [Page 2]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


Table of Contents

   <a href="#section-1">1</a>. Introduction ....................................................<a href="#page-4">4</a>
      <a href="#section-1.1">1.1</a>. Acronyms ...................................................<a href="#page-4">4</a>
      <a href="#section-1.2">1.2</a>. Terminology ................................................<a href="#page-5">5</a>
   <a href="#section-2">2</a>. Flow Categorization .............................................<a href="#page-6">6</a>
   <a href="#section-3">3</a>. Hash-Based Load Distribution in LAG/ECMP ........................<a href="#page-6">6</a>
   <a href="#section-4">4</a>. Mechanisms for Optimizing LAG/ECMP Component Link Utilization ...<a href="#page-8">8</a>
      <a href="#section-4.1">4.1</a>. Differences in LAG vs. ECMP ................................<a href="#page-9">9</a>
      <a href="#section-4.2">4.2</a>. Operational Overview ......................................<a href="#page-10">10</a>
      <a href="#section-4.3">4.3</a>. Large Flow Recognition ....................................<a href="#page-11">11</a>
           <a href="#section-4.3.1">4.3.1</a>. Flow Identification ................................<a href="#page-11">11</a>
           4.3.2. Criteria and Techniques for Large Flow
                  Recognition ........................................<a href="#page-12">12</a>
           <a href="#section-4.3.3">4.3.3</a>. Sampling Techniques ................................<a href="#page-12">12</a>
           <a href="#section-4.3.4">4.3.4</a>. Inline Data Path Measurement .......................<a href="#page-14">14</a>
           4.3.5. Use of Multiple Methods for Large Flow
                  Recognition ........................................<a href="#page-15">15</a>
      <a href="#section-4.4">4.4</a>. Options for Load Rebalancing ..............................<a href="#page-15">15</a>
           <a href="#section-4.4.1">4.4.1</a>. Alternative Placement of Large Flows ...............<a href="#page-15">15</a>
           <a href="#section-4.4.2">4.4.2</a>. Redistributing Small Flows .........................<a href="#page-16">16</a>
           <a href="#section-4.4.3">4.4.3</a>. Component Link Protection Considerations ...........<a href="#page-16">16</a>
           <a href="#section-4.4.4">4.4.4</a>. Algorithms for Load Rebalancing ....................<a href="#page-17">17</a>
           <a href="#section-4.4.5">4.4.5</a>. Example of Load Rebalancing ........................<a href="#page-17">17</a>
   <a href="#section-5">5</a>. Information Model for Flow Rebalancing .........................<a href="#page-18">18</a>
      <a href="#section-5.1">5.1</a>. Configuration Parameters for Flow Rebalancing .............<a href="#page-18">18</a>
      <a href="#section-5.2">5.2</a>. System Configuration and Identification Parameters ........<a href="#page-19">19</a>
      <a href="#section-5.3">5.3</a>. Information for Alternative Placement of Large Flows ......<a href="#page-20">20</a>
      <a href="#section-5.4">5.4</a>. Information for Redistribution of Small Flows .............<a href="#page-21">21</a>
      <a href="#section-5.5">5.5</a>. Export of Flow Information ................................<a href="#page-21">21</a>
      <a href="#section-5.6">5.6</a>. Monitoring Information ....................................<a href="#page-21">21</a>
           <a href="#section-5.6.1">5.6.1</a>. Interface (Link) Utilization .......................<a href="#page-21">21</a>
           <a href="#section-5.6.2">5.6.2</a>. Other Monitoring Information .......................<a href="#page-22">22</a>
   <a href="#section-6">6</a>. Operational Considerations .....................................<a href="#page-23">23</a>
      <a href="#section-6.1">6.1</a>. Rebalancing Frequency .....................................<a href="#page-23">23</a>
      <a href="#section-6.2">6.2</a>. Handling Route Changes ....................................<a href="#page-23">23</a>
      <a href="#section-6.3">6.3</a>. Forwarding Resources ......................................<a href="#page-23">23</a>
   <a href="#section-7">7</a>. Security Considerations ........................................<a href="#page-23">23</a>
   <a href="#section-8">8</a>. References .....................................................<a href="#page-24">24</a>
      <a href="#section-8.1">8.1</a>. Normative References ......................................<a href="#page-24">24</a>
      <a href="#section-8.2">8.2</a>. Informative References ....................................<a href="#page-25">25</a>
   <a href="#appendix-A">Appendix A</a>.  Internet Traffic Analysis and Load-Balancing
                Simulation ...........................................<a href="#page-28">28</a>
   Acknowledgements ..................................................<a href="#page-28">28</a>
   Contributors ......................................................<a href="#page-28">28</a>
   Authors' Addresses ................................................<a href="#page-29">29</a>





<span class="grey">Krishnan, et al.              Informational                     [Page 3]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


<span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>.  Introduction</span>

   Networks extensively use link aggregation groups (LAGs) [<a href="#ref-802.1AX" title="&quot;IEEE Standard for Local and metropolitan area networks - Link Aggregation&quot;">802.1AX</a>] and
   equal-cost multipaths (ECMPs) [<a href="./rfc2991" title="&quot;Multipath Issues in Unicast and Multicast Next-Hop Selection&quot;">RFC2991</a>] as techniques for capacity
   scaling.  For the problems addressed by this document, network
   traffic can be predominantly categorized into two traffic types:
   long-lived large flows and other flows.  These other flows, which
   include long-lived small flows, short-lived small flows, and short-
   lived large flows, are referred to as "small flows" in this document.
   Long-lived large flows are simply referred to as "large flows".

   Stateless hash-based techniques [<a href="#ref-ITCOM" title="&quot;Internet traffic load balancing using dynamic hashing with flow volume&quot;">ITCOM</a>] [<a href="./rfc2991" title="&quot;Multipath Issues in Unicast and Multicast Next-Hop Selection&quot;">RFC2991</a>] [<a href="./rfc2992" title="&quot;Analysis of an Equal-Cost Multi-Path Algorithm&quot;">RFC2992</a>] [<a href="./rfc6790" title="&quot;The Use of Entropy Labels in MPLS Forwarding&quot;">RFC6790</a>]
   are often used to distribute both large flows and small flows over
   the component links in a LAG/ECMP.  However, the traffic may not be
   evenly distributed over the component links due to the traffic
   pattern.

   This document describes mechanisms for optimizing LAG/ECMP component
   link utilization when using hash-based techniques.  The mechanisms
   comprise the following steps: 1) recognizing large flows in a router,
   and 2) assigning the large flows to specific LAG/ECMP component links
   or redistributing the small flows when a component link on the router
   is congested.

   It is useful to keep in mind that in typical use cases for these
   mechanisms, the large flows consume a significant amount of bandwidth
   on a link, e.g., greater than 5% of link bandwidth.  The number of
   such flows would necessarily be fairly small, e.g., on the order of
   10s or 100s per LAG/ECMP.  In other words, the number of large flows
   is NOT expected to be on the order of millions of flows.  Examples of
   such large flows would be IPsec tunnels in service provider backbone
   networks or storage backup traffic in data center networks.

<span class="h3"><a class="selflink" id="section-1.1" href="#section-1.1">1.1</a>.  Acronyms</span>

   DoS:    Denial of Service

   ECMP:   Equal-Cost Multipath

   GRE:    Generic Routing Encapsulation

   IPFIX:  IP Flow Information Export

   LAG:    Link Aggregation Group

   MPLS:   Multiprotocol Label Switching

   NVGRE:  Network Virtualization using Generic Routing Encapsulation



<span class="grey">Krishnan, et al.              Informational                     [Page 4]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   PBR:    Policy-Based Routing

   QoS:    Quality of Service

   STT:    Stateless Transport Tunneling

   VXLAN:  Virtual eXtensible LAN

<span class="h3"><a class="selflink" id="section-1.2" href="#section-1.2">1.2</a>.  Terminology</span>

   Central management entity:
      An entity that is capable of monitoring information about link
      utilization and flows in routers across the network and may be
      capable of making traffic-engineering decisions for placement of
      large flows.  It may include the functions of a collector
      [<a href="./rfc7011" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information&quot;">RFC7011</a>].

   ECMP component link:
      An individual next hop within an ECMP group.  An ECMP component
      link may itself comprise a LAG.

   ECMP table:
      A table that is used as the next hop of an ECMP route that
      comprises the set of ECMP component links and the weights
      associated with each of those ECMP component links.  The input for
      looking up the table is the hash value for the packet, and the
      weights are used to determine which values of the hash function
      map to a given ECMP component link.

   Flow (large or small):
      A sequence of packets for which ordered delivery should be
      maintained, e.g., packets belonging to the same TCP connection.

   LAG component link:
      An individual link within a LAG.  A LAG component link is
      typically a physical link.

   LAG table:
      A table that is used as the output port, which is a LAG, that
      comprises the set of LAG component links and the weights
      associated with each of those component links.  The input for
      looking up the table is the hash value for the packet, and the
      weights are used to determine which values of the hash function
      map to a given LAG component link.

   Large flow(s):
      Refers to long-lived large flow(s).




<span class="grey">Krishnan, et al.              Informational                     [Page 5]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Small flow(s):
      Refers to any of, or a combination of, long-lived small flow(s),
      short-lived small flows, and short-lived large flow(s).

<span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>.  Flow Categorization</span>

   In general, based on the size and duration, a flow can be categorized
   into any one of the following four types, as shown in Figure 1:

   o  short-lived large flow (SLLF),

   o  short-lived small flow (SLSF),

   o  long-lived large flow (LLLF), and

   o  long-lived small flow (LLSF).

        Flow Bandwidth
            ^
            |--------------------|--------------------|
            |                    |                    |
      Large |      SLLF          |       LLLF         |
      Flow  |                    |                    |
            |--------------------|--------------------|
            |                    |                    |
      Small |      SLSF          |       LLSF         |
      Flow  |                    |                    |
            +--------------------+--------------------+--&gt;Flow Duration
                 Short-Lived            Long-Lived
                 Flow                   Flow

               Figure 1: Flow Categorization

   In this document, as mentioned earlier, we categorize long-lived
   large flows as "large flows", and all of the others (long-lived small
   flows, short-lived small flows, and short-lived large flows) as
   "small flows".

<span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>.  Hash-Based Load Distribution in LAG/ECMP</span>

   Hash-based techniques are often used for load balancing of traffic to
   select among multiple available paths within a LAG/ECMP group.  The
   advantages of hash-based techniques for load distribution are the
   preservation of the packet sequence in a flow and the real-time
   distribution without maintaining per-flow state in the router.  Hash-
   based techniques use a combination of fields in the packet's headers





<span class="grey">Krishnan, et al.              Informational                     [Page 6]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   to identify a flow, and the hash function computed using these fields
   is used to generate a unique number that identifies a link/path in a
   LAG/ECMP group.  The result of the hashing procedure is a many-to-one
   mapping of flows to component links.

   Hash-based techniques produce good results with respect to
   utilization of the individual component links if:

   o  the traffic mix constitutes flows such that the result of the hash
      function across these flows is fairly uniform so that a similar
      number of flows is mapped to each component link,

   o  the individual flow rates are much smaller as compared to the link
      capacity, and

   o  the differences in flow rates are not dramatic.

   However, if one or more of these conditions are not met, hash-based
   techniques may result in imbalance in the loads on individual
   component links.

   An example is illustrated in Figure 2.  As shown, there are two
   routers, R1 and R2, and there is a LAG between them that has three
   component links (1), (2), and (3).  A total of ten flows need to be
   distributed across the links in this LAG.  The result of applying the
   hash-based technique is as follows:

   o  Component link (1) has three flows (two small flows and one large
      flow), and the link utilization is normal.

   o  Component link (2) has three flows (three small flows and no large
      flows), and the link utilization is light.

      -  The absence of any large flow causes the component link to be
         underutilized.

   o  Component link (3) has four flows (two small flows and two large
      flows), and the link capacity is exceeded resulting in congestion.

      -  The presence of two large flows causes congestion on this
         component link.










<span class="grey">Krishnan, et al.              Informational                     [Page 7]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


                  +-----------+ -&gt;     +-----------+
                  |           | -&gt;     |           |
                  |           | ===&gt;   |           |
                  |        (1)|--------|(1)        |
                  |           | -&gt;     |           |
                  |           | -&gt;     |           |
                  |   (R1)    | -&gt;     |     (R2)  |
                  |        (2)|--------|(2)        |
                  |           | -&gt;     |           |
                  |           | -&gt;     |           |
                  |           | ===&gt;   |           |
                  |           | ===&gt;   |           |
                  |        (3)|--------|(3)        |
                  |           |        |           |
                  +-----------+        +-----------+

            Where: -&gt;   small flow
                   ===&gt; large flow

                Figure 2: Unevenly Utilized Component Links

   This document presents mechanisms for addressing the imbalance in
   load distribution resulting from commonly used hash-based techniques
   for LAG/ECMP that are shown in the above example.  The mechanisms use
   large flow awareness to compensate for the imbalance in load
   distribution.

<span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>.  Mechanisms for Optimizing LAG/ECMP Component Link Utilization</span>

   The suggested mechanisms in this document are local optimization
   solutions; they are local in the sense that both the identification
   of large flows and rebalancing of the load can be accomplished
   completely within individual routers in the network without the need
   for interaction with other routers.

   This approach may not yield a global optimization of the placement of
   large flows across multiple routers in a network, which may be
   desirable in some networks.  On the other hand, a local approach may
   be adequate for some environments for the following reasons:

   1)  Different links within a network experience different levels of
       utilization; thus, a "targeted" solution is needed for those hot
       spots in the network.  An example is the utilization of a LAG
       between two routers that needs to be optimized.







<span class="grey">Krishnan, et al.              Informational                     [Page 8]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   2)  Some networks may lack end-to-end visibility, e.g., when a
       certain network, under the control of a given operator, is a
       transit network for traffic from other networks that are not
       under the control of the same operator.

<span class="h3"><a class="selflink" id="section-4.1" href="#section-4.1">4.1</a>.  Differences in LAG vs. ECMP</span>

   While the mechanisms explained herein are applicable to both LAGs and
   ECMP groups, it is useful to note that there are some key differences
   between the two that may impact how effective the mechanisms are.
   This relates, in part, to the localized information with which the
   mechanisms are intended to operate.

   A LAG is usually established across links that are between two
   adjacent routers.  As a result, the scope of the problem of
   optimizing the bandwidth utilization on the component links is fairly
   narrow.  It simply involves rebalancing the load across the component
   links between these two routers, and there is no impact whatsoever to
   other parts of the network.  The scheme works equally well for
   unicast and multicast flows.

   On the other hand, with ECMP, redistributing the load across
   component links that are part of the ECMP group may impact traffic
   patterns at all of the routers that are downstream of the given
   router between itself and the destination.  The local optimization
   may result in congestion at a downstream node.  (In its simplest
   form, an ECMP group may be used to distribute traffic on component
   links that are between two adjacent routers, and in that case, the
   ECMP group is no different than a LAG for the purpose of this
   discussion.  It should be noted that an ECMP component link may
   itself comprise a LAG, in which case the scheme may be further
   applied to the component links within the LAG.)

   To demonstrate the limitations of local optimization, consider a two-
   level Clos network topology as shown in Figure 3 with three leaf
   routers (L1, L2, and L3) and two spine routers (S1 and S2).  Assume
   all of the links are 10 Gbps.

   Let L1 have two flows of 4 Gbps each towards L3, and let L2 have one
   flow of 7 Gbps also towards L3.  If L1 balances the load optimally
   between S1 and S2, and L2 sends the flow via S1, then the downlink
   from S1 to L3 would get congested, resulting in packet discards.  On
   the other hand, if L1 had sent both its flows towards S1 and L2 had
   sent its flow towards S2, there would have been no congestion at
   either S1 or S2.






<span class="grey">Krishnan, et al.              Informational                     [Page 9]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-10" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


                    +-----+     +-----+
                    | S1  |     | S2  |
                    +-----+     +-----+
                     / \ \       / /\
                    / +---------+ /  \
                   / /  \  \     /    \
                  / /    \  +------+   \
                 / /      \    /    \   \
              +-----+    +-----+   +-----+
              | L1  |    | L2  |   | L3  |
              +-----+    +-----+   +-----+

              Figure 3: Two-Level Clos Network

   The other issue with applying this scheme to ECMP groups is that it
   may not apply equally to unicast and multicast traffic because of the
   way multicast trees are constructed.

   Finally, it is possible for a single physical link to participate as
   a component link in multiple ECMP groups, whereas with LAGs, a link
   can participate as a component link of only one LAG.

<span class="h3"><a class="selflink" id="section-4.2" href="#section-4.2">4.2</a>.  Operational Overview</span>

   The various steps in optimizing LAG/ECMP component link utilization
   in networks are detailed below:

   Step 1:
      This step involves recognizing large flows in routers and
      maintaining the mapping for each large flow to the component link
      that it uses.  Recognition of large flows is explained in <a href="#section-4.3">Section</a>
      <a href="#section-4.3">4.3</a>.

   Step 2:
      The egress component links are periodically scanned for link
      utilization, and the imbalance for the LAG/ECMP group is
      monitored.  If the imbalance exceeds a certain threshold, then
      rebalancing is triggered.  Measurement of the imbalance is
      discussed further in <a href="#section-5.1">Section 5.1</a>.  In addition to the imbalance,
      further criteria (such as the maximum utilization of any of the
      component links) may also be used to determine whether or not to
      trigger rebalancing.  The use of sampling techniques for the
      measurement of egress component link utilization, including the
      issues of depending on ingress sampling for these measurements,
      are discussed in <a href="#section-4.3.3">Section 4.3.3</a>.






<span class="grey">Krishnan, et al.              Informational                    [Page 10]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-11" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Step 3:
      As a part of rebalancing, the operator can choose to rebalance the
      large flows by placing them on lightly loaded component links of
      the LAG/ECMP group, redistribute the small flows on the congested
      link to other component links of the group, or a combination of
      both.

   All of the steps identified above can be done locally within the
   router itself or could involve the use of a central management
   entity.

   Providing large flow information to a central management entity
   provides the capability to globally optimize flow distribution as
   described in <a href="#section-4.1">Section 4.1</a>.  Consider the following example.  A router
   may have three ECMP next hops that lead down paths P1, P2, and P3.  A
   couple of hops downstream on path P1, there may be a congested link,
   while paths P2 and P3 may be underutilized.  This is something that
   the local router does not have visibility into.  With the help of a
   central management entity, the operator could redistribute some of
   the flows from P1 to P2 and/or P3, resulting in a more optimized flow
   of traffic.

   The steps described above are especially useful when bundling links
   of different bandwidths, e.g., 10 Gbps and 100 Gbps as described in
   [<a href="./rfc7226" title="&quot;Requirements for Advanced Multipath in MPLS Networks&quot;">RFC7226</a>].

<span class="h3"><a class="selflink" id="section-4.3" href="#section-4.3">4.3</a>.  Large Flow Recognition</span>

<span class="h4"><a class="selflink" id="section-4.3.1" href="#section-4.3.1">4.3.1</a>.  Flow Identification</span>

   Flows are typically identified using one or more fields from the
   packet header, for example:

   o  Layer 2: Source Media Access Control (MAC) address, destination
      MAC address, VLAN ID.

   o  IP header: IP protocol, IP source address, IP destination address,
      flow label (IPv6 only).

   o  Transport protocol header: Source port number, destination port
      number.  These apply to protocols such as TCP, UDP, and the Stream
      Control Transmission Protocol (SCTP).

   o  MPLS labels.

   For tunneling protocols like Generic Routing Encapsulation (GRE)
   [<a href="./rfc2784" title="&quot;Generic Routing Encapsulation (GRE)&quot;">RFC2784</a>], Virtual eXtensible LAN (VXLAN) [<a href="./rfc7348" title="&quot;Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks&quot;">RFC7348</a>], Network
   Virtualization using Generic Routing Encapsulation (NVGRE) [<a href="#ref-NVGRE" title="&quot;NVGRE: Network Virtualization using Generic Routing Encapsulation&quot;">NVGRE</a>],



<span class="grey">Krishnan, et al.              Informational                    [Page 11]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-12" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Stateless Transport Tunneling (STT) [<a href="#ref-STT" title="&quot;A Stateless Transport Tunneling Protocol for Network Virtualization (STT)&quot;">STT</a>], Layer 2 Tunneling Protocol
   (L2TP) [<a href="./rfc3931" title="&quot;Layer Two Tunneling Protocol - Version 3 (L2TPv3)&quot;">RFC3931</a>], etc., flow identification is possible based on
   inner and/or outer headers as well as fields introduced by the tunnel
   header, as any or all such fields may be used for load balancing
   decisions [<a href="./rfc5640" title="&quot;Load- Balancing for Mesh Softwires&quot;">RFC5640</a>].

   The above list is not exhaustive.

   The mechanisms described in this document are agnostic to the fields
   that are used for flow identification.

   This method of flow identification is consistent with that of IPFIX
   [<a href="./rfc7011" title="&quot;Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information&quot;">RFC7011</a>].

<span class="h4"><a class="selflink" id="section-4.3.2" href="#section-4.3.2">4.3.2</a>.  Criteria and Techniques for Large Flow Recognition</span>

   From the perspective of bandwidth and time duration, in order to
   recognize large flows, we define an observation interval and measure
   the bandwidth of the flow over that interval.  A flow that exceeds a
   certain minimum bandwidth threshold over that observation interval
   would be considered a large flow.

   The two parameters -- the observation interval and the minimum
   bandwidth threshold over that observation interval -- should be
   programmable to facilitate handling of different use cases and
   traffic characteristics.  For example, a flow that is at or above 10%
   of link bandwidth for a time period of at least one second could be
   declared a large flow [<a href="#ref-DEVOFLOW" title="&quot;DevoFlow: Cost-Effective Flow Management for High Performance Enterprise Networks&quot;">DEVOFLOW</a>].

   In order to avoid excessive churn in the rebalancing, once a flow has
   been recognized as a large flow, it should continue to be recognized
   as a large flow for as long as the traffic received during an
   observation interval exceeds some fraction of the bandwidth
   threshold, for example, 80% of the bandwidth threshold.

   Various techniques to recognize a large flow are described in
   Sections <a href="#section-4.3.3">4.3.3</a>, <a href="#section-4.3.4">4.3.4</a>, and <a href="#section-4.3.5">4.3.5</a>.

<span class="h4"><a class="selflink" id="section-4.3.3" href="#section-4.3.3">4.3.3</a>.  Sampling Techniques</span>

   A number of routers support sampling techniques such as sFlow
   [<a href="#ref-sFlow-v5" title="&quot;sFlow version 5&quot;">sFlow-v5</a>] [<a href="#ref-sFlow-LAG" title="&quot;sFlow LAG Counters Structure&quot;">sFlow-LAG</a>], Packet Sampling (PSAMP) [<a href="./rfc5475" title="&quot;Sampling and Filtering Techniques for IP Packet Selection&quot;">RFC5475</a>], and
   NetFlow Sampling [<a href="./rfc3954" title="&quot;Cisco Systems NetFlow Services Export Version 9&quot;">RFC3954</a>].  For the purpose of large flow
   recognition, sampling needs to be enabled on all of the egress ports
   in the router where such measurements are desired.






<span class="grey">Krishnan, et al.              Informational                    [Page 12]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-13" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Using sFlow as an example, processing in an sFlow collector can
   provide an approximate indication of the mapping of large flows to
   each of the component links in each LAG/ECMP group.  Assuming
   sufficient control plane resources are available, it is possible to
   implement this part of the collector function in the control plane of
   the router to reduce dependence on a central management entity.

   If egress sampling is not available, ingress sampling can suffice
   since the central management entity used by the sampling technique
   typically has visibility across multiple routers in a network and can
   use the samples from an immediately downstream router to make
   measurements for egress traffic at the local router.

   The option of using ingress sampling for this purpose may not be
   available if the downstream router is under the control of a
   different operator or if the downstream device does not support
   sampling.

   Alternatively, since sampling techniques require that the sample be
   annotated with the packet's egress port information, ingress sampling
   may suffice.  However, this means that sampling would have to be
   enabled on all ports, rather than only on those ports where such
   monitoring is desired.  There is one situation in which this approach
   may not work.  If there are tunnels that originate from the given
   router and if the resulting tunnel comprises the large flow, then
   this cannot be deduced from ingress sampling at the given router.
   Instead, for this scenario, if egress sampling is unavailable, then
   ingress sampling from the downstream router must be used.


   To illustrate the use of ingress versus egress sampling, we refer to
   Figure 2.  Since we are looking at rebalancing flows at R1, we would
   need to enable egress sampling on ports (1), (2), and (3) on R1.  If
   egress sampling is not available and if R2 is also under the control
   of the same administrator, enabling ingress sampling on R2's ports
   (1), (2), and (3) would also work, but it would necessitate the
   involvement of a central management entity in order for R1 to obtain
   large flow information for each of its links.  Finally, R1 can only
   enable ingress sampling on all of its ports (not just the ports that
   are part of the LAG/ECMP group being monitored), and that would
   suffice if the sampling technique annotates the samples with the
   egress port information.









<span class="grey">Krishnan, et al.              Informational                    [Page 13]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-14" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   The advantages and disadvantages of sampling techniques are as
   follows.

   Advantages:

   o  Supported in most existing routers.

   o  Requires minimal router resources.

   Disadvantage:

   o  In order to minimize the error inherent in sampling, there is a
      minimum delay for the recognition time of large flows, and in the
      time that it takes to react to this information.

   With sampling, the detection of large flows can be done on the order
   of one second [<a href="#ref-DEVOFLOW" title="&quot;DevoFlow: Cost-Effective Flow Management for High Performance Enterprise Networks&quot;">DEVOFLOW</a>].  A discussion on determining the
   appropriate sampling frequency is available in [<a href="#ref-SAMP-BASIC" title="&quot;Packet Sampling Basics&quot;">SAMP-BASIC</a>].

<span class="h4"><a class="selflink" id="section-4.3.4" href="#section-4.3.4">4.3.4</a>.  Inline Data Path Measurement</span>

   Implementations may perform recognition of large flows by performing
   measurements on traffic in the data path of a router.  Such an
   approach would be expected to operate at the interface speed on every
   interface, accounting for all packets processed by the data path of
   the router.  An example of such an approach is described in IPFIX
   [<a href="./rfc5470" title="&quot;Architecture for IP Flow Information Export&quot;">RFC5470</a>].

   Using inline data path measurement, a faster and more accurate
   indication of large flows mapped to each of the component links in a
   LAG/ECMP group may be possible (as compared to the sampling-based
   approach).

   The advantages and disadvantages of inline data path measurement are
   as follows:

   Advantages:

   o  As link speeds get higher, sampling rates are typically reduced to
      keep the number of samples manageable, which places a lower bound
      on the detection time.  With inline data path measurement, large
      flows can be recognized in shorter windows on higher link speeds
      since every packet is accounted for [<a href="#ref-NDTM" title="&quot;New Directions in Traffic Measurement and Accounting&quot;">NDTM</a>].

   o  Inline data path measurement eliminates the potential dependence
      on a central management entity for large flow recognition.





<span class="grey">Krishnan, et al.              Informational                    [Page 14]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-15" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Disadvantage:

   o  Inline data path measurement is more resource intensive in terms
      of the table sizes required for monitoring all flows.

   As mentioned earlier, the observation interval for determining a
   large flow and the bandwidth threshold for classifying a flow as a
   large flow should be programmable parameters in a router.

   The implementation details of inline data path measurement of large
   flows is vendor dependent and beyond the scope of this document.

<span class="h4"><a class="selflink" id="section-4.3.5" href="#section-4.3.5">4.3.5</a>.  Use of Multiple Methods for Large Flow Recognition</span>

   It is possible that a router may have line cards that support a
   sampling technique while other line cards support inline data path
   measurement.  As long as there is a way for the router to reliably
   determine the mapping of large flows to component links of a LAG/ECMP
   group, it is acceptable for the router to use more than one method
   for large flow recognition.

   If both methods are supported, inline data path measurement may be
   preferable because of its speed of detection [<a href="#ref-FLOW-ACC" title="&quot;Packet Sampling for Flow Accounting: Challenges and Limitations&quot;">FLOW-ACC</a>].

<span class="h3"><a class="selflink" id="section-4.4" href="#section-4.4">4.4</a>.  Options for Load Rebalancing</span>

   The following subsections describe suggested techniques for load
   balancing.  Equipment vendors may implement more than one technique,
   including those not described in this document, and allow the
   operator to choose between them.

   Note that regardless of the method used, perfect rebalancing of large
   flows may not be possible since flows arrive and depart at different
   times.  Also, any flows that are moved from one component link to
   another may experience momentary packet reordering.

<span class="h4"><a class="selflink" id="section-4.4.1" href="#section-4.4.1">4.4.1</a>.  Alternative Placement of Large Flows</span>

   Within a LAG/ECMP group, member component links with the least
   average link utilization are identified.  Some large flow(s) from the
   heavily loaded component links are then moved to those lightly loaded
   member component links using a PBR rule in the ingress processing
   element(s) in the routers.

   With this approach, only certain large flows are subjected to
   momentary flow reordering.





<span class="grey">Krishnan, et al.              Informational                    [Page 15]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-16" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Moving a large flow will increase the utilization of the link that it
   is moved to, potentially once again creating an imbalance in the
   utilization across the component links.  Therefore, when moving a
   large flow, care must be taken to account for the existing load and
   the future load after the large flow has been moved.  Further, the
   appearance of new large flows may require a rearrangement of the
   placement of existing flows.

   Consider a case where there is a LAG compromising four 10 Gbps
   component links and there are four large flows, each of 1 Gbps.
   These flows are each placed on one of the component links.
   Subsequently, a fifth large flow of 2 Gbps is recognized, and to
   maintain equitable load distribution, it may require placement of one
   of the existing 1 Gbps flow to a different component link.  This
   would still result in some imbalance in the utilization across the
   component links.

<span class="h4"><a class="selflink" id="section-4.4.2" href="#section-4.4.2">4.4.2</a>.  Redistributing Small Flows</span>

   Some large flows may consume the entire bandwidth of the component
   link(s).  In this case, it would be desirable for the small flows to
   not use the congested component link(s).

   o  The LAG/ECMP table is modified to include only non-congested
      component link(s).  Small flows hash into this table to be mapped
      to a destination component link.  Alternatively, if certain
      component links are heavily loaded but not congested, the output
      of the hash function can be adjusted to account for large flow
      loading on each of the component links.

   o  The PBR rules for large flows (refer to <a href="#section-4.4.1">Section 4.4.1</a>) must have
      strict precedence over the LAG/ECMP table lookup result.

   This method works on some existing router hardware.  The idea is to
   prevent, or reduce the probability, that a small flow hashes into the
   congested component link(s).

   With this approach, the small flows that are moved would be subject
   to reordering.

<span class="h4"><a class="selflink" id="section-4.4.3" href="#section-4.4.3">4.4.3</a>.  Component Link Protection Considerations</span>

   If desired, certain component links may be reserved for link
   protection.  These reserved component links are not used for any
   flows in the absence of any failures.  When there is a failure of one
   or more component links, all the flows on the failed component
   link(s) are moved to the reserved component link(s).  The mapping
   table of large flows to component links simply replaces the failed



<span class="grey">Krishnan, et al.              Informational                    [Page 16]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-17" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   component link with the reserved component link.  Likewise, the
   LAG/ECMP table replaces the failed component link with the reserved
   component link.

<span class="h4"><a class="selflink" id="section-4.4.4" href="#section-4.4.4">4.4.4</a>.  Algorithms for Load Rebalancing</span>

   Specific algorithms for placement of large flows are out of the scope
   of this document.  One possibility is to formulate the problem for
   large flow placement as the well-known bin-packing problem and make
   use of the various heuristics that are available for that problem
   [<a href="#ref-BIN-PACK" title="&quot;Approximation Algorithms for Bin-Packing -- An Updated Survey&quot;">BIN-PACK</a>].

<span class="h4"><a class="selflink" id="section-4.4.5" href="#section-4.4.5">4.4.5</a>.  Example of Load Rebalancing</span>

   Optimizing LAG/ECMP component utilization for the use case in Figure
   2 is depicted below in Figure 4.  The large flow rebalancing
   explained in <a href="#section-4.4.1">Section 4.4.1</a> is used.  The improved link utilization is
   as follows:

   o  Component link (1) has three flows (two small flows and one large
      flow), and the link utilization is normal.

   o  Component link (2) has four flows (three small flows and one large
      flow), and the link utilization is normal now.

   o  Component link (3) has three flows (two small flows and one large
      flow), and the link utilization is normal now.
























<span class="grey">Krishnan, et al.              Informational                    [Page 17]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-18" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


                +-----------+ -&gt;     +-----------+
                |           | -&gt;     |           |
                |           | ===&gt;   |           |
                |        (1)|--------|(1)        |
                |           |        |           |
                |           | ===&gt;   |           |
                |           | -&gt;     |           |
                |           | -&gt;     |           |
                |   (R1)    | -&gt;     |     (R2)  |
                |        (2)|--------|(2)        |
                |           |        |           |
                |           | -&gt;     |           |
                |           | -&gt;     |           |
                |           | ===&gt;   |           |
                |        (3)|--------|(3)        |
                |           |        |           |
                +-----------+        +-----------+

          Where: -&gt;   small flow
                 ===&gt; large flow

              Figure 4: Evenly Utilized Composite Links

   Basically, the use of the mechanisms described in <a href="#section-4.4.1">Section 4.4.1</a>
   resulted in a rebalancing of flows where one of the large flows on
   component link (3), which was previously congested, was moved to
   component link (2), which was previously underutilized.

<span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>.  Information Model for Flow Rebalancing</span>

   In order to support flow rebalancing in a router from an external
   system, the exchange of some information is necessary between the
   router and the external system.  This section provides an exemplary
   information model covering the various components needed for this
   purpose.  The model is intended to be informational and may be used
   as a guide for the development of a data model.

<span class="h3"><a class="selflink" id="section-5.1" href="#section-5.1">5.1</a>.  Configuration Parameters for Flow Rebalancing</span>

   The following parameters are required for configuration of this
   feature:

   o  Large flow recognition parameters:

      -  Observation interval: The observation interval is the time
         period in seconds over which packet arrivals are observed for
         the purpose of large flow recognition.




<span class="grey">Krishnan, et al.              Informational                    [Page 18]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-19" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


      -  Minimum bandwidth threshold: The minimum bandwidth threshold
         would be configured as a percentage of link speed and
         translated into a number of bytes over the observation
         interval.  A flow for which the number of bytes received over a
         given observation interval exceeds this number would be
         recognized as a large flow.

      -  Minimum bandwidth threshold for large flow maintenance: The
         minimum bandwidth threshold for large flow maintenance is used
         to provide hysteresis for large flow recognition.  Once a flow
         is recognized as a large flow, it continues to be recognized as
         a large flow until it falls below this threshold.  This is also
         configured as a percentage of link speed and is typically lower
         than the minimum bandwidth threshold defined above.

   o  Imbalance threshold: A measure of the deviation of the component
      link utilizations from the utilization of the overall LAG/ECMP
      group.  Since component links can be different speeds, the
      imbalance can be computed as follows.  Let the utilization of each
      component link in a LAG/ECMP group with n links of speed b_1, b_2
      .. b_n be u_1, u_2 .. u_n.  The mean utilization is computed as

      u_ave = [ (u_1 * b_1) + (u_2 * b_2) + .. + (u_n * b_n) ] /
              [b_1 + b_2 + .. + b_n].

      The imbalance is then computed as

      max_{i=1..n} | u_i - u_ave |.

   o  Rebalancing interval: The minimum amount of time between
      rebalancing events.  This parameter ensures that rebalancing is
      not invoked too frequently as it impacts packet ordering.

   These parameters may be configured on a system-wide basis or may
   apply to an individual LAG/ECMP group.  They may be applied to an
   ECMP group, provided that the component links are not shared with any
   other ECMP group.

<span class="h3"><a class="selflink" id="section-5.2" href="#section-5.2">5.2</a>.  System Configuration and Identification Parameters</span>

   The following parameters are useful for router configuration and
   operation when using the mechanisms in this document.

   o  IP address: The IP address of a specific router that the feature
      is being configured on or that the large flow placement is being
      applied to.





<span class="grey">Krishnan, et al.              Informational                    [Page 19]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-20" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   o  LAG ID: Identifies the LAG on a given router.  The LAG ID may be
      required when configuring this feature (to apply a specific set of
      large flow identification parameters to the LAG) and will be
      required when specifying flow placement to achieve the desired
      rebalancing.

   o  Component Link ID: Identifies the component link within a LAG or
      ECMP group.  This is required when specifying flow placement to
      achieve the desired rebalancing.

   o  Component Link Weight: The relative weight to be applied to
      traffic for a given component link when using hash-based
      techniques for load distribution.

   o  ECMP group: Identifies a particular ECMP group.  The ECMP group
      may be required when configuring this feature (to apply a specific
      set of large flow identification parameters to the ECMP group) and
      will be required when specifying flow placement to achieve the
      desired rebalancing.  We note that multiple ECMP groups can share
      an overlapping set (or non-overlapping subset) of component links.
      This document does not deal with the complexity of addressing such
      configurations.

   The feature may be configured globally for all LAGs and/or for all
   ECMP groups, or it may be configured specifically for a given LAG or
   ECMP group.

<span class="h3"><a class="selflink" id="section-5.3" href="#section-5.3">5.3</a>.  Information for Alternative Placement of Large Flows</span>

   In cases where large flow recognition is handled by a central
   management entity (see <a href="#section-4.3.3">Section 4.3.3</a>), an information model for flows
   is required to allow the import of large flow information to the
   router.

   Typical fields used for identifying large flows were discussed in
   <a href="#section-4.3.1">Section 4.3.1</a>.  The IPFIX information model [<a href="./rfc7012" title="&quot;Information Model for IP Flow Information Export (IPFIX)&quot;">RFC7012</a>] can be
   leveraged for large flow identification.

   Large flow placement is achieved by specifying the relevant flow
   information along with the following:

   o  For LAG: router's IP address, LAG ID, LAG component link ID.

   o  For ECMP: router's IP address, ECMP group, ECMP component link ID.

   In the case where the ECMP component link itself comprises a LAG, we
   would have to specify the parameters for both the ECMP group as well
   as the LAG to which the large flow is being directed.



<span class="grey">Krishnan, et al.              Informational                    [Page 20]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-21" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


<span class="h3"><a class="selflink" id="section-5.4" href="#section-5.4">5.4</a>.  Information for Redistribution of Small Flows</span>

   Redistribution of small flows is done using the following:

   o  For LAG: The LAG ID and the component link IDs along with the
      relative weight of traffic to be assigned to each component link
      ID are required.

   o  For ECMP: The ECMP group and the ECMP next hop along with the
      relative weight of traffic to be assigned to each ECMP next hop
      are required.

   It is possible to have an ECMP next hop that itself comprises a LAG.
   In that case, we would have to specify the new weights for both the
   ECMP component links and the LAG component links.

   In the case where an ECMP component link itself comprises a LAG, we
   would have to specify new weights for both the component links within
   the ECMP group as well as the component links within the LAG.

<span class="h3"><a class="selflink" id="section-5.5" href="#section-5.5">5.5</a>.  Export of Flow Information</span>

   Exporting large flow information is required when large flow
   recognition is being done on a router but the decision to rebalance
   is being made in a central management entity.  Large flow information
   includes flow identification and the component link ID that the flow
   is currently assigned to.  Other information such as flow QoS and
   bandwidth may be exported too.

   The IPFIX information model [<a href="./rfc7012" title="&quot;Information Model for IP Flow Information Export (IPFIX)&quot;">RFC7012</a>] can be leveraged for large flow
   identification.

<span class="h3"><a class="selflink" id="section-5.6" href="#section-5.6">5.6</a>.  Monitoring Information</span>

<span class="h4"><a class="selflink" id="section-5.6.1" href="#section-5.6.1">5.6.1</a>.  Interface (Link) Utilization</span>

   The incoming bytes (ifInOctets), outgoing bytes (ifOutOctets), and
   interface speed (ifSpeed) can be obtained, for example, from the
   Interfaces table (ifTable) in the MIB module defined in [<a href="./rfc1213" title="&quot;Management Information Base for Network Management of TCP/IP-based internets: MIB-II&quot;">RFC1213</a>].

   The link utilization can then be computed as follows:

   Incoming link utilization = (delta_ifInOctets * 8) / (ifSpeed * T)

   Outgoing link utilization = (delta_ifOutOctets * 8) / (ifSpeed * T)






<span class="grey">Krishnan, et al.              Informational                    [Page 21]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-22" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   Where T is the interval over which the utilization is being measured,
   delta_ifInOctets is the change in ifInOctets over that interval, and
   delta_ifOutOctets is the change in ifOutOctets over that interval.

   For high-speed Ethernet links, the etherStatsHighCapacityTable in the
   MIB module defined in [<a href="./rfc3273" title="&quot;Remote Network Monitoring Management Information Base for High Capacity Networks&quot;">RFC3273</a>] can be used.

   Similar results may be achieved using the corresponding objects of
   other interface management data models such as YANG [<a href="./rfc7223" title="&quot;A YANG Data Model for Interface Management&quot;">RFC7223</a>] if
   those are used instead of MIBs.

   For scalability, it is recommended to use the counter push mechanism
   in [<a href="#ref-sFlow-v5" title="&quot;sFlow version 5&quot;">sFlow-v5</a>] for the interface counters.  Doing so would help avoid
   counter polling through the MIB interface.

   The outgoing link utilization of the component links within a
   LAG/ECMP group can be used to compute the imbalance (see <a href="#section-5.1">Section 5.1</a>)
   for the LAG/ECMP group.

<span class="h4"><a class="selflink" id="section-5.6.2" href="#section-5.6.2">5.6.2</a>.  Other Monitoring Information</span>

   Additional monitoring information that is useful includes:

   o  Number of times rebalancing was done.

   o  Time since the last rebalancing event.

   o  The number of large flows currently rebalanced by the scheme.

   o  A list of the large flows that have been rebalanced including

      -  the rate of each large flow at the time of the last rebalancing
         for that flow,

      -  the time that rebalancing was last performed for the given
         large flow, and

      -  the interfaces that the large flows was (re)directed to.

   o  The settings for the weights of the interfaces within a LAG/ECMP
      group used by the small flows that depend on hashing.










<span class="grey">Krishnan, et al.              Informational                    [Page 22]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-23" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


<span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>.  Operational Considerations</span>

<span class="h3"><a class="selflink" id="section-6.1" href="#section-6.1">6.1</a>.  Rebalancing Frequency</span>

   Flows should be rebalanced only when the imbalance in the utilization
   across component links exceeds a certain threshold.  Frequent
   rebalancing to achieve precise equitable utilization across component
   links could be counterproductive as it may result in moving flows
   back and forth between the component links, impacting packet ordering
   and system stability.  This applies regardless of whether large flows
   or small flows are redistributed.  It should be noted that reordering
   is a concern for TCP flows with even a few packets because three out-
   of-order packets would trigger sufficient duplicate ACKs to the
   sender, resulting in a retransmission [<a href="./rfc5681" title="&quot;TCP Congestion Control&quot;">RFC5681</a>].

   The operator would have to experiment with various values of the
   large flow recognition parameters (minimum bandwidth threshold,
   minimum bandwidth threshold for large flow maintenance, and
   observation interval) and the imbalance threshold across component
   links to tune the solution for their environment.

<span class="h3"><a class="selflink" id="section-6.2" href="#section-6.2">6.2</a>.  Handling Route Changes</span>

   Large flow rebalancing must be aware of any changes to the Forwarding
   Information Base (FIB).  In cases where the next hop of a route no
   longer to points to the LAG or to an ECMP group, any PBR entries
   added as described in Sections <a href="#section-4.4.1">4.4.1</a> and <a href="#section-4.4.2">4.4.2</a> must be withdrawn in
   order to avoid the creation of forwarding loops.

<span class="h3"><a class="selflink" id="section-6.3" href="#section-6.3">6.3</a>.  Forwarding Resources</span>

   Hash-based techniques used for load balancing with LAG/ECMP are
   usually stateless.  The mechanisms described in this document require
   additional resources in the forwarding plane of routers for creating
   PBR rules that are capable of overriding the forwarding decision from
   the hash-based approach.  These resources may limit the number of
   flows that can be rebalanced and may also impact the latency
   experienced by packets due to the additional lookups that are
   required.

<span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>.  Security Considerations</span>

   This document does not directly impact the security of the Internet
   infrastructure or its applications.  In fact, it could help if there
   is a DoS attack pattern that causes a hash imbalance resulting in
   heavy overloading of large flows to certain LAG/ECMP component links.





<span class="grey">Krishnan, et al.              Informational                    [Page 23]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-24" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   An attacker with knowledge of the large flow recognition algorithm
   and any stateless distribution method can generate flows that are
   distributed in a way that overloads a specific path.  This could be
   used to cause the creation of PBR rules that exhaust the available
   PBR rule capacity on routers in the network.  If PBR rules are
   consequently discarded, this could result in congestion on the
   attacker-selected path.  Alternatively, tracking large numbers of PBR
   rules could result in performance degradation.

<span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>.  References</span>

<span class="h3"><a class="selflink" id="section-8.1" href="#section-8.1">8.1</a>.  Normative References</span>

   [<a id="ref-802.1AX">802.1AX</a>]    IEEE, "IEEE Standard for Local and metropolitan area
                networks - Link Aggregation", IEEE Std 802.1AX-2008,
                2008.

   [<a id="ref-RFC2991">RFC2991</a>]    Thaler, D. and C. Hopps, "Multipath Issues in Unicast
                and Multicast Next-Hop Selection", <a href="./rfc2991">RFC 2991</a>, November
                2000, &lt;<a href="http://www.rfc-editor.org/info/rfc2991">http://www.rfc-editor.org/info/rfc2991</a>&gt;.

   [<a id="ref-RFC7011">RFC7011</a>]    Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
                "Specification of the IP Flow Information Export (IPFIX)
                Protocol for the Exchange of Flow Information", STD 77,
                <a href="./rfc7011">RFC 7011</a>, September 2013,
                &lt;<a href="http://www.rfc-editor.org/info/rfc7011">http://www.rfc-editor.org/info/rfc7011</a>&gt;.

   [<a id="ref-RFC7012">RFC7012</a>]    Claise, B., Ed., and B. Trammell, Ed., "Information
                Model for IP Flow Information Export (IPFIX)", <a href="./rfc7012">RFC 7012</a>,
                September 2013,
                &lt;<a href="http://www.rfc-editor.org/info/rfc7012">http://www.rfc-editor.org/info/rfc7012</a>&gt;.




















<span class="grey">Krishnan, et al.              Informational                    [Page 24]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-25" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


<span class="h3"><a class="selflink" id="section-8.2" href="#section-8.2">8.2</a>.  Informative References</span>

   [<a id="ref-BIN-PACK">BIN-PACK</a>]   Coffman, Jr., E., Garey, M., and D. Johnson.
                "Approximation Algorithms for Bin-Packing -- An Updated
                Survey" (in "Algorithm Design for Computer System
                Design"), Springer, 1984.

   [<a id="ref-CAIDA">CAIDA</a>]      "Caida Traffic Analysis Research",
                &lt;<a href="http://www.caida.org/research/traffic-analysis/">http://www.caida.org/research/traffic-analysis/</a>&gt;.

   [<a id="ref-DEVOFLOW">DEVOFLOW</a>]   Mogul, J., Tourrilhes, J., Yalagandula, P., Sharma, P.,
                Curtis, R., and S. Banerjee, "DevoFlow: Cost-Effective
                Flow Management for High Performance Enterprise
                Networks", Proceedings of the ACM SIGCOMM, 2010.

   [<a id="ref-FLOW-ACC">FLOW-ACC</a>]   Zseby, T., Hirsch, T., and B. Claise, "Packet Sampling
                for Flow Accounting: Challenges and Limitations",
                Proceedings of the 9th international Passive and Active
                Measurement Conference, 2008.

   [<a id="ref-ITCOM">ITCOM</a>]      Jo, J., Kim, Y., Chao, H., and F. Merat, "Internet
                traffic load balancing using dynamic hashing with flow
                volume", SPIE ITCOM, 2002.

   [<a id="ref-NDTM">NDTM</a>]       Estan, C. and G. Varghese, "New Directions in Traffic
                Measurement and Accounting", Proceedings of ACM SIGCOMM,
                August 2002.

   [<a id="ref-NVGRE">NVGRE</a>]      Garg, P. and Y. Wang, "NVGRE: Network Virtualization
                using Generic Routing Encapsulation", Work in Progress,
                <a href="./draft-sridharan-virtualization-nvgre-07">draft-sridharan-virtualization-nvgre-07</a>, November 2014.

   [<a id="ref-RFC2784">RFC2784</a>]    Farinacci, D., Li, T., Hanks, S., Meyer, D., and P.
                Traina, "Generic Routing Encapsulation (GRE)", <a href="./rfc2784">RFC 2784</a>,
                March 2000, &lt;<a href="http://www.rfc-editor.org/info/rfc2784">http://www.rfc-editor.org/info/rfc2784</a>&gt;.

   [<a id="ref-RFC6790">RFC6790</a>]    Kompella, K., Drake, J., Amante, S., Henderickx, W., and
                L. Yong, "The Use of Entropy Labels in MPLS Forwarding",
                <a href="./rfc6790">RFC 6790</a>, November 2012,
                &lt;<a href="http://www.rfc-editor.org/info/rfc6790">http://www.rfc-editor.org/info/rfc6790</a>&gt;.

   [<a id="ref-RFC1213">RFC1213</a>]    McCloghrie, K. and M. Rose, "Management Information Base
                for Network Management of TCP/IP-based internets:
                MIB-II", STD 17, <a href="./rfc1213">RFC 1213</a>, March 1991,
                &lt;<a href="http://www.rfc-editor.org/info/rfc1213">http://www.rfc-editor.org/info/rfc1213</a>&gt;.






<span class="grey">Krishnan, et al.              Informational                    [Page 25]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-26" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   [<a id="ref-RFC2992">RFC2992</a>]    Hopps, C., "Analysis of an Equal-Cost Multi-Path
                Algorithm", <a href="./rfc2992">RFC 2992</a>, November 2000,
                &lt;<a href="http://www.rfc-editor.org/info/rfc2992">http://www.rfc-editor.org/info/rfc2992</a>&gt;.

   [<a id="ref-RFC3273">RFC3273</a>]    Waldbusser, S., "Remote Network Monitoring Management
                Information Base for High Capacity Networks", <a href="./rfc3273">RFC 3273</a>,
                July 2002, &lt;<a href="http://www.rfc-editor.org/info/rfc3273">http://www.rfc-editor.org/info/rfc3273</a>&gt;.

   [<a id="ref-RFC3931">RFC3931</a>]    Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed.,
                "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", <a href="./rfc3931">RFC</a>
                <a href="./rfc3931">3931</a>, March 2005,
                &lt;<a href="http://www.rfc-editor.org/info/rfc3931">http://www.rfc-editor.org/info/rfc3931</a>&gt;.

   [<a id="ref-RFC3954">RFC3954</a>]    Claise, B., Ed., "Cisco Systems NetFlow Services Export
                Version 9", <a href="./rfc3954">RFC 3954</a>, October 2004,
                &lt;<a href="http://www.rfc-editor.org/info/rfc3954">http://www.rfc-editor.org/info/rfc3954</a>&gt;.

   [<a id="ref-RFC5470">RFC5470</a>]    Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
                "Architecture for IP Flow Information Export", <a href="./rfc5470">RFC 5470</a>,
                March 2009, &lt;<a href="http://www.rfc-editor.org/info/rfc5470">http://www.rfc-editor.org/info/rfc5470</a>&gt;.

   [<a id="ref-RFC5475">RFC5475</a>]    Zseby, T., Molina, M., Duffield, N., Niccolini, S., and
                F. Raspall, "Sampling and Filtering Techniques for IP
                Packet Selection", <a href="./rfc5475">RFC 5475</a>, March 2009,
                &lt;<a href="http://www.rfc-editor.org/info/rfc5475">http://www.rfc-editor.org/info/rfc5475</a>&gt;.

   [<a id="ref-RFC5640">RFC5640</a>]    Filsfils, C., Mohapatra, P., and C. Pignataro, "Load-
                Balancing for Mesh Softwires", <a href="./rfc5640">RFC 5640</a>, August 2009,
                &lt;<a href="http://www.rfc-editor.org/info/rfc5640">http://www.rfc-editor.org/info/rfc5640</a>&gt;.

   [<a id="ref-RFC5681">RFC5681</a>]    Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
                Control", <a href="./rfc5681">RFC 5681</a>, September 2009,
                &lt;<a href="http://www.rfc-editor.org/info/rfc5681">http://www.rfc-editor.org/info/rfc5681</a>&gt;.

   [<a id="ref-RFC7223">RFC7223</a>]    Bjorklund, M., "A YANG Data Model for Interface
                Management", <a href="./rfc7223">RFC 7223</a>, May 2014,
                &lt;<a href="http://www.rfc-editor.org/info/rfc7223">http://www.rfc-editor.org/info/rfc7223</a>&gt;.

   [<a id="ref-RFC7226">RFC7226</a>]    Villamizar, C., Ed., McDysan, D., Ed., Ning, S., Malis,
                A., and L. Yong, "Requirements for Advanced Multipath in
                MPLS Networks", <a href="./rfc7226">RFC 7226</a>, May 2014,
                &lt;<a href="http://www.rfc-editor.org/info/rfc7226">http://www.rfc-editor.org/info/rfc7226</a>&gt;.

   [<a id="ref-SAMP-BASIC">SAMP-BASIC</a>] Phaal, P. and S. Panchen, "Packet Sampling Basics",
                &lt;<a href="http://www.sflow.org/packetSamplingBasics/">http://www.sflow.org/packetSamplingBasics/</a>&gt;.






<span class="grey">Krishnan, et al.              Informational                    [Page 26]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-27" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


   [<a id="ref-sFlow-v5">sFlow-v5</a>]   Phaal, P. and M. Lavine, "sFlow version 5", July 2004,
                &lt;<a href="http://www.sflow.org/sflow_version_5.txt">http://www.sflow.org/sflow_version_5.txt</a>&gt;.

   [<a id="ref-sFlow-LAG">sFlow-LAG</a>]  Phaal, P. and A. Ghanwani, "sFlow LAG Counters
                Structure", September 2012,
                &lt;<a href="http://www.sflow.org/sflow_lag.txt">http://www.sflow.org/sflow_lag.txt</a>&gt;.

   [<a id="ref-STT">STT</a>]        Davie, B., Ed., and J. Gross, "A Stateless Transport
                Tunneling Protocol for Network Virtualization (STT)",
                Work in Progress, <a href="./draft-davie-stt-06">draft-davie-stt-06</a>, April 2014.

   [<a id="ref-RFC7348">RFC7348</a>]    Mahalingam, M., Dutt, D., Duda, K., Agarwal, P.,
                Kreeger, L., Sridhar, T., Bursell, M., and C. Wright,
                "Virtual eXtensible Local Area Network (VXLAN): A
                Framework for Overlaying Virtualized Layer 2 Networks
                over Layer 3 Networks", <a href="./rfc7348">RFC 7348</a>, August 2014,
                &lt;<a href="http://www.rfc-editor.org/info/rfc7348">http://www.rfc-editor.org/info/rfc7348</a>&gt;.

   [<a id="ref-YONG">YONG</a>]       Yong, L. and P. Yang, "Enhanced ECMP and Large Flow
                Aware Transport", Work in Progress,
                <a href="./draft-yong-pwe3-enhance-ecmp-lfat-01">draft-yong-pwe3-enhance-ecmp-lfat-01</a>, March 2010.






























<span class="grey">Krishnan, et al.              Informational                    [Page 27]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-28" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


<span class="h2"><a class="selflink" id="appendix-A" href="#appendix-A">Appendix A</a>.  Internet Traffic Analysis and Load-Balancing Simulation</span>

   Internet traffic [<a href="#ref-CAIDA" title="&quot;Caida Traffic Analysis Research&quot;">CAIDA</a>] has been analyzed to obtain flow statistics
   such as the number of packets in a flow and the flow duration.  The
   5-tuple in the packet header (IP source address, IP destination
   address, transport protocol source port number, transport protocol
   destination port number, and IP protocol) is used for flow
   identification.  The analysis indicates that &lt; ~2% of the flows take
   ~30% of total traffic volume while the rest of the flows (&gt; ~98%)
   contributes ~70% [<a href="#ref-YONG" title="&quot;Enhanced ECMP and Large Flow Aware Transport&quot;">YONG</a>].

   The simulation has shown that, given Internet traffic patterns, the
   hash-based technique does not evenly distribute flows over ECMP
   paths.  Some paths may be &gt; 90% loaded while others are &lt; 40% loaded.
   The greater the number of ECMP paths, the more severe is the
   imbalance in the load distribution.  This implies that hash-based
   distribution can cause some paths to become congested while other
   paths are underutilized [<a href="#ref-YONG" title="&quot;Enhanced ECMP and Large Flow Aware Transport&quot;">YONG</a>].

   The simulation also shows substantial improvement by using the large
   flow-aware, hash-based distribution technique described in this
   document.  In using the same simulated traffic, the improved
   rebalancing can achieve &lt; 10% load differences among the paths.  It
   proves how large flow-aware, hash-based distribution can effectively
   compensate the uneven load balancing caused by hashing and the
   traffic characteristics [<a href="#ref-YONG" title="&quot;Enhanced ECMP and Large Flow Aware Transport&quot;">YONG</a>].

Acknowledgements

   The authors would like to thank the following individuals for their
   review and valuable feedback on earlier versions of this document:
   Shane Amante, Fred Baker, Michael Bugenhagen, Zhen Cao, Brian
   Carpenter, Benoit Claise, Michael Fargano, Wes George, Sriganesh
   Kini, Roman Krzanowski, Andrew Malis, Dave McDysan, Pete Moyer, Peter
   Phaal, Dan Romascanu, Curtis Villamizar, Jianrong Wong, George Yum,
   and Weifeng Zhang.  As a part of the IETF Last Call process, valuable
   comments were received from Martin Thomson and Carlos Pignataro.

Contributors

   Sanjay Khanna
   Cisco Systems
   EMail: sanjakha@gmail.com








<span class="grey">Krishnan, et al.              Informational                    [Page 28]</span></pre>
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-29" ></span>
<span class="grey"><a href="./rfc7424">RFC 7424</a>       Optimizing Load Distribution over LAG/ECMP   January 2015</span>


Authors' Addresses

   Ram Krishnan
   Brocade Communications
   San Jose, CA 95134
   United States
   Phone: +1-408-406-7890
   EMail: ramkri123@gmail.com


   Lucy Yong
   Huawei USA
   5340 Legacy Drive
   Plano, TX 75025
   United States
   Phone: +1-469-277-5837
   EMail: lucy.yong@huawei.com


   Anoop Ghanwani
   Dell
   5450 Great America Pkwy
   Santa Clara, CA 95054
   United States
   Phone: +1-408-571-3228
   EMail: anoop@alumni.duke.edu


   Ning So
   Vinci Systems
   2613 Fairbourne Cir
   Plano, TX 75093
   United States
   EMail: ningso@yahoo.com


   Bhumip Khasnabish
   ZTE Corporation
   New Jersey 07960
   United States
   Phone: +1-781-752-8003
   EMail: vumip1@gmail.com









Krishnan, et al.              Informational                    [Page 29]
</pre>