1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458
|
.. SPDX-License-Identifier: GPL-2.0
=======================================
DSA switch configuration from userspace
=======================================
The DSA switch configuration is not integrated into the main userspace
network configuration suites by now and has to be performed manualy.
.. _dsa-config-showcases:
Configuration showcases
-----------------------
To configure a DSA switch a couple of commands need to be executed. In this
documentation some common configuration scenarios are handled as showcases:
*single port*
Every switch port acts as a different configurable Ethernet port
*bridge*
Every switch port is part of one configurable Ethernet bridge
*gateway*
Every switch port except one upstream port is part of a configurable
Ethernet bridge.
The upstream port acts as different configurable Ethernet port.
All configurations are performed with tools from iproute2, which is available
at https://www.kernel.org/pub/linux/utils/net/iproute2/
Through DSA every port of a switch is handled like a normal linux Ethernet
interface. The CPU port is the switch port connected to an Ethernet MAC chip.
The corresponding linux Ethernet interface is called the master interface.
All other corresponding linux interfaces are called slave interfaces.
The slave interfaces depend on the master interface being up in order for them
to send or receive traffic. Prior to kernel v5.12, the state of the master
interface had to be managed explicitly by the user. Starting with kernel v5.12,
the behavior is as follows:
- when a DSA slave interface is brought up, the master interface is
automatically brought up.
- when the master interface is brought down, all DSA slave interfaces are
automatically brought down.
In this documentation the following Ethernet interfaces are used:
*eth0*
the master interface
*eth1*
another master interface
*lan1*
a slave interface
*lan2*
another slave interface
*lan3*
a third slave interface
*wan*
A slave interface dedicated for upstream traffic
Further Ethernet interfaces can be configured similar.
The configured IPs and networks are:
*single port*
* lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
* lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7)
* lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11)
*bridge*
* br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
*gateway*
* br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
* wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
.. _dsa-tagged-configuration:
Configuration with tagging support
----------------------------------
The tagging based configuration is desired and supported by the majority of
DSA switches. These switches are capable to tag incoming and outgoing traffic
without using a VLAN based configuration.
*single port*
.. code-block:: sh
# configure each interface
ip addr add 192.0.2.1/30 dev lan1
ip addr add 192.0.2.5/30 dev lan2
ip addr add 192.0.2.9/30 dev lan3
# For kernels earlier than v5.12, the master interface needs to be
# brought up manually before the slave ports.
ip link set eth0 up
# bring up the slave interfaces
ip link set lan1 up
ip link set lan2 up
ip link set lan3 up
*bridge*
.. code-block:: sh
# For kernels earlier than v5.12, the master interface needs to be
# brought up manually before the slave ports.
ip link set eth0 up
# bring up the slave interfaces
ip link set lan1 up
ip link set lan2 up
ip link set lan3 up
# create bridge
ip link add name br0 type bridge
# add ports to bridge
ip link set dev lan1 master br0
ip link set dev lan2 master br0
ip link set dev lan3 master br0
# configure the bridge
ip addr add 192.0.2.129/25 dev br0
# bring up the bridge
ip link set dev br0 up
*gateway*
.. code-block:: sh
# For kernels earlier than v5.12, the master interface needs to be
# brought up manually before the slave ports.
ip link set eth0 up
# bring up the slave interfaces
ip link set wan up
ip link set lan1 up
ip link set lan2 up
# configure the upstream port
ip addr add 192.0.2.1/30 dev wan
# create bridge
ip link add name br0 type bridge
# add ports to bridge
ip link set dev lan1 master br0
ip link set dev lan2 master br0
# configure the bridge
ip addr add 192.0.2.129/25 dev br0
# bring up the bridge
ip link set dev br0 up
.. _dsa-vlan-configuration:
Configuration without tagging support
-------------------------------------
A minority of switches are not capable to use a taging protocol
(DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based
configuration.
*single port*
The configuration can only be set up via VLAN tagging and bridge setup.
.. code-block:: sh
# tag traffic on CPU port
ip link add link eth0 name eth0.1 type vlan id 1
ip link add link eth0 name eth0.2 type vlan id 2
ip link add link eth0 name eth0.3 type vlan id 3
# For kernels earlier than v5.12, the master interface needs to be
# brought up manually before the slave ports.
ip link set eth0 up
ip link set eth0.1 up
ip link set eth0.2 up
ip link set eth0.3 up
# bring up the slave interfaces
ip link set lan1 up
ip link set lan2 up
ip link set lan3 up
# create bridge
ip link add name br0 type bridge
# activate VLAN filtering
ip link set dev br0 type bridge vlan_filtering 1
# add ports to bridges
ip link set dev lan1 master br0
ip link set dev lan2 master br0
ip link set dev lan3 master br0
# tag traffic on ports
bridge vlan add dev lan1 vid 1 pvid untagged
bridge vlan add dev lan2 vid 2 pvid untagged
bridge vlan add dev lan3 vid 3 pvid untagged
# configure the VLANs
ip addr add 192.0.2.1/30 dev eth0.1
ip addr add 192.0.2.5/30 dev eth0.2
ip addr add 192.0.2.9/30 dev eth0.3
# bring up the bridge devices
ip link set br0 up
*bridge*
.. code-block:: sh
# tag traffic on CPU port
ip link add link eth0 name eth0.1 type vlan id 1
# For kernels earlier than v5.12, the master interface needs to be
# brought up manually before the slave ports.
ip link set eth0 up
ip link set eth0.1 up
# bring up the slave interfaces
ip link set lan1 up
ip link set lan2 up
ip link set lan3 up
# create bridge
ip link add name br0 type bridge
# activate VLAN filtering
ip link set dev br0 type bridge vlan_filtering 1
# add ports to bridge
ip link set dev lan1 master br0
ip link set dev lan2 master br0
ip link set dev lan3 master br0
ip link set eth0.1 master br0
# tag traffic on ports
bridge vlan add dev lan1 vid 1 pvid untagged
bridge vlan add dev lan2 vid 1 pvid untagged
bridge vlan add dev lan3 vid 1 pvid untagged
# configure the bridge
ip addr add 192.0.2.129/25 dev br0
# bring up the bridge
ip link set dev br0 up
*gateway*
.. code-block:: sh
# tag traffic on CPU port
ip link add link eth0 name eth0.1 type vlan id 1
ip link add link eth0 name eth0.2 type vlan id 2
# For kernels earlier than v5.12, the master interface needs to be
# brought up manually before the slave ports.
ip link set eth0 up
ip link set eth0.1 up
ip link set eth0.2 up
# bring up the slave interfaces
ip link set wan up
ip link set lan1 up
ip link set lan2 up
# create bridge
ip link add name br0 type bridge
# activate VLAN filtering
ip link set dev br0 type bridge vlan_filtering 1
# add ports to bridges
ip link set dev wan master br0
ip link set eth0.1 master br0
ip link set dev lan1 master br0
ip link set dev lan2 master br0
# tag traffic on ports
bridge vlan add dev lan1 vid 1 pvid untagged
bridge vlan add dev lan2 vid 1 pvid untagged
bridge vlan add dev wan vid 2 pvid untagged
# configure the VLANs
ip addr add 192.0.2.1/30 dev eth0.2
ip addr add 192.0.2.129/25 dev br0
# bring up the bridge devices
ip link set br0 up
Forwarding database (FDB) management
------------------------------------
The existing DSA switches do not have the necessary hardware support to keep
the software FDB of the bridge in sync with the hardware tables, so the two
tables are managed separately (``bridge fdb show`` queries both, and depending
on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb
add`` or ``bridge fdb del`` command acts upon entries from one or both tables).
Up until kernel v4.14, DSA only supported user space management of bridge FDB
entries using the bridge bypass operations (which do not update the software
FDB, just the hardware one) using the ``self`` flag (which is optional and can
be omitted).
.. code-block:: sh
bridge fdb add dev swp0 00:01:02:03:04:05 self static
# or shorthand
bridge fdb add dev swp0 00:01:02:03:04:05 static
Due to a bug, the bridge bypass FDB implementation provided by DSA did not
distinguish between ``static`` and ``local`` FDB entries (``static`` are meant
to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent
to the host port). Instead, all FDB entries with the ``self`` flag (implicit or
explicit) are treated by DSA as ``static`` even if they are ``local``.
.. code-block:: sh
# This command:
bridge fdb add dev swp0 00:01:02:03:04:05 static
# behaves the same for DSA as this command:
bridge fdb add dev swp0 00:01:02:03:04:05 local
# or shorthand, because the 'local' flag is implicit if 'static' is not
# specified, it also behaves the same as:
bridge fdb add dev swp0 00:01:02:03:04:05
The last command is an incorrect way of adding a static bridge FDB entry to a
DSA switch using the bridge bypass operations, and works by mistake. Other
drivers will treat an FDB entry added by the same command as ``local`` and as
such, will not forward it, as opposed to DSA.
Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of
adding a bridge FDB entry to the switch: the bridge bypass discussed above, as
well as a new mode using the ``master`` flag which installs FDB entries in the
software bridge too.
.. code-block:: sh
bridge fdb add dev swp0 00:01:02:03:04:05 master static
Since kernel v5.14, DSA has gained stronger integration with the bridge's
software FDB, and the support for its bridge bypass FDB implementation (using
the ``self`` flag) has been removed. This results in the following changes:
.. code-block:: sh
# This is the only valid way of adding an FDB entry that is supported,
# compatible with v4.14 kernels and later:
bridge fdb add dev swp0 00:01:02:03:04:05 master static
# This command is no longer buggy and the entry is properly treated as
# 'local' instead of being forwarded:
bridge fdb add dev swp0 00:01:02:03:04:05
# This command no longer installs a static FDB entry to hardware:
bridge fdb add dev swp0 00:01:02:03:04:05 static
Script writers are therefore encouraged to use the ``master static`` set of
flags when working with bridge FDB entries on DSA switch interfaces.
Affinity of user ports to CPU ports
-----------------------------------
Typically, DSA switches are attached to the host via a single Ethernet
interface, but in cases where the switch chip is discrete, the hardware design
may permit the use of 2 or more ports connected to the host, for an increase in
termination throughput.
DSA can make use of multiple CPU ports in two ways. First, it is possible to
statically assign the termination traffic associated with a certain user port
to be processed by a certain CPU port. This way, user space can implement
custom policies of static load balancing between user ports, by spreading the
affinities according to the available CPU ports.
Secondly, it is possible to perform load balancing between CPU ports on a per
packet basis, rather than statically assigning user ports to CPU ports.
This can be achieved by placing the DSA masters under a LAG interface (bonding
or team). DSA monitors this operation and creates a mirror of this software LAG
on the CPU ports facing the physical DSA masters that constitute the LAG slave
devices.
To make use of multiple CPU ports, the firmware (device tree) description of
the switch must mark all the links between CPU ports and their DSA masters
using the ``ethernet`` reference/phandle. At startup, only a single CPU port
and DSA master will be used - the numerically first port from the firmware
description which has an ``ethernet`` property. It is up to the user to
configure the system for the switch to use other masters.
DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow
changing the DSA master of a user port. The ``IFLA_DSA_MASTER`` u32 netlink
attribute contains the ifindex of the master device that handles each slave
device. The DSA master must be a valid candidate based on firmware node
information, or a LAG interface which contains only slaves which are valid
candidates.
Using iproute2, the following manipulations are possible:
.. code-block:: sh
# See the DSA master in current use
ip -d link show dev swp0
(...)
dsa master eth0
# Static CPU port distribution
ip link set swp0 type dsa master eth1
ip link set swp1 type dsa master eth0
ip link set swp2 type dsa master eth1
ip link set swp3 type dsa master eth0
# CPU ports in LAG, using explicit assignment of the DSA master
ip link add bond0 type bond mode balance-xor && ip link set bond0 up
ip link set eth1 down && ip link set eth1 master bond0
ip link set swp0 type dsa master bond0
ip link set swp1 type dsa master bond0
ip link set swp2 type dsa master bond0
ip link set swp3 type dsa master bond0
ip link set eth0 down && ip link set eth0 master bond0
ip -d link show dev swp0
(...)
dsa master bond0
# CPU ports in LAG, relying on implicit migration of the DSA master
ip link add bond0 type bond mode balance-xor && ip link set bond0 up
ip link set eth0 down && ip link set eth0 master bond0
ip link set eth1 down && ip link set eth1 master bond0
ip -d link show dev swp0
(...)
dsa master bond0
Notice that in the case of CPU ports under a LAG, the use of the
``IFLA_DSA_MASTER`` netlink attribute is not strictly needed, but rather, DSA
reacts to the ``IFLA_MASTER`` attribute change of its present master (``eth0``)
and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly,
when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports
that were assigned to this interface to the first physical DSA master which is
eligible, based on the firmware description (it effectively reverts to the
startup configuration).
In a setup with more than 2 physical CPU ports, it is therefore possible to mix
static user to CPU port assignment with LAG between DSA masters. It is not
possible to statically assign a user port towards a DSA master that has any
upper interfaces (this includes LAG devices - the master must always be the LAG
in this case).
Live changing of the DSA master (and thus CPU port) affinity of a user port is
permitted, in order to allow dynamic redistribution in response to traffic.
Physical DSA masters are allowed to join and leave at any time a LAG interface
used as a DSA master; however, DSA will reject a LAG interface as a valid
candidate for being a DSA master unless it has at least one physical DSA master
as a slave device.
|