1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
|
.. SPDX-License-Identifier: BSD-3-Clause
Copyright(c) 2018-2022 Intel Corporation.
IFPGA Rawdev Driver
======================
FPGA is used more and more widely in Cloud and NFV, one primary reason is
that FPGA not only provides ASIC performance but also it's more flexible
than ASIC.
FPGA uses Partial Reconfigure (PR) Parts of Bit Stream to achieve its
flexibility. That means one FPGA Device Bit Stream is divided into many Parts
of Bit Stream(each Part of Bit Stream is defined as AFU-Accelerated Function
Unit), and each AFU is a hardware acceleration unit which can be dynamically
reloaded respectively.
By PR (Partial Reconfiguration) AFUs, one FPGA resources can be time-shared by
different users. FPGA hot upgrade and fault tolerance can be provided easily.
The SW IFPGA Rawdev Driver (**ifpga_rawdev**) provides a Rawdev driver
that utilizes Intel FPGA Software Stack OPAE(Open Programmable Acceleration
Engine) for FPGA management.
Implementation details
----------------------
Each instance of IFPGA Rawdev Driver is probed by Intel FpgaDev. In coordination
with OPAE share code IFPGA Rawdev Driver provides common FPGA management ops
for FPGA operation, OPAE provides all following operations:
- FPGA PR (Partial Reconfiguration) management
- FPGA AFUs Identifying
- FPGA Thermal Management
- FPGA Power Management
- FPGA Performance reporting
- FPGA Remote Debug
All configuration parameters are taken by vdev_ifpga_cfg driver. Besides
configuration, vdev_ifpga_cfg driver also hot plugs in IFPGA Bus.
All of the AFUs of one FPGA may share same PCI BDF and AFUs scan depend on
IFPGA Rawdev Driver so IFPGA Bus takes AFU device scan and AFU drivers probe.
All AFU device driver bind to AFU device by its UUID (Universally Unique
Identifier).
To avoid unnecessary code duplication and ensure maximum performance,
handling of AFU devices is left to different PMDs; all the design as
summarized by the following block diagram::
+---------------------------------------------------------------+
| Application(s) |
+----------------------------.----------------------------------+
|
|
+----------------------------'----------------------------------+
| DPDK Framework (APIs) |
+----------|------------|--------.---------------------|--------+
/ \ |
/ \ |
+-------'-------+ +-------'-------+ +--------'--------+
| Eth PMD | | Crypto PMD | | |
+-------.-------+ +-------.-------+ | |
| | | |
| | | |
+-------'-------+ +-------'-------+ | IFPGA |
| Eth AFU Dev | |Crypto AFU Dev | | Rawdev Driver |
+-------.-------+ +-------.-------+ |(OPAE Share Code)|
| | | |
| | Rawdev | |
+-------'------------------'-------+ Ops | |
| IFPGA Bus | -------->| |
+-----------------.----------------+ +--------.--------+
| |
Hot-plugin -->| |
| |
+-----------------'------------------+ +--------'--------+
| vdev_ifpga_cfg driver | | Intel FpgaDev |
+------------------------------------+ +-----------------+
Run-time parameters
-------------------
This driver is invoked automatically in systems added with Intel FPGA,
but PR and IFPGA Bus scan is triggered by command line using
``--vdev 'ifpga_rawdev_cfg`` EAL option.
The following device parameters are supported:
- ``ifpga`` [string]
Provide a specific Intel FPGA device PCI BDF. Can be provided multiple
times for additional instances.
- ``port`` [int]
Each FPGA can provide many channels to PR AFU by software, each channels
is identified by this parameter.
- ``afu_bts`` [string]
If null, the AFU Bit Stream has been PR in FPGA, if not forces PR and
identifies AFU Bit Stream file.
IFPGA AFU Driver
================
AFU (Acceleration Function Unit) is a function or set of functions
that perform various acceleration task on FPGA platform.
The image of AFU is called as GBS (Green Bit Stream)
which can be used by PR (Partial Reconfigure) tool to load into the FPGA,
different AFUs can be dynamically reloaded respectively.
AFU has two main communication paths between the host:
- FPGA to host transactions
The FPGA accesses host memory using a 512 bits data path.
This data path has separate channels for read and write traffic
allowing for simultaneous read and write to occur.
The read and write channels support bursts of 1, 2, and 4 cache lines.
- Host to FPGA (MMIO) transactions
The host can access a 256 KB address space within the FPGA.
This address space contains Device Feature Header (DFHs)
and the control and status registers of the AFU hardware.
AFU must implement the following registers:
- AFU DFH - a 64-bit header at MMIO address offset 0x0
- AFU ID - a 128-bit UUID at MMIO address offset 0x2
The AFU is enumerated and recorded by IFPGA Rawdev Driver.
Then AFU devices are created with the help of IFPGA Bus Driver,
AFU driver probe these AFU devices and expose them
as standard raw devices for application to access.
Implementation details
----------------------
IFPGA Rawdev Driver identifies AFU in FPGA, AFU location (PF/VF address)
and UUID are taken by ``ifpga_rawdev_cfg`` vdev driver
which hot plug AFU into IFPGA Bus.
IFPGA Bus takes AFU device scan and AFU driver probe.
All AFU device driver bind to AFU device by its dedicated UUID.
To avoid unnecessary code duplication and ensure maximum performance,
AFU driver implements the common part of raw device driver.
Several specific AFU drivers are provided for reference.
The design is summarized by the following block diagram::
+---------------------------------------------------------------+
| Application(s) |
+----------------------------.----------------------------------+
|
+----------------------------'----------------------------------+
| DPDK Framework (Rawdev APIs) |
+-----------------+------------------------------------+--------+
| |
+-----------------'----------------+ |
| IFPGA AFU Driver | +--------'--------+
| | | |
|+---------------+ +--------------+| | |
|| AFU Dev1 PMD | | AFU Dev2 PMD || | |
|+-------+-------+ +-------+------+| | |
+--------|-----------------|-------+ | |
| | | |
+--------'------+ +-------'-------+ | IFPGA |
| AFU Dev1 | | AFU Dev2 | | Rawdev Driver |
+-------.-------+ +-------.-------+ | |
| | Rawdev | |
+-------'------------------'-------+ Ops | |
| IFPGA Bus |--------->| |
+-----------------.----------------+ +--------.--------+
| |
Hot-plugin -->| |
| |
+-----------------'------------------+ +--------'--------+
| ifpga_rawdev_cfg vdev driver | | Intel FpgaDev |
+------------------------------------+ +-----------------+
How to test AFU function
------------------------
Suppose AFU is found in FPGA at PCI address 31:00.0,
then you can create and test a AFU device by following steps in application.
#. rte_vdev_init("ifpga_rawdev_cfg0", "ifpga=31:00.0,port=0")
#. rawdev = rte_rawdev_pmd_get_named_dev("afu_0|31:00.0")
#. rte_rawdev_configure(rawdev->dev_id, &cfg, sizeof(cfg))
#. rte_rawdev_selftest(rawdev->dev_id)
#. rte_vdev_uninit("ifpga_rawdev_cfg0")
AFU device name format used in ``rte_rawdev_pmd_get_named_dev`` is ``afu_[port]|[BDF]``.
Please refer to OPAE documentation for the meaning of port.
Each AFU device has specific configuration data, they are defined in ``rte_pmd_afu.h``.
Open FPGA Stack
=====================
Open FPGA Stack (OFS) is a collection of RTL and open source software providing
interfaces to access the instantiated RTL easily in an FPGA. OFS leverages the
DFL for the implementation of the FPGA RTL design.
OFS designs allow for the arrangement of software interfaces across multiple
PCIe endpoints. Some of these interfaces may be PFs defined in the static region
that connect to interfaces in an IP that is loaded via Partial Reconfiguration (PR).
And some of these interfaces may be VFs defined in the PR region that can be
reconfigured by the end-user. Furthermore, these PFs/VFs may use DFLs such that
features may be discovered and accessed in user space with the aid of a generic
kernel driver like vfio-pci. The diagram below depicts an example design with one
PF and two VFs. In this example, it will export the management functions via PF0
and acceleration functions via VF0 and VF1, leverage VFIO to export the MMIO space
to an application.::
+-----------------+ +-------------+ +------------+
| FPGA Management | | DPDK App | | User App |
| App | | | | |
+--------+--------+ +------+------+ +-----+------+
| | |
+--------+--------+ +------+------+ |
| IFPGA PMD | | AFU PMD | |
+--------+--------+ +------+------+ |
| | |
+--------+------------------+---------------+------+
| VFIO-PCI |
+--------+------------------+---------------+------+
| | |
+--------+--------+ +------+------+ +-----+------+
| PF0 | | PF0_VF0 | | PF0_VF1 |
+-----------------+ +-------------+ +------------+
As accelerators are specialized hardware, they are typically limited in the
number installed in a given system. Many use cases require them to be shared
across multiple software contexts or threads of software execution, either
through partitioning of individual dedicated resources, or virtualization of
shared resources. OFS provides several models to share the AFU resources via
PR mechanism and hardware-based virtualization schemes.
#. Legacy model.
With legacy model FPGA cards like Intel PAC N3000 or N5000, there is
a notion that the boundary between the AFU and the shell is also the unit of
PR for those FPGA platforms. This model is only able to handle a
single context, because it only has one PR engine, and one PR region which
has an associated Port device.
#. Multiple VFs per PR slot.
In this model, available AFU resources may allow instantiation of many VFs
which have a dedicated PCIe function with their own dedicated MMIO space, or
partition a region of MMIO space on a single PCIe function. Intel PAC N6000
card has implemented this model.
In this model, the AFU/PR slot was not connected to port device. For DFL's view,
the Next_AFU pointer in FIU feature header of port device points to NULL in this
model. On the other hand, each VF can start with an AFU feature header without
being connected to a FIU Port feature header.
The VFs are created through the Linux kernel driver before we use them in DPDK.
OFS provides the diversity for accessing the AFU resource to RTL developer.
An IP designer may choose to add more than one PF for interfacing with IP
on the FPGA and choose different model to access the AFU resource.
There is one reference architecture design using the "Multiple VFs per PR slot"
model for OFS as illustrated below. In this reference design, it exports the
FPGA management functions via PF0. PF1 will bind with DPDK virtio driver
presenting itself as a network interface to the application. PF2 will bind to the
vfio-pci driver allowing the user space software to discover and interface
with the specific workload like diagnostic test. It leverages AFU PMD driver to
access the AFU resources in DPDK.::
+----------------------+
| PF/VF mux/demux |
+--+--+-----+------+-+-+
| | | | |
+------------------------+ | | | |
PF0 | +---------+ +-+ | |
+---+---+ | +---+----+ | |
| DFH | | | DFH | | |
+-------+ +-----+----+ +--------+ | |
| FME | | VirtIO | | Test | | |
+---+---+ +----------+ +--------+ | |
| PF1 PF2 | |
| | |
| +----------+ |
| | ++
| | |
| | PF0_VF0 | PF0_VF1
| +-----------------+-----------+------------+
| | +-----+-----------+--------+ |
| | | | | | |
| | +------+ | +--+ -+ +--+---+ | |
| | | Port | | | DFH | | DFH | | |
+-----------+ +------+ | +-----+ +------+ | |
| | | DEV | | DEV | | |
| | +-----+ +------+ | |
| | PR Slot | |
| +--------------------------+ |
| Port Gasket |
+------------------------------------------+
|
