1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
|
===============================================================================
===============================================================================
Reference
=========
- Intel® Communications Chipset 4xxx/4xxxx Series Software for Linux*
Getting Started Guide
===============================================================================
Overview
=========
Intel® QuickAssist Technology provides security and compression acceleration
capabilities used to improve performance and efficiency across the data center.
Previously existing build system has been replaced by Autotools solution
in order to provide with the widely known and easy to use tool to build and
install source packages.
===============================================================================
Licensing
=========
This product is released under the BSD-3-Clause License.
Files within this project have various inbound licenses, listed below:
- Dual BSD/GPLv2 License
- BSD License
This package also links against files with the following licenses:
- OpenSSL License
- ZLIB License
For BSD-3-Clause license, please see the file LICENSE contained in the top
level folder.
===============================================================================
QATlib User's Guide
===================
Web-based documentation for QATlib and related components is available at:
https://intel.github.io/quickassist/qatlib/index.html
Using the QATlib package
========================
Note: The build instructions provided are for RPM-based Linux distributions
such as Fedora, Red Hat Enterprise Linux (RHEL), CentOS, and Rocky Linux.
For Debian-based distributions like Debian and Ubuntu, refer to the
installation guide at:
https://intel.github.io/quickassist/qatlib/install.html#installing-from-sources
These instructions assume operation as a non-root user with sudo
privileges. Users on other distributions may need to adapt the command-lines
accordingly.
Check System Prerequisites
==========================
There are some prerequisites to running the software in this package. If
running from a distro many of these are taken care of, if not here they
are:
* platform must have an Intel® Communications device installed.
Supported devices:
---------------------------
Driver Name | PFid | VFid
---------------------------
4xxx | 4940 | 4941
401xx | 4942 | 4943
402xx | 4944 | 4945
420xx | 4946 | 4947
---------------------------
Examples in this doc are for the 4940 device. See
https://intel.github.io/quickassist/qatlib/requirements.html#supported-devices
for more details of other devices.
* check there's a PF device present
lspci -d 8086:<PFid> will return the list of devices installed, e.g.
"lspci -d 8086:4940"
Note: Later, after "systemctl start qat" or "make install" steps, the
corresponding Virtual Function devices will also be visible and bound
to the vfio-pci driver.
lspci -d 8086:<VFid> will list VF devices which have been created, e.g.
"lspci -d 8086:4941"
* firmware must be available
Check that these files exist:
/lib/firmware/qat_4xxx.bin or /lib/firmware/qat_4xxx.bin.xz
/lib/firmware/qat_4xxx_mmp.bin or /lib/firmware/qat_4xxx_mmp.bin.xz
If not, download the firmware images from linux-firmware and copy them
to /lib/firmware:
wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/qat_4xxx.bin
wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/qat_4xxx_mmp.bin
sudo mv qat_4xxx.bin /lib/firmware
sudo mv qat_4xxx_mmp.bin /lib/firmware
See Supported Devices table linked above for firmware filenames for
other devices. If firmware is not present on linux-firmware please
contact qat-linux@intel.com
On updating any firmware files run "sudo dracut --force" to update
initramfs.
* kernel driver must be running
Use "lsmod | grep qat" to check that these kernel modules are running:
intel_qat
qat_4xxx (See Supported Devices table for other qat module names)
They should load by default if using any of the following:
* A recent Linux kernel (see https://intel.github.io/quickassist/qatlib/requirements.html#kernel-firmware-requirements )
* Fedora 36+ (39+ for 420xx)
* RHEL 8.4+ (for compression use 9.0+)
* each PF device must be bound to a qat kernel module
Use ls <Driver name> to show the BDFs of each bound PF, e.g.
"cd /sys/bus/pci/drivers; ls 4xxx;"
* BIOS settings
Intel VT-d and SR-IOV must be enabled in the platform BIOS.
Consult your platform guide on how to do this.
If using an Intel BKC these usually default to on, you can verify by
rebooting, entering F2 on the console to get to the BIOS menus and
checking these are enabled:
EDKII Menu
-> Socket Configuration
-> IIO Configuration
-> Intel VT for Directed I/O (VT-d)
-> Intel VT for Directed I/O
EDKII Menu
-> Platform Configuration
-> Miscellaneous Configuration
-> SR-IOV Support
* GRUB Configuration Updates
- Step 1: Edit the GRUB configuration file.
* Open the file with a text editor, for example, `sudo vi /etc/default/grub`.
- Step 2: Add the necessary parameters to the `GRUB_CMDLINE_LINUX` entry.
* Enable Intel IOMMU by adding "intel_iommu=on". The line should look like:
`GRUB_CMDLINE_LINUX="... intel_iommu=on"`.
* Add VFIO-PCI device IDs. For a device with the ID 8086:4941, add
"vfio-pci.ids=8086:4941". The line should look like:
`GRUB_CMDLINE_LINUX="... vfio-pci.ids=8086:4941"`.
* Refer to the list of supported devices at
https://intel.github.io/quickassist/qatlib/requirements.html#supported-devices
for other device IDs.
- Step 3: Update the GRUB configuration and reboot the system.
* Generate a new GRUB configuration file:
`sudo grub2-mkconfig -o /etc/grub2-efi.cfg`.
* Reboot the system to apply the changes:
`sudo shutdown -r now`.
===============================================================================
Compilation and installation - quickstart instructions
======================================================
Note, more detailed instructions in following section.
---------------------------------------------------------------------------
Summary if running from Fedora 34+:
# Install qatlib
sudo dnf install -y qatlib
# Add your user to qat group and re-login to make the change effective
sudo usermod -a -G qat `whoami`
sudo su -l $USER
# Enable qat service and make persistent after reboot
sudo systemctl enable qat
sudo systemctl start qat
# The library is now ready to use with your application
# You can also follow these steps to try out a pre-built sample
# application:
# Install qatlib-tests rpm
sudo dnf install -y qatlib-tests
# cpa_sample_code requires a minimum of 500MB to run its compression demo
# To increase amount of locked memory for your user to 500MB:
sudo cp /etc/security/limits.conf /etc/security/limits.conf.qatlib_bak
echo `whoami` - memlock 500000 | sudo tee -a /etc/security/limits.conf > /dev/null
# Re-login in order to make the change effective
sudo su -l $USER
# Run it! (takes several minutes to complete)
cpa_sample_code
----------------------------------------------------------------------------
Summary if building from sources.
# Install dependencies
sudo dnf install -y gcc systemd-devel automake autoconf autoconf-archive libtool
sudo dnf install -y pkg-config openssl-devel zlib-devel nasm numactl-devel
# Clone QATlib into ~/qatlib, i.e. in your home dir
cd ~
git clone https://github.com/intel/qatlib.git
# If instead you upgrade an earlier clone, follow the steps in
# "Remove the libraries and cleanup" before fetching.
# Build and install library
cd qatlib
./autogen.sh
./configure --enable-service
make -j
sudo make install
# Add your user to the "qat" group which was automatically
# created by --enable-service. Then re-login to make the change
# effective, this will also move you back into your home directory
sudo usermod -a -G qat `whoami`
sudo su -l $USER
# The library is now ready to use with your application
# You can also follow these steps to try out a sample application:
# cpa_sample_code requires a minimum of 500MB to run its compression demo
# To increase the amount of locked memory for your user to 500MB:
sudo cp /etc/security/limits.conf /etc/security/limits.conf.qatlib_bak
echo `whoami` - memlock 500000 | sudo tee -a /etc/security/limits.conf > /dev/null
# Re-login in order to make the change effective
sudo su -l $USER
# Compression sample code expects to find data files at a known location,
# so call the samples-install target to put them there
cd qatlib
sudo make samples-install
# Run it! (takes several minutes to complete)
cpa_sample_code
# No need to leave the samples installed, so cleanup
sudo make samples-uninstall
===============================================================================
Compilation and installation - detailed instructions
====================================================
1) Install compilation dependencies
If running from a distro most or all of QATlib depends are taken care
of, if not use your OS-specific commands to install the following:
tools:
gcc
make
autotools (automake, autoconf, autoconf-archive, libtool)
pkg-config
systemd-devel
nasm
Note: If nasm compiler is unavailable see
--disable-fast-crc-in-assembler option in Configuration section below
libraries:
openssl-devel
zlib-devel
numactl-devel
----------------------------------------------------------------------------
2) Download package and configure it
To generate configure script - call:
./autogen.sh
To setup the default build configuration - call:
./configure
Full list of configuration options can be found under
"Configuration options" section below or by calling:
./configure --help
To run in Managed Mode, enable qat service. This can be done
automatically during installation by calling:
./configure --enable-service
In some case, e.g. containers, it may be preferable to build without
the qat service, to avoid a dependency on systemd. This is referred to
as Standalone Mode. In this case use:
./configure --enable-systemd=no
See also the "Configuration and tuning section" below.
----------------------------------------------------------------------------
3) Configure the system
Create the "qat" group.
qat.service requires "qat" group
This is automatically created by configure --enable-service so this
step can be skipped. To manually create the group:
sudo groupadd qat
Add your user to "qat" group.
To be able to use QATlib functionalities as non root user, user must
be part of the qat group. To add your user to the qat group
sudo usermod -a -G qat <your user>
Set maximum amount of locked memory for your user.
The maximum amount of locked memory should be set correctly by
defining memlock limit which is 16MB per VF plus whatever
the application requires.
For cpa_sample_code running crypto at least 200MB are required,
if running compression set at least 500MB.
To set 500MB add a line like this
<your user> - memlock 500000
in the file /etc/security/limits.conf.
After making user changes it's necessary to re-login for them to
take effect
sudo su -l $USER
----------------------------------------------------------------------------
4) Compile and install the library
Build and install:
make -j
sudo make install
If the service was not configured to start automatically, by using
--enable-service then it will need to be started. To start it
and make persistent after reboot:
systemctl enable qat
systemctl start qat
----------------------------------------------------------------------------
5) Run sample application
Sample code is built by default.
Information on running the performance sample-code is at
./quickassist/lookaside/access_layer/src/sample_code/README.txt
Examples:
Compression sample code expects to find data files at a known
location, so call the samples-install target to put them there:
sudo make samples-install
Run all performance tests:
./cpa_sample_code
Symmetric crypto tests only:
./cpa_sample_code runTests=1
Asymmetric crypto RSA tests only:
./cpa_sample_code runTests=2
Compression tests only:
./cpa_sample_code runTests=32
Examples of other samples for a specific functionality:
Run just one symmetric cipher operation:
./cipher_sample
Run just one compress/decompress operation:
./dc_stateless_sample
Run just one chaining hash then compress operation:
./chaining_sample
----------------------------------------------------------------------------
6) Remove the libraries and cleanup
Note: These make commands should be run using the Makefiles that were
generated by the original call to the configure script. I.e. there is
no need to call ./configure again. If it is ever called again it must
be called with exactly the same options as were originally used so
whatever was installed can be correctly cleaned up.
Uninstall:
sudo make samples-uninstall #if samples installed
sudo systemctl stop qat
sudo make uninstall
Clean up:
make clean
make distclean
----------------------------------------------------------------------------
7) Configuration and tuning
There are parameters which can be tweaked to optimize for an
application's workload which might work in two modes:
managed (when qat service is enabled) and standalone (without service)
MANAGED MODE
In Managed mode, qatmgr manages allocation of the VF resources to
each process on process startup.
A sample config file can be found in quickassist/utilities/service/qat.
it can be copied to /etc/sysconfig/qat and used to set the following
parameters:
POLICY
Indicates how many VFs will be assigned to each process.
POLICY=0 or not set
(Default) One VF from each PF will be assigned to each process.
POLICY=N
N VFs will be assigned to each process.
ServicesEnabled
Tells the kernel driver which Services to enable on PFs. All VFs on
a PF have the same service.
Unset
(Default) Every even-numbered PF has sym;asym,
every odd-numbered PF has dc. I.e. PF0, PF2 ... support crypto
and PF1, PF3 ... support data compression.
ServicesEnabled=sym
All PFs, and so all VFs, have sym only.
ServicesEnabled=asym
All PFs, and so all VFs, have asym only.
ServicesEnabled=sym;asym
All PFs, and so all VFs, have sym;asym only.
ServicesEnabled=dc
All PFs, and so all VFs, have dc only.
ServicesEnabled=sym;dc
All PFs, and so all VFs, have sym;dc only.
ServicesEnabled=asym;dc
All PFs, and so all VFs, have asym;dc only.
ServicesEnabled=dcc
All PFs, and so all VFs, have dc instances enabled for chaining
operations (hash then compress) only.
Note: Throughput is lower than dc. Only use this setting when
chaining is needed.
The following examples are based on a one-socket QAT 4xxx platform
with 4 PFs, each PF has 16 VFs. Each VF enabled for sym, asym, dc
has 4 instances. Each VF enabled for mixed services like sym;asym,
sym;dc, asym;dc has 2 instances of each type, so sym;asym has 2 cy
instances (note that cy instance is and sym and asym together), sym;dc
has 2 sym instances and 2 dc instances, asym;dc has 2 asym instances
and 2 dc instances.
An instance is an abstraction used on the APIs to identify a unique path
to the hardware. In a multi-threaded process, typically at least one
instance is needed per thread.
Usage
Flexibility
On default configuration, with neither param set, each process
will be allocated one VF from each PF, so 4 VFs. It will have
2 sym;asym VFs, so 4 sym instances and 4 asym instances
and 2 dc VFs, so 8 dc instances.
The maximum number of processes is 16. With N sockets,
the maximum number of processes is still 16, each will have
instances as above * N.
Scalability and flexibility
POLICY=2. ServicesEnabled unset. Each process will be
allocated 2 VFs, 1 with sym;asym, 1 with dc. So each will have
2 sym instances, 2 asym instance and 4 dc instances.
The maximum number of processes is 32. 16 VFs * 4 PFs / POLICY.
N sockets will have 32 * N.
Scalability and flexibility for crypto symmetric or asymmetric
plus compression.
POLICY=1. ServicesEnabled sym;dc or asym;dc. Each process will be
allocated 1 VF, with sym;dc or asym;dc. So each will have
2 sym or asym instances and 2 dc instances.
The maximum number of processes is 64. 16 VFs * 4 PFs / POLICY.
N sockets will have 64 * N.
Crypto-only scalability
POLICY=1. ServicesEnabled=sym;asym. All VFs have sym;asym.
Each process will be allocated 1 VF so will have 2 sym instances
and 2 asym instances.
The maximum number of processes is 64. 16 VFs * 4 PFs.
N sockets will have 64 * N.
Note: This configuration also provides optimized-throughput for
asym.
Crypto-only optimized-throughput
POLICY=0. ServicesEnabled=sym;asym. All VFs have sym;asym.
Each process will be allocated 1 VF per PF which will have 2 sym
instances and 2 asym instances, so each process will have
(2 * number of PFs) of each instance type.
The maximum number of processes is 16.
Sym-only optimized-throughput
POLICY=0. ServicesEnabled=sym. All VFs have sym.
Each process will be allocated 1 VF per PF which will have
4 sym instances, so each process will have
(4 * number of PFs) instances.
The maximum number of processes is 16.
Compression-only scalability
POLICY=1. ServicesEnabled=dc. All VFs have dc. Each process
will be allocated 1 VF so will have 4 dc instances.
The maximum number of processes is 64. 16 VFs * 4 PFs.
N sockets will have 64 * N.
Compression-only optimized-throughput
POLICY=0. ServicesEnabled=dc. All VFs have dc.
Each process will be allocated 1 VF per PF which will have
4 dc instances, so each process will have
(4 * number of PFs) instances.
The maximum number of processes is 16.
Compression chaining scalability:
POLICY=1. ServicesEnabled=dcc. All VFs have dc with sym
capabilities but only for chaining operations. Each process
will be allocated 1 VF so will have 4 dc instances.
The maximum number of processes is 64. 16 VFs * 4 PFs.
N sockets will have 64 * N.
STANDALONE MODE
In standalone mode, there is no qat service running.
This is the preferred way to run on containers where a subset of VFs
are explicitly passed through and so no qatmgr managed resource
mgr is needed.
Each standalone process assumes VF devices which it can't open are in
use by another process and looks for unopened VFs.
In standalone mode /quickassist/utilities/service/qat_init.sh can be
called on the host to bind VFs to vfio-pci and to enable services based
on the ServicesEnabled param. In this mode the POLICY parameter should
not be set in /etc/sysconfig/qat, if it is set it will be ignored,
instead an environment variable QAT_POLICY can be set. Please note that
the QAT_POLICY=0 is not a valid setting in standalone mode. The valid
setting range starts from 1 and goes up to max PFs * 16.
Note1: The ServicesEnabled configuration requires Linux kernel v6.0
or later and applies to all PFs on the platform. The script below can be
used to view the PF/VF service config. Config can also be set per PF,
follow the instructions provided in the following link:
https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-driver-qat.
If an alternative config is set per PF following those instructions,
then neither ServicesEnabled nor POLICY should be set in
/etc/sysconfig/qat, else it will not be deterministic which instances
a process receives.
RUNNING IN A VIRTUAL MACHINE / GUEST
For virtualization setup please refer to the following link:
https://intel.github.io/quickassist/qatlib/running_in_vm.html
When passing VFs to a guest, the BDFs on the guest should facilitate qatlib
recognizing whether VFs are from the same PF or not.
So the libvirt XML file should specify that VFs from the same host
(same domain + bus) are assigned to a common (domain + bus) on the guest,
which is different to the (domain + bus) used for VFs from other PFs.
e.g. if VF on host 0000:6b:00.1 maps to <xxxx:xx>:00.1 on guest
then 0000:6b:00.6 should map to <xxxx:xx>:00.6
and 0000:7a:00.3 should map to <yyyy:yy>:00.3
The first VF, mapped to function='0x0', should also set multifunction='on'.
Sufficient VFs should be passed from the host to the guest to satisfy the
type of services and number of processes needed by the guest. See here for
more information on host configuration:
https://intel.github.io/quickassist/qatlib/configuration.html#
If using the default kernel configuration, at least 2 VFs are needed per
process so that the process has both CY and DC instances.
Set either POLICY=0 or POLICY=2 (or 4, 6, ...) in /etc/sysconfig/qat on
the guest and restart qatmgr.
SCRIPT TO VIEW QAT PF/VF SERVICE CONFIG
------------------------------------------------------------------------
#!/bin/bash
printf "%-13s | %-5s | %-12s | %-12s | %-10s\n" "VFIO GROUP" "NODE" \
"PF BDF" "VF BDF" "SERVICES"
echo "--------------------------------------------------------------"
for vfio_group in /dev/vfio/*; do
if [ $vfio_group = "/dev/vfio/vfio" ]; then
continue
fi
group=${vfio_group##*/}
# Skip the new 'devices' folder in kernel 6.14+
if [ "$group" = "devices" ]; then
continue
fi
# Check if the group is numeric (additional safety check)
if ! [[ "$group" =~ ^[0-9]+$ ]]; then
continue
fi
# assume one bdf per iommu group
bdf=$(ls /sys/kernel/iommu_groups/$group/devices/)
# Check if bdf was found
if [ -z "$bdf" ]; then
continue
fi
vendor=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/vendor)
node=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/numa_node)
did=$(cat /sys/kernel/iommu_groups/$group/devices/$bdf/device)
if [ "$vendor" != "0x8086" ]; then
continue
fi
if [ "$did" != "0x4941" ] && [ "$did" != "0x4943" ] && [ "$did" != "0x4945" ] && [ "$did" != "0x4947" ]; then
continue
fi
regex='([a-z0-9]+):([a-z0-9]+):.*'
[[ $bdf =~ $regex ]]
pf_domain=${BASH_REMATCH[1]}
pf_bus=${BASH_REMATCH[2]}
pf_bdf="$pf_domain:$pf_bus:00.0"
printf "%-15s %-7s %-14s %-14s %-10s\n" "$vfio_group" "$node" \
"$pf_bdf" "$bdf" \
"$(cat /sys/bus/pci/devices/$pf_bdf/qat/cfg_services)"
done
------------------------------------------------------------------------
Note2: As the library assumes that all VFs from the same PF have
the same services, take care when passing VFs through to a VM that all
VFs from the same Host PF are passed through with a common Domain/Bus.
===============================================================================
Full list of Configuration options
==================================
Typical examples of how to update configuration options are as follows:
./configure ICP_ANY_FLAG=value or
./configure --enable-something
If it is required to use more than one flag at once:
./configure ICP_ANY_PATH=path ICP_ANY_NAME=name --enable-something
Features flags:
Enables or disables the additional features supported by the package
--disable-option-checking
Ignores unrecognized configure options when run along with it.
--disable-FEATURE
Does not include FEATURE (same as --enable-FEATURE=no).
--enable-FEATURE[=ARG]
Includes FEATURE [ARG=yes].
--enable-silent-rules
Less verbose build output (undo: "make V=1").
--disable-silent-rules
Verbose build output (undo: "make V=0").
--enable-dependency-tracking
Does not reject slow dependency extractors.
--disable-dependency-tracking
Speeds up one-time build
--enable-icp-debug
Enables debugging.
--disable-param-check
Disables parameters checking in the top-level APIs
(Use for performance optimization).
--disable-stats
Disables statistic collection (Use for performance optimization).
--disable-fast-crc-in-assembler
Force use of C code instead of faster assembler implementation
of CRC for DC integrityCrc feature. Not recommended unless
nasm assembler compiler is unavailable.
--enable-icp-log-syslog
Enables debugging messages to be outputted to the
system log instead of standard output.
--enable-icp-trace
Enables tracing for the Cryptography API.
--enable-dc-error-simulation
Enables Data Compression Error Simulation.
--enable-hb-error-simulation
Enables Heartbeat Error Simulation.
--enable-legacy-lib-names
Enables legacy names for libraries.
--enable-systemd
Enable systemd support. Set to No to remove dependency on
systemd [default=yes] Note: --enable-service should not be used
in this case.
--enable-service
Automatically creates "qat" group and enables systemd service
during installation.
--enable-legacy-algorithms
Enable deprecated legacy crypto algorithms. See the README.md
for the list of algorithms which are deprecated by default.
--enable-icp-thread-specific-usdm
USDM allocates and handles memory specific to threads.
(For multi-thread apps, allocated memory information will be
maintained separately for each thread; employs thread local storage
feature i.e. TLS. It avoids locking that was needed when a global
data structure being used in non thread-specific implementation).
NOTE: Any memory allocated by a thread must be freed by the same
thread. If it passes the memory to other threads for use, it's
responsible for any synchronisation between those threads.
The thread which did the allocation must live until after all
threads using the memory are finished with it, as any thread
memory not yet freed may be cleaned up on termination of the thread.
--enable-icp-without-qp-submission-lock
This allows for the removal of Queue Pair (QP) submission
locks and is designed to optimize performance in environments
where QPs assigned to one thread are not shared by another thread.
In these scenarios, the frequent invocation of mutex_lock/unlock
during hardware request submissions is unnecessary and can
affect performance. By enabling this flag, the locks around QP
submissions are eliminated, thus reducing the cost of offload.
As QPs are assigned to instances, only enable this if instances
are not shared across threads, else undefined behaviour could occur.
--enable-treat-crc-from-comp-engine-as-error
When the device calculates a CRC over the uncompressed data,
it usually uses the decompression engine to do the CRC calculation.
In compression operation cases where the stored block is returned
e.g. due to input data being incompressible, the returned CRC is
calculated using the compression engine, which should give exactly
the same CRC as the decompression engine would. Enabling this option
causes any case where the CRC is calculated by the compression engine
to be reported as a CPA_DC_E2E_NO_DECOMPRESSION error, rather than as a
successful operation. Please see below link for firmware needed to support
this feature. https://intel.github.io/quickassist/RN/In-Tree/in_tree_firmware_RN.html#qat-2-0-in-tree-kernel-requirements
MAX_MR
Number of Miller Rabin rounds for prime operations. Setting this
to a smaller value reduces the memory usage required by the
driver.
Type: value
Default value: 50
===============================================================================
Common issues
=============
Additional details are available at:
https://intel.github.io/quickassist/qatlib/index.html
Issue: errors like these are seen in system logs:
* 4xxx 0000:6b:00.0: IOMMU should be enabled for SR-IOV to work
* vfio-pci: probe of 0000:6b:00.1 failed with error -22
Likely cause: Check VT-d, SR-IOV and intel_iommu settings are correct as
described above. One way to check that is to run:
dmesg | grep "Virtualization Technology"
If you get output similar to below, everything is ready, otherwise
there is something misconfigured.
"[57.503644] DMAR: Intel(R) Virtualization Technology for Directed I/O"
Issue: On running sample code:
qaeMemInit started
Open failed on /dev/vfio/15
No device found
ADF_UIO_PROXY err: icp_adf_userProcessToStart: Failed to start SSL...
Likely cause: Incorrect permissions. Use "id <your user>" to verify the
user is in the qat group and logout/login to the shell to ensure group
changes take effect.
Issue: On running sample code:
qaeMemInit started
No device found
ADF_UIO_PROXY err: icp_adf_userProcessToStart: Failed to start SSL...
Likely cause: (1) No PF driver available. Check that PFs are available and
bound to qat_4xxx: (See Supported Devices table for other devices)
sudo lspci -vvd:4940 | grep "Kernel driver in use".
Upgrade to a recent Linux Kernel.
(2) No VFs available. Check VFs are available and bound to vfio-pci
sudo lspci -vvd:4941 | grep "Kernel driver in use"
Issue: On running ./autogen.sh following warning appears:
aclocal: warning: couldn't open directory 'm4': No such file or dir...
Likely cause: This warning can be ignored, as is resolved subsequently
Issue: After upgrading to new version following error may be received:
This qatlib v21.8 received response from incompatible qatmgr v0.0
Likely cause: if qat service was already running during upgrade, it won't
be restarted automatically.
After upgrading ensure that qat service is restarted.
Issue: "DC Instances are not present" error when trying to run
compression operations, e.g. using "cpa_sample_code runTests=32"
Likely cause: QAT driver in Linux kernel before v5.17 doesn't support
compression service. Upgrade to a later kernel.
Issue: "Could not open corpus file: /usr/local/share/qat/calgary"
seen when running compression sample code.
Likely cause: sample code data files not installed at the expected location
Run: sudo make samples-install
Issue: Error "memFreeNUMA:1313 Address to be freed cannot be NULL"
seen when running cpa_sample_code.
Also dmesg | grep MEMLOCK shows messages like this:
"hvfio_pin_pages_remote: RLIMIT_MEMLOCK (204800000) exceeded"
Likely cause: Not enough locked memory. Increase the amount of locked
memory for your user – see instructions above.
After updating re-login so the changes take effect:
sudo su -l $USER
Issue: error on make install or on systemctl start qat
"Job for qat.service failed because the control process exited with
error code"
System logs (dmesg) show QAT kernel module failed with error like:
"4xxx 0000:6b:00.0: enabling device (0140 -> 0142)
QAT: authentication error (FCU_STATUS = 0x3),retry = 0
4xxx 0000:6b:00.0: Failed to load MMP
4xxx 0000:6b:00.0: Failed to load acceleration FW
4xxx 0000:6b:00.0: Resetting device qat_dev0
4xxx: probe of 0000:6b:00.0 failed with error -14"
Likely cause: firmware files not present in initramfs on module loading
early in boot process.
Fix: First make sure you have the firmware installed in /lib/firmware,
see pre-requisites section above.
In order to load the firmware, the driver must be reloaded, i.e. for
4xxx driver run:
"sudo rmmod qat_4xxx; sudo modprobe qat_4xxx; sudo systemctl start qat".
For a persistent change (on future reboots) run "sudo dracut --force".
See Supported Devices table for other driver names.
Issue: X kernel taint flag seen on SUSE from SLES15-SP4 onwards
"intel_qat: externally supported module,setting X kernel taint flag."
"qat_4xxx: externally supported module,setting X kernel taint flag."
Likely cause: This is normal on load of the kernel driver in a SUSE distro.
See explanation for X flag here:
https://www.suse.com/support/kb/doc/?id=000016321
It indicates that the modules are supported by external parties, in
this case Intel.
Fix: None necessary, qatlib can be used as normal despite this message.
Issue: make: *** No rule to make target 'sample-uninstall'. Stop.
Likely cause: There was a name change between 21.11 and 22.07, note the
extra 's'. Sorry if it causes confusion.
Fix: sudo make samples-uninstall
Issue: on Ubuntu on make install, this error may be seen:
“Failed to start qat.service: Unit qat.service not found.”
Likely cause: The qat.service file is not installed by Ubuntu in the
expected directory
Fix: Explicitly configure the systemd path to where Ubuntu has installed
the qat.service, e.g.
./configure --enable-service systemdsystemunitdir=/lib/systemd/system/
===============================================================================
Legal/Disclaimers
===================
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL(R) PRODUCTS.
NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL
PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S
TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY
WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO
SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING
TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY
PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel products are
not intended for use in medical, life saving, life sustaining, critical control
or safety systems, or in nuclear facility applications.
Intel may make changes to specifications and product descriptions at any time,
without notice.
(C) Intel Corporation 2026
* Other names and brands may be claimed as the property of others.
===============================================================================
|
