===============================================================================
                  On Using Debian's Intel-MKL Package / FAQ
===============================================================================

                    -- Mo Zhou <cdluminate@gmail.com>, Last Update: Jan 06 2019

Table of Contents

  -- User Part --

  (*) Guide: Installing Debian's intel-mkl packages
  (*) Guide: Linking againset Debian's intel-mkl libraries
  (*) Guide: Performance
  (*) Guide: Switching BLAS/LAPACK backend via update-alternatives
  (*) Guide: Switching BLAS/LAPACK backend via environment variable
  (*) Sidenote: Building custom library from MKL with selected symbols
  (*) Sidenote: Where are MKL documentations and examples?
  (*) Sidenote: some of upstream's components aren't there

  -- Debian Developer Part --

  (*) License: Is intel-mkl redistributable?
  (*) Upstream's MKL source directory hierarchy?
  (*) Debian package layout/organization?
  (*) MKL's priority is 1 in the update-alternatives system
  (*) Known ISSUES

-------------------------------------------------------------------------------
(*) Guide: Installing Debian's intel-mkl packages
-------------------------------------------------------------------------------

(0) If you merely want to install a new BLAS/LAPACK alternative, i.e.

      $ sudo apt install libmkl-rt

    Only shared objects will be installed.
    There will be neither static libraries, nor headers.

(1) If you want only shared objects, static libs, headers for single node:

      $ sudo apt install intel-mkl

    Cluster-related components aren't included.
    For i386 architecture this is the top level metapackage.

(2) If you want (1) plus cluster support:

      $ sudo apt install intel-mkl-cluster

    This is only available on amd64.

(3) If you want everything from intel-mkl, i.e. (2) plus multi-arch packages:

      $ sudo dpkg --add-architecture i386
      $ sudo apt update; sudo apt upgrade
      $ sudo apt install intel-mkl-full

    This is only available on amd64.
    And it's the top level metapackage for amd64.

-------------------------------------------------------------------------------
(*) Guide: Linking againset Debian's intel-mkl libraries
-------------------------------------------------------------------------------

Apart from linking against MKL manually, some pkg-config files are provided
by Debian packages (libmkl-dev). pkg-config files can be found under this
directory:

    /usr/lib/x86_64-linux-gnu/pkgconfig/

Please refer upstream documentation for more detail:

 * https://software.intel.com/en-us/mkl-linux-developer-guide-using-the-single-dynamic-library
 * intel-mkl-doc /usr/share/doc/intel-mkl/common/mkl_link_line_advisor.htm
 * intel-mkl-linktool:i386 (command line version of the above advisor)

-------------------------------------------------------------------------------
(*) Guide: Performance
-------------------------------------------------------------------------------

If the performance drop is not introduced by mixing the usage of different
threading libraries, tweaking environment variable MKL_NUM_THREADS might
help. Refer upstream doc for more detail:

https://software.intel.com/en-us/mkl-linux-developer-guide-managing-performance-and-memory
https://software.intel.com/en-us/mkl-linux-developer-guide-improving-performance-for-small-size-problems

-------------------------------------------------------------------------------
(*) Guide: Switching BLAS/LAPACK backend via update-alternatives
-------------------------------------------------------------------------------

If you want to change the BLAS/LAPACK backend, and you have the root
permission, the alternative mechanism is a good solution.

First let's check the current alternative status

  $ update-alternatives --get-selections | grep blas
  $ update-alternatives --get-selections | grep lapack

Then, for example, let's switch the default BLAS implementation

  $ sudo update-alternatives --config libblas.so.3-${DEB_HOST_MULTIARCH}

In the above command line, the actual value of placeholder
"${DEB_HOST_MULTIARCH}" can be found by issuing this command

  $ dpkg-architecture -qDEB_HOST_MULTIARCH

By the way, frontends of update-alternatives such as rover(1) will also work if
you prefer them. For more information about usage of Debian's alternatives
mechanism,see manual page update-alternatives(1).

-------------------------------------------------------------------------------
(*) Guide: Switching BLAS/LAPACK backend via environment variable
-------------------------------------------------------------------------------

If you have NO root permission, but still want to change the BLAS/LAPACK
backend for some program, there are still solutions. Recall that as elaborated
in manual page ls.so(8), both LD_LIBRARY_PATH and LD_PRELOAD can be used as
solution for this purpose. Here is the LD_LIBRARY_PATH example:

  $ export LD_LIBRARY_PATH=usr/lib/${DEB_HOST_MULTIARCH}/mkl/

The placeholder "${DEB_HOST_MULTIARCH}" is explained above. The above
solution works because we've put two symbol links in that directory:

  usr/lib/${DEB_HOST_MULTIARCH}/mkl/libblas.so.3
  usr/lib/${DEB_HOST_MULTIARCH}/mkl/liblapack.so.3

They are both symlinked to libmkl_rt.so . For detail see ld.so(8).

-------------------------------------------------------------------------------
(*) Sidenote: Building custom library from MKL with selected symbols
-------------------------------------------------------------------------------

MKL provides a builder for building custom library from MKL.
The user can decide which symbol to include in this custom library.
After extracting upstream tarball, this builder locates here:

  opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/tools/builder

In the Debian package that tool is moved to /usr/share/intel-mkl/ . Besides,
since Debian package doesn't retain the upstream directory structure, you
need to patch the builder so that it works correctly. However, due to intel's
ISSL license, we cannot install a patched builder for you. Here goes the
patch:

```
--- makefile.bak	2018-05-11 14:59:29.533147763 +0000
+++ makefile	2018-05-11 15:01:37.454016074 +0000
@@ -76,14 +76,11 @@
 
 ##------------------------------------------------------------------------------
 
-ifndef MKLROOT
-MKLROOT = ../..
-endif
-
-mklia32_libpath=$(MKLROOT)/lib/ia32
-mklintel64_libpath=$(MKLROOT)/lib/intel64
-compileria32_libpath=$(MKLROOT)/../compiler/lib/ia32
-compilerintel64_libpath=$(MKLROOT)/../compiler/lib/intel64
+MKLROOT = /usr
+mklia32_libpath=$(MKLROOT)/lib/i386-linux-gnu
+mklintel64_libpath=$(MKLROOT)/lib/x86_64-linux-gnu
+compileria32_libpath=$(MKLROOT)/lib/i386-linux-gnu
+compilerintel64_libpath=$(MKLROOT)/lib/x86_64-linux-gnu
 
 #ifndef export
 export=user_example_list
```

-------------------------------------------------------------------------------
(*) Sidenote: Where are MKL documentations and examples?
-------------------------------------------------------------------------------

Install package 'intel-mkl-doc', and open this URL in your browser:

  file:///usr/share/doc/intel-mkl/documentation_2020/en/mkl/ps2020/get_started.htm

This is an MKL documentation overview page. However network access
is required to browse further documents pointed by the page.

You can find some examples and miscellaneous files here after installing
the intel-mkl-doc package:

  /usr/share/doc/intel-mkl/

-------------------------------------------------------------------------------
(*) Sidenote: some of upstream's components aren't there
-------------------------------------------------------------------------------

Indeed some files from upstream are not included in the Debian package because
they are troublesome or unusable due to some reason.

-------------------------------------------------------------------------------
(*) License: Is intel-mkl redistributable?
-------------------------------------------------------------------------------

As declared by upstream at this page

  https://software.intel.com/en-us/mkl/license-faq
  "Yes, redistribution is allowed per the terms of the ISSL."

Even if registration is required before downloading the MKL tarball,
the tarball itself is re-distributable and licensed under ISSL.
See communication with upstream for more detail:

  https://github.com/intel/mkl-dnn/issues/206

The original tarball is accessible here:

  https://software.intel.com/en-us/mkl

Click "Free Download". The tarball will be available after registration.
There are several hashsums in rules.

Note that, Intel provides MKL through other channels to, e.g. Pypi, APT, RPM.

-------------------------------------------------------------------------------
(*) Upstream's MKL source directory hierarchy?
-------------------------------------------------------------------------------

https://software.intel.com/en-us/mkl-linux-developer-guide

See section "Structure of the Intel Math Kernel Library" and
"Appendix C: Directory Structure in Detail".

-------------------------------------------------------------------------------
(*) Debian package layout/organization?
-------------------------------------------------------------------------------

Installation and Usage of this package are explained in previous sections of
this document, and they should have given you a basic idea how this package is
organized.

There is a detailed binary package index in debian/control. Files are carefully
split into small packages. It must be pointed out that the two most important
nodes in the dependency graph are libmkl-rt and libmkl-dev. Here are several
important notes about the dependency graph:

  * libmkl-rt is a run-time dispatching library. It auto matically selects
    shared object from which symbols are loaded. Typically it select one shared
    object from each of the three layers: interface layer, threading layer, and
    computational layer. To reduce my burden, I grouped the children librarires
    of libmkl-rt into three meta packages, respectively.

  * libmkl-rt plus the basic set of static libraries plus the headers equal
    to libmkl-dev.

  * intel-mkl metapackage is simply a wrapper of libmkl-dev. I expect that most
    user pull mkl by installing intel-mkl.

  * intel-mkl-cluster is intel-mkl + cluster support.

  * intel-mkl-full is intel-mkl + cluster support + multiarch (i386) libraries
    + all other misc stuff (e.g. custom library builder).

The python script debian/control.py generates all the .install files and all
the .lintian-overrides files. It scans upstream tarball and put files in a
proper package. I think the comments in the script should be enough for one to
fully understand what it does.

-------------------------------------------------------------------------------
(*) MKL's priority is 1 in the update-alternatives system
-------------------------------------------------------------------------------

Due to possible license violation, the default priority of MKL in the
alternatives mechanism is set to 1. See the following link for discussion:

  https://lists.debian.org/debian-science/2018/04/msg00071.html

However, if the user explicitly agrees to use MKL as the default BLAS/LAPACK,
libmkl_rt.so will be selected and be set in manual mode, overriding any
other alternatives. In contrast, we will make sure MKL is not selected
if the user rejects to use MKL as default.

-------------------------------------------------------------------------------
(*) Known ISSUES
-------------------------------------------------------------------------------

* [cantfix] Performance on AMD CPUs. MKL looks specific to Intel CPUs.

* [wontfix] should we conflict with the MKL packages came from Intel's APT
  channel? I think there is no need to do that. Upstream .deb packages
  install stuff to /opt . There is no conflict.

* [wontfix] move headers out from directory /usr/include/mkl, to /usr/include.
  I think we'd better not move the headers to the public space. The MKL
  library ships fftw3 header, which is installed at

      /usr/include/mkl/fftw/fftw3.h

  And the above location won't cause any confusion, compared to

      /usr/include/fftw/fftw3.h  (from: libmkl-dev)
      /usr/include/fftw3.h       (from: libfftw3-dev)

  If we move the MKL headers out from a private directory.

* [wontfix] symbols control file. None of the shared object has SONAME.
  And dpkg-gensymbols won't generate symbols list. (dpkg-gensymbols)