=============================================================================== On Using Debian's Intel-MKL Package / FAQ =============================================================================== -- Mo Zhou , Last Update: Jan 06 2019 Table of Contents -- User Part -- (*) Guide: Installing Debian's intel-mkl packages (*) Guide: Linking againset Debian's intel-mkl libraries (*) Guide: Performance (*) Guide: Switching BLAS/LAPACK backend via update-alternatives (*) Guide: Switching BLAS/LAPACK backend via environment variable (*) Sidenote: Building custom library from MKL with selected symbols (*) Sidenote: Where are MKL documentations and examples? (*) Sidenote: some of upstream's components aren't there -- Debian Developer Part -- (*) License: Is intel-mkl redistributable? (*) Upstream's MKL source directory hierarchy? (*) Debian package layout/organization? (*) MKL's priority is 1 in the update-alternatives system (*) Known ISSUES ------------------------------------------------------------------------------- (*) Guide: Installing Debian's intel-mkl packages ------------------------------------------------------------------------------- (0) If you merely want to install a new BLAS/LAPACK alternative, i.e. $ sudo apt install libmkl-rt Only shared objects will be installed. There will be neither static libraries, nor headers. (1) If you want only shared objects, static libs, headers for single node: $ sudo apt install intel-mkl Cluster-related components aren't included. For i386 architecture this is the top level metapackage. (2) If you want (1) plus cluster support: $ sudo apt install intel-mkl-cluster This is only available on amd64. (3) If you want everything from intel-mkl, i.e. (2) plus multi-arch packages: $ sudo dpkg --add-architecture i386 $ sudo apt update; sudo apt upgrade $ sudo apt install intel-mkl-full This is only available on amd64. And it's the top level metapackage for amd64. ------------------------------------------------------------------------------- (*) Guide: Linking againset Debian's intel-mkl libraries ------------------------------------------------------------------------------- Apart from linking against MKL manually, some pkg-config files are provided by Debian packages (libmkl-dev). pkg-config files can be found under this directory: /usr/lib/x86_64-linux-gnu/pkgconfig/ Please refer upstream documentation for more detail: * https://software.intel.com/en-us/mkl-linux-developer-guide-using-the-single-dynamic-library * intel-mkl-doc /usr/share/doc/intel-mkl/common/mkl_link_line_advisor.htm * intel-mkl-linktool:i386 (command line version of the above advisor) ------------------------------------------------------------------------------- (*) Guide: Performance ------------------------------------------------------------------------------- If the performance drop is not introduced by mixing the usage of different threading libraries, tweaking environment variable MKL_NUM_THREADS might help. Refer upstream doc for more detail: https://software.intel.com/en-us/mkl-linux-developer-guide-managing-performance-and-memory https://software.intel.com/en-us/mkl-linux-developer-guide-improving-performance-for-small-size-problems ------------------------------------------------------------------------------- (*) Guide: Switching BLAS/LAPACK backend via update-alternatives ------------------------------------------------------------------------------- If you want to change the BLAS/LAPACK backend, and you have the root permission, the alternative mechanism is a good solution. First let's check the current alternative status $ update-alternatives --get-selections | grep blas $ update-alternatives --get-selections | grep lapack Then, for example, let's switch the default BLAS implementation $ sudo update-alternatives --config libblas.so.3-${DEB_HOST_MULTIARCH} In the above command line, the actual value of placeholder "${DEB_HOST_MULTIARCH}" can be found by issuing this command $ dpkg-architecture -qDEB_HOST_MULTIARCH By the way, frontends of update-alternatives such as rover(1) will also work if you prefer them. For more information about usage of Debian's alternatives mechanism,see manual page update-alternatives(1). ------------------------------------------------------------------------------- (*) Guide: Switching BLAS/LAPACK backend via environment variable ------------------------------------------------------------------------------- If you have NO root permission, but still want to change the BLAS/LAPACK backend for some program, there are still solutions. Recall that as elaborated in manual page ls.so(8), both LD_LIBRARY_PATH and LD_PRELOAD can be used as solution for this purpose. Here is the LD_LIBRARY_PATH example: $ export LD_LIBRARY_PATH=usr/lib/${DEB_HOST_MULTIARCH}/mkl/ The placeholder "${DEB_HOST_MULTIARCH}" is explained above. The above solution works because we've put two symbol links in that directory: usr/lib/${DEB_HOST_MULTIARCH}/mkl/libblas.so.3 usr/lib/${DEB_HOST_MULTIARCH}/mkl/liblapack.so.3 They are both symlinked to libmkl_rt.so . For detail see ld.so(8). ------------------------------------------------------------------------------- (*) Sidenote: Building custom library from MKL with selected symbols ------------------------------------------------------------------------------- MKL provides a builder for building custom library from MKL. The user can decide which symbol to include in this custom library. After extracting upstream tarball, this builder locates here: opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/tools/builder In the Debian package that tool is moved to /usr/share/intel-mkl/ . Besides, since Debian package doesn't retain the upstream directory structure, you need to patch the builder so that it works correctly. However, due to intel's ISSL license, we cannot install a patched builder for you. Here goes the patch: ``` --- makefile.bak 2018-05-11 14:59:29.533147763 +0000 +++ makefile 2018-05-11 15:01:37.454016074 +0000 @@ -76,14 +76,11 @@ ##------------------------------------------------------------------------------ -ifndef MKLROOT -MKLROOT = ../.. -endif - -mklia32_libpath=$(MKLROOT)/lib/ia32 -mklintel64_libpath=$(MKLROOT)/lib/intel64 -compileria32_libpath=$(MKLROOT)/../compiler/lib/ia32 -compilerintel64_libpath=$(MKLROOT)/../compiler/lib/intel64 +MKLROOT = /usr +mklia32_libpath=$(MKLROOT)/lib/i386-linux-gnu +mklintel64_libpath=$(MKLROOT)/lib/x86_64-linux-gnu +compileria32_libpath=$(MKLROOT)/lib/i386-linux-gnu +compilerintel64_libpath=$(MKLROOT)/lib/x86_64-linux-gnu #ifndef export export=user_example_list ``` ------------------------------------------------------------------------------- (*) Sidenote: Where are MKL documentations and examples? ------------------------------------------------------------------------------- Install package 'intel-mkl-doc', and open this URL in your browser: file:///usr/share/doc/intel-mkl/documentation_2019/en/mkl/ps2019/get_started.htm This is an MKL documentation overview page. However network access is required to browse further documents pointed by the page. You can find some examples and miscellaneous files here after installing the intel-mkl-doc package: /usr/share/doc/intel-mkl/ ------------------------------------------------------------------------------- (*) Sidenote: some of upstream's components aren't there ------------------------------------------------------------------------------- Indeed some files from upstream are not included in the Debian package because they are troublesome or unusable due to some reason. ------------------------------------------------------------------------------- (*) License: Is intel-mkl redistributable? ------------------------------------------------------------------------------- As declared by upstream at this page https://software.intel.com/en-us/mkl/license-faq "Yes, redistribution is allowed per the terms of the ISSL." Even if registration is required before downloading the MKL tarball, the tarball itself is re-distributable and licensed under ISSL. See communication with upstream for more detail: https://github.com/intel/mkl-dnn/issues/206 The original tarball is accessible here: https://software.intel.com/en-us/mkl Click "Free Download". The tarball will be available after registration. There are several hashsums in rules. Note that, Intel provides MKL through other channels to, e.g. Pypi, APT, RPM. ------------------------------------------------------------------------------- (*) Upstream's MKL source directory hierarchy? ------------------------------------------------------------------------------- https://software.intel.com/en-us/mkl-linux-developer-guide See section "Structure of the Intel Math Kernel Library" and "Appendix C: Directory Structure in Detail". ------------------------------------------------------------------------------- (*) Debian package layout/organization? ------------------------------------------------------------------------------- Installation and Usage of this package are explained in previous sections of this document, and they should have given you a basic idea how this package is organized. There is a detailed binary package index in debian/control. Files are carefully split into small packages. It must be pointed out that the two most important nodes in the dependency graph are libmkl-rt and libmkl-dev. Here are several important notes about the dependency graph: * libmkl-rt is a run-time dispatching library. It auto matically selects shared object from which symbols are loaded. Typically it select one shared object from each of the three layers: interface layer, threading layer, and computational layer. To reduce my burden, I grouped the children librarires of libmkl-rt into three meta packages, respectively. * libmkl-rt plus the basic set of static libraries plus the headers equal to libmkl-dev. * intel-mkl metapackage is simply a wrapper of libmkl-dev. I expect that most user pull mkl by installing intel-mkl. * intel-mkl-cluster is intel-mkl + cluster support. * intel-mkl-full is intel-mkl + cluster support + multiarch (i386) libraries + all other misc stuff (e.g. custom library builder). The python script debian/control.py generates all the .install files and all the .lintian-overrides files. It scans upstream tarball and put files in a proper package. I think the comments in the script should be enough for one to fully understand what it does. ------------------------------------------------------------------------------- (*) MKL's priority is 1 in the update-alternatives system ------------------------------------------------------------------------------- Due to possible license violation, the default priority of MKL in the alternatives mechanism is set to 1. See the following link for discussion: https://lists.debian.org/debian-science/2018/04/msg00071.html However, if the user explicitly agrees to use MKL as the default BLAS/LAPACK, libmkl_rt.so will be selected and be set in manual mode, overriding any other alternatives. In contrast, we will make sure MKL is not selected if the user rejects to use MKL as default. ------------------------------------------------------------------------------- (*) Known ISSUES ------------------------------------------------------------------------------- * [cantfix] Performance on AMD CPUs. MKL looks specific to Intel CPUs. * [wontfix] should we conflict with the MKL packages came from Intel's APT channel? I think there is no need to do that. Upstream .deb packages install stuff to /opt . There is no conflict. * [wontfix] move headers out from directory /usr/include/mkl, to /usr/include. I think we'd better not move the headers to the public space. The MKL library ships fftw3 header, which is installed at /usr/include/mkl/fftw/fftw3.h And the above location won't cause any confusion, compared to /usr/include/fftw/fftw3.h (from: libmkl-dev) /usr/include/fftw3.h (from: libfftw3-dev) If we move the MKL headers out from a private directory. * [wontfix] symbols control file. None of the shared object has SONAME. And dpkg-gensymbols won't generate symbols list. (dpkg-gensymbols)