Including external shared intel's mkl library in c extension for python
Question: I wrote a python c extension that uses routine from Intel's math kernel
library (mkl). This is the first time that I write a c extension. I just
learned that today.

The c extension compiled. But when I import it in python, it says undefined
symbol, and can't find a function that is defined in the mkl.

How to include any external c library in a python c extension?

Thank you for your help.

mkl_helper.c:

    
    
    #include "Python.h"
    #include "numpy/arrayobject.h"
    #include "mkl.h"
    
    static PyObject* test4 (PyObject *self, PyObject *args)
    {
        // test4 (m, n,
        //        a, ja, ia,
        //        c, jc, ic)
    
        PyArrayObject *shape_array;
        PyArrayObject *a_array;   // csr_matrix.data
        PyArrayObject *ja_array;  // csr_matrix.indices
        PyArrayObject *ia_array;  // csr_matrix.indptr
        PyArrayObject *c_array;
        PyArrayObject *jc_array;
        PyArrayObject *ic_array;
    
        if (!PyArg_ParseTuple(args, "O!O!O!O!O!O!O!", 
                                &PyArray_Type, &shape_array,
                                &PyArray_Type, &a_array,
                                &PyArray_Type, &ja_array,
                                &PyArray_Type, &ia_array,
                                &PyArray_Type, &c_array,
                                &PyArray_Type, &jc_array,
                                &PyArray_Type, &ic_array))
        {
            return NULL;
        }
    
        long  * ptr_int     = shape_array->data;
        int m               = ptr_int[0];
        int n               = ptr_int[1];
        int k               = n;
    
        float *  a_data_ptr =  a_array->data;
        float * ja_data_ptr = ja_array->data;
        float * ia_data_ptr = ia_array->data;
        float *  c_data_ptr =  c_array->data;
        float * jc_data_ptr = jc_array->data;
        float * ic_data_ptr = ic_array->data;
    
        char trans  = 'T';
        int sort    = 0;
        int nzmax   = n*n;
        int info    = -3;
        int request = 0;
    
        mkl_scsrmultcsr(&trans, &request, &sort,
           &m, &n, &k,
           a_data_ptr, ja_data_ptr, ia_data_ptr,
           a_data_ptr, ja_data_ptr, ia_data_ptr,
           c_data_ptr, jc_data_ptr, ic_data_ptr,
           &nzmax, &info);
    
        return PyInt_FromLong(info);
    
    }
    
    static struct PyMethodDef methods[] = {
        {"test4", test4, METH_VARARGS, "test2(arr1)\n take a numpy array and return its shape as a tuple"},
        {NULL, NULL, 0, NULL}
    };
    
    PyMODINIT_FUNC
    initmkl_helper (void)
    {
        (void)Py_InitModule("mkl_helper", methods);
        import_array();
    }
    

setup.py:

    
    
    from distutils.core import setup, Extension
    import numpy as np
    
    ext_modules = [ Extension('mkl_helper', sources = ['mkl_helper.c']) ]
    
    setup(
            name = 'mkl_helper',
            version = '1.0',
            include_dirs = [np.get_include()], #Add Include path of numpy
            ext_modules = ext_modules
    )
    

test.py:

    
    
    import mkl_helper
    

result of running test.py:

    
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: /home/rxu/local/lib/python2.7/site-packages/mkl_helper.so: undefined symbol: mkl_scsrmultcsr
    

Update 6/16/2016:

this seems to be useful:

1.12. Providing a C API for an Extension Module in
<https://docs.python.org/2/extending/extending.html> says even including one c
extension in another c extension can have problems if linked as share library.
so, i guess I have to link the mkl as a static library? or add inlcude mkl.h
to the python.h?

But then, in python (without c), I can use
ctypes.cdll.LoadLibrary("./mkl_rt.so") to load the mkl's shared library and
then use c function from the shared library without problem (as in
[here](http://stackoverflow.com/questions/37536106/directly-use-intel-mkl-
library-on-scipy-sparse-matrix-to-calculate-a-dot-a-t-wit)). Yet Python/C api
cannot do the same thing in c?

For linking external c library staticly, the setup.py might need: extra
objects in class distutils.core.Extensions at
<https://docs.python.org/2/distutils/apiref.html?highlight=include#distutils.ccompiler.CCompiler.add_include_dir>

Related question about cython with no answer: [Combining Cython with
MKL](http://stackoverflow.com/questions/33596817/combining-cython-with-mkl)

This one seems more helpful: [Python, ImportError: undefined symbol:
g_utf8_skip](http://stackoverflow.com/questions/27149849/python-importerror-
undefined-symbol-g-utf8-skip)

This one use dlopen which is deprecated: [Undefined Symbol in C++ When Loading
a Python Shared Library](http://stackoverflow.com/questions/8302810/undefined-
symbol-in-c-when-loading-a-python-shared-library?rq=1)


Answer: oopcode's answer in [Python, ImportError: undefined symbol:
g_utf8_skip](http://stackoverflow.com/questions/27149849/python-importerror-
undefined-symbol-g-utf8-skip) works. The situation improved with the
following.

importing the c extension into python has no error. Calling the c extension
from python give the following error: Intel MKL FATAL ERROR: Cannot load
libmkl_mc.so or libmkl_def.so.

I remember when I manually compiled numpy with mkl, the site.cfg file asked
for library path and include path for the intel's mkl. Guess I need to add the
library path to the extra_link_args as well... But that didn't work.

Someone with anaconda has that error too as in
[here](https://github.com/SiLab-Bonn/pyBAR/issues/54). Similar case at intel's
forum [here](https://software.intel.com/en-us/forums/intel-math-kernel-
library/topic/296094).

This stackoverflow question says extra_compile_args is also needed: [How to
pass flag to gcc in Python setup.py
script](http://stackoverflow.com/questions/1676384/how-to-pass-flag-to-gcc-in-
python-setup-py-script)

setup.py

    
    
    from distutils.core import setup, Extension
    import numpy as np
    
    extra_link_args=["-I", "(intel's dir)/intel/compilers_and_libraries_2016.3.210/linux/mkl/include", "-L", "(intel's dir)/intel/mkl/lib/intel64/libmkl_mc.so", "-mkl"] 
    ext_modules = [ Extension('mkl_helper', sources = ['mkl_helper.c'], extra_link_args=extra_link_args) ]
    
    
    setup(
            name = 'mkl_helper',
            version = '1.0',
            include_dirs = [np.get_include()], #Add Include path of numpy
            ext_modules = ext_modules
    )
    

Update: I finally got it working as in [here](https://software.intel.com/en-
us/forums/intel-math-kernel-library/topic/640387) But mkl stillll just use
only one of the 12 cpu.

