Embedding Python in C: Error when attempting to call Python code in a C callback called by Python code
Question: I have some C code that calls a Python function. This Python function accepts
an address and uses WINFUNCTYPE to eventually convert it to a function that
Python can call. The C function send as a parameter to the Python function
will eventually call another Python function. It is at this last step which
causes a crash. So in short I go from C -> Python -> C -> Python. The last C
-> Python causes a crash. I've been trying to understand the problem, but I
have been unable to.

Can someone point out my problem?

C code compiled with Visual Studio 2010 and run with the args
"c:\\...\crash.py" and "func1":

    
    
    #include <stdlib.h>
    #include <stdio.h>
    
    #include <Python.h>
    
    PyObject* py_lib_mod_dict; //borrowed
    
    void __stdcall cfunc1()
    {
        PyObject* py_func;
        PyObject* py_ret;
        int size;
        PyGILState_STATE gil_state;
    
        gil_state = PyGILState_Ensure();
        printf("Hello from cfunc1!\n");
    
        size = PyDict_Size(py_lib_mod_dict);
        printf("The dictionary has %d items!\n", size);
        printf("Calling with GetItemString\n");
        py_func = PyDict_GetItemString(py_lib_mod_dict, "func2"); //fails here when cfunc1 is called via callback... will not even go to the next line!
        printf("Done with GetItemString\n");
        py_ret = PyObject_CallFunction(py_func, 0);
    
        if (py_ret)
        {
            printf("PyObject_CallFunction from cfunc1 was successful!\n");
            Py_DECREF(py_ret);
        }
        else
            printf("PyObject_CallFunction from cfunc1 failed!\n");
    
        printf("Goodbye from cfunc1!\n");
        PyGILState_Release(gil_state);
    }
    
    int wmain(int argc, wchar_t** argv)
    {
        PyObject* py_imp_str;
        PyObject* py_imp_handle;
        PyObject* py_imp_dict; //borrowed
        PyObject* py_imp_load_source; //borrowed
        PyObject* py_dir; //stolen
        PyObject* py_lib_name; //stolen
        PyObject* py_args_tuple;
        PyObject* py_lib_mod;
        PyObject* py_func;
        PyObject* py_ret;
    
        Py_Initialize();
    
        //import our python script
        py_dir = PyUnicode_FromWideChar(argv[1], wcslen(argv[1]));
        py_imp_str = PyString_FromString("imp");
        py_imp_handle = PyImport_Import(py_imp_str);
        py_imp_dict = PyModule_GetDict(py_imp_handle); //borrowed
        py_imp_load_source = PyDict_GetItemString(py_imp_dict, "load_source"); //borrowed
        py_lib_name = PyUnicode_FromWideChar(argv[2], wcslen(argv[2]));
    
        py_args_tuple = PyTuple_New(2);
        PyTuple_SetItem(py_args_tuple, 0, py_lib_name); //stolen
        PyTuple_SetItem(py_args_tuple, 1, py_dir); //stolen
    
        py_lib_mod = PyObject_CallObject(py_imp_load_source, py_args_tuple);
        py_lib_mod_dict = PyModule_GetDict(py_lib_mod); //borrowed
    
        printf("Calling cfunc1 from main!\n");
        cfunc1();
    
        py_func = PyDict_GetItem(py_lib_mod_dict, py_lib_name);
        py_ret = PyObject_CallFunction(py_func, "(I)", &cfunc1);
    
        if (py_ret)
        {
            printf("PyObject_CallFunction from wmain was successful!\n");
            Py_DECREF(py_ret);
        }
        else
            printf("PyObject_CallFunction from wmain failed!\n");
    
        Py_DECREF(py_imp_str);
        Py_DECREF(py_imp_handle);
        Py_DECREF(py_args_tuple);
        Py_DECREF(py_lib_mod);
    
        Py_Finalize();
    
        fflush(stderr);
        fflush(stdout);
        return 0;
    }
    

Python code:

    
    
    from ctypes import *
    
    def func1(cb):
        print "Hello from func1!"
        cb_proto = WINFUNCTYPE(None)
        print "C callback: " + hex(cb)
        call_me = cb_proto(cb)
        print "Calling callback from func1."
        call_me()
        print "Goodbye from func1!"
    
    def func2():
        print "Hello and goodbye from func2!"
    

Output:

    
    
    Calling cfunc1 from main!
    Hello from cfunc1!
    The dictionary has 88 items!
    Calling with GetItemString
    Done with GetItemString
    Hello and goodbye from func2!
    PyObject_CallFunction from cfunc1 was successful!
    Goodbye from cfunc1!
    Hello from func1!
    C callback: 0x1051000
    Calling callback from func1.
    Hello from cfunc1!
    The dictionary has 88 items!
    Calling with GetItemString
    PyObject_CallFunction from wmain failed!
    

I added a PyErr_Print() to the end and this was the result:

    
    
    Traceback (most recent call last):
      File "C:\Programming\crash.py", line 9, in func1
        call_me()
    WindowsError: exception: access violation writing 0x0000000C
    

EDIT: Fixed a bug that abarnert pointed out. Output is unaffected.  
EDIT: Added in the code that resolved the bug (acquiring the GIL lock in
cfunc1). Thanks again abarnert.


Answer: The problem is this code:

    
    
    py_func = PyDict_GetItemString(py_lib_mod_dict, "func2"); //fails here when cfunc1 is called via callback... will not even go to the next line!
    printf("Done with GetItemString\n");
    py_ret = PyObject_CallFunction(py_func, 0);
    
    Py_DECREF(py_func);
    

As [the docs](http://docs.python.org/2/c-api/dict.html#PyDict_GetItemString)
say, `PyDict_GetItemString` returns a borrowed reference. So, the first time
you call here, you borrow the reference, and decref it, causing it to be
destroyed. The next time you call, you get back garbage, and try to call it.

So, to fix it, just remove the `Py_DECREF(py_func)` (or add
`Py_INCREF(py_func)` after the `pyfunc =` line).

Actually, you will usually get back a special "dead" object, so you can test
this pretty easily: put a `PyObject_Print(py_func, stdout)` after the `py_func
=` line and after the `Py_DECREF` line, and you'll probably see something like
`<function func2 at 0x10b9f1230>` the first time, `<refcnt 0 at 0x10b9f1230>`
the second and third times (and you won't see the fourth, because it'll crash
before you get there).

I don't have a Windows box handy, but changing `wmain`, `wchar_t`,
`PyUnicode_FromWideChar`, `WINFUNCTYPE`, etc. to `main`, `char`,
`PyString_FromString`, `CFUNCTYPE`, etc., I was able to build and run your
code, and I get a crash in the same place… and the fix works.

Also… shouldn't you be holding the GIL inside `cfunc1`? I don't often write
code like this, so maybe I'm wrong. And I don't get a crash with the code as-
is. Obviously, spawning a thread to run `cfunc1` _does_ crash, and
`PyGILState_Ensure`/`Release` solves that crash… but that doesn't prove you
need anything in the single-threaded case. So maybe this isn't relevant… but
if you get another crash after fixing the first one (in the threaded case,
mine looked like `Fatal Python error: PyEval_SaveThread: NULL tstate`), look
into this.

By the way, if you're new to Python extending and embedding: A huge number of
unexplained crashes are, like this one, caused by manual refcounting errors.
That's the reason things like `boost::python`, etc. exist. It's not that it's
impossible to get it right with the plain C API, just that it's so easy to get
it wrong, and you will have to get used to debugging problems like this.

