Boost Python, propagate C++ callbacks to Python causing segmentation fault
Question: I have the following listener in C++ that receives a Python object to
propagate the callbacks.

    
    
    class PyClient {
        private:
            std::vector<DipSubscription *> subs;
    
            subsFactory *sub;
    
            class GeneralDataListener: public SubscriptionListener {
                private:
                    PyClient * client;
    
                public:
                    GeneralDataListener(PyClient *c):client(c){
                        client->pyListener.attr("log_message")("Handler created");
                    }
    
                    void handleMessage(Subscription *sub, Data &message) {
                        // Lock the execution of this method
                        PyGILState_STATE state = PyGILState_Ensure();
                        client->pyListener.attr("log_message")("Data received for topic");
                        ...
                        // This method ends modifying the value of the Python object
                        topicEntity.attr("save_value")(valueKey, extractDipValue(valueKey.c_str(), message))
                        // Release the lock
                        PyGILState_Release(state);
                    }
    
                    void connected(Subscription *sub) {
                        client->pyListener.attr("connected")(sub->getTopicName());
                    }
    
                    void disconnected(Subscription *sub, char* reason) {
                        std::string s_reason(reason);
                        client->pyListener.attr("disconnected")(sub->getTopicName(), s_reason);
                    }
    
                    void handleException(Subscription *sub, Exception &ex) {
                        client->pyListener.attr("handle_exception")(sub->getTopicName())(ex.what());
                    }
            };
    
            GeneralDataListener *handler;
    
        public:
            python::object pyListener;
    
    
            PyClient(python::object pyList): pyListener(pyList) {
                std::ostringstream iss;
                iss << "Listener" << getpid();
                sub = Sub::create(iss.str().c_str());
                createSubscriptions();
            }
    
            ~PyClient() {
                for (unsigned int i = 0; i < subs.size(); i++) {
                    if (subs[i] == NULL) {
                        continue;
                    }
    
                    sub->destroySubscription(subs[i]);
                }
            }
    };
    
    
    BOOST_PYTHON_MODULE(pytest)
    {
        // There is no need to expose more methods as will be used as callbacks
        Py_Initialize();
        PyEval_InitThreads();
        python::class_<PyClient>("PyClient",     python::init<python::object>())
            .def("pokeHandler", &PyClient::pokeHandler);
    };
    

Then, I have my Python program, which is like this:

    
    
    import sys
    import time
    
    import pytest
    
    
    class Entity(object):
        def __init__(self, entity, mapping):
            self.entity = entity
            self.mapping = mapping
            self.values = {}
            for field in mapping:
                self.values[field] = ""
    
            self.updated = False
    
        def save_value(self, field, value):
            self.values[field] = value
            self.updated = True
    
    
    class PyListener(object):
        def __init__(self):
            self.listeners = 0
            self.mapping = ["value"]
    
            self.path_entity = {}
            self.path_entity["path/to/node"] = Entity('Name', self.mapping)
    
        def connected(self, topic):
            print "%s topic connected" % topic
    
        def disconnected(self, topic, reason):
            print "%s topic disconnected, reason: %s" % (topic, reason)
    
        def handle_message(self, topic):
            print "Handling message from topic %s" % topic
    
        def handle_exception(self, topic, exception):
            print "Exception %s in topic %s" % (exception, topic)
    
        def log_message(self, message):
           print message
    
        def sample(self):
            for path, entity in self.path_entity.iteritems():
                if not entity.updated:
                    return False
    
                sample = " ".join([entity.values[field] for field in dip_entity.mapping])
                print "%d %s %d %s" % (0, entity.entity, 4324, sample)
                entity.updated = False
    
            return True
    
    
    if __name__ == "__main__":
        sys.settrace(trace)
        py_listener = PyListener()
        sub = pytest.PyClient(py_listener)
    
        while True:
            if py_listener.sample():
                break
    

So, finally, my problem seems to be that when I start running the while True
in the Python program the script gets stuck checking if the entity is updated,
and randomly, when the C++ listener tries to invoke the callback I get a
segmentation fault.

The same if I just try time.sleep in the python script and call sample time by
time. I know it will be solved if I call sample from the C++ code, but this
script will be run by other Python module that will call the sample method
given a specific delay.So the expected functioning will be for the C++ to
update the value of the entities and the Python script to just read them.

I've debug the error with gdb, but the stack trace I'm getting is not much
explanatory:

    
    
    #0  0x00007ffff7a83717 in PyFrame_New () from /lib64/libpython2.7.so.1.0
    #1  0x00007ffff7af58dc in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
    #2  0x00007ffff7af718d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
    #3  0x00007ffff7af7292 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
    #4  0x00007ffff7b106cf in run_mod () from /lib64/libpython2.7.so.1.0
    #5  0x00007ffff7b1188e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
    #6  0x00007ffff7b12b19 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
    #7  0x00007ffff7b23b1f in Py_Main () from /lib64/libpython2.7.so.1.0
    #8  0x00007ffff6d50af5 in __libc_start_main () from /lib64/libc.so.6
    #9  0x0000000000400721 in _start ()
    

And if debug with sys.trace inside Python the last line before the
segmentation fault is always in the sample method, but it may vary.

I'm not sure how can I solve this communication problems, so any advice in the
right direction will be much appreciated.

**Edit** Modify the PyDipClient reference to PyClient.

What is happening is I start the program from the Python main method, if then
the C++ listener tries to callback the Python listener it crashes with the
segmentation fault error, the only thread I believe is created is when I
create a subscription, but that is code from inside a library that I don't
know how is working exactly.

If I remove all the callbacks to the Python listener, and force the methods
from Python (like calling the pokehandler) everything is working perfectly.


Answer: The most likely culprit is that the [Global Interpreter
Lock](http://wiki.python.org/moin/GlobalInterpreterLock) (GIL) is not being
held by a thread when it is invoking Python code, resulting in undefined
behavior. Verify all paths that make Python calls, such as
`GeneralDataListener`'s functions, acquire the GIL before invoking Python
code. If copies of `PyClient` are being made, then `pyListener` needs to be
managed in a manner that allows the GIL to be held when it is copied and
destroyed.

Furthermore, consider the [rule of
three](http://stackoverflow.com/q/4172722/1053968) for `PyClient`. Do the
copy-constructor and assignment operator need to do anything with regards to
the subscription?

* * *

The GIL is a mutex around the CPython interpreter. This mutex prevents
parallel operations to be performed on Python objects. Thus, at any point in
time, a max of one thread, the one that has acquired the GIL, is allowed to
perform operations on Python objects. When multiple threads are present,
invoking Python code whilst not holding the GIL results in undefined behavior.

C or C++ threads are sometimes referred to as alien threads in the Python
documentation. The Python interpreter has no ability to control the alien
thread. Therefore, alien threads are responsible for managing the GIL to
permit concurrent or parallel execution with Python threads.

In the current code:

  * `GeneralDataListener::handle_message()` manages the GIL in a non-exception safe manner. For example, if the listener's `log_message()` method throws an exception, the stack will unwind and not release the GIL as `PyGILState_Release()` will not be invoked.
    
        void handleMessage(...)
    {
      PyGILState_STATE state = PyGILState_Ensure();
      client->pyListener.attr("log_message")(...);
      ...
    
      PyGILState_Release(state); // Not called if Python throws.
    }
    

  * `GeneralDataListener::connected()`, `GeneralDataListener:: disconnected()`, and `GeneralDataListener:: handleException()` are explicitly invoking Python code, but do not explicitly manage the GIL. If the caller does not own the GIL, then undefined behavior is invoked as Python code is being executed without the GIL.
    
        void connected(...)
    {
      // GIL not being explicitly managed.
      client->pyListener.attr("connected")(...);
    }
    

  * `PyClient`'s implicitly created copy-constructor and assignment operator do not manage the GIL, but may indirectly invoke Python code when copying the `pyListener` data member. If copies are being made, then the caller needs to hold the GIL when the `PyClient::pyListener` object is being copied and destroyed. If the `pyListener` is not managed on the free space, then the caller must be Python aware and have acquired the GIL during the destruction of the entire `PyClient` object.

To resolve these, consider:

  * Using a [Resource Acquisition Is Initialization](http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization) (RAII) guard class to help manage the GIL in an exception safe manner. For example, with the following gil_lock class, when a gil_lock object is created, the calling thread will acquire the GIL. When the gil_lock object is destructed, it releases the GIL
    
        /// @brief RAII class used to lock and unlock the GIL.
    class gil_lock
    {
    public:
      gil_lock()  { state_ = PyGILState_Ensure(); }
      ~gil_lock() { PyGILState_Release(state_);   }
    private:
      PyGILState_STATE state_;
    };
    
    ...
    
    void handleMessage(...)
    {
      gil_lock lock;
      client->pyListener.attr("log_message")(...);
      ...
    }
    

  * Explicitly manage the GIL in any code path that is invokes Python code from within an alien thread.
    
        void connected(...)
    {
      gil_lock lock;
      client->pyListener.attr("connected")(...);
    }
    

  * Making `PyClient` non-copyable or explicitly creating the copy-constructor and assignment operator. If copies are being made, then change `pyListener` to be held by a type that allows for explicit destruction while the GIL is being held. One solution is to use a `boost::shared_ptr<python::object>` that manages a copy of the `python::object` provided to the `PyClient` during construction, and has a custom deleter that is GIL aware. Alternatively, one could use something like [`boost::optional`](http://www.boost.org/doc/libs/1_58_0/libs/optional/doc/html/index.html).
    
        class PyClient
    {
    public:
    
      PyClient(const boost::python::object& object)
        : pyListener(
            new boost::python::object(object),  // GIL locked, so copy.
            [](boost::python::object* object)   // Delete needs GIL.
            {
              gil_lock lock;
              delete object;
            }
          )
      {
        ...
      }
    
    private:
      boost::shared_ptr<boost::python::object> pyListener;;
    };
    

Note that by managing the `boost::python::object` on the free-space, one can
freely copy the `shared_ptr` without holding the GIL. On the other hand, if
one was using something like `boost::optional` to manage the Python object,
then one would need to hold the GIL during copy-construction, assignment, and
destruction.

Consider reading [this](http://stackoverflow.com/a/20828366/1053968) answer
for more details on callbacks into Python and subtle details, such as GIL
management during copy-construction and destruction.

