Sequence of object cleanup and functions called by atexit in Python module
Question: I am integrating a legacy C++ library with Python using boost-python. The
legacy library has some global initialization and then the classes in it use
application wide data. I need to ensure that the shutdown function of the
legacy library is called after all wrapped objects are destroyed and thought
this might be achieved by registering a shutdown function using atexit.
However, I found that the wrapped objects are being cleaned up after atexit
calls the shutdown function, causing multiple segfaults within the legacy
library!

I can achieve the desired behavior by calling del on the wrapped objects
before exiting, but was hoping to leave deletion to Python. I have checked out
the red warning box in the [documentation of
object.**__del__**](https://docs.python.org/2/reference/datamodel.html#object.__del__),
and am wondering if my ideal world is unreachable.

Any suggestions for ensuring a shutdown method is called after all objects are
cleaned up when wrapping legacy code in a python module?

Some platform details in case they are important:

  * Python 2.7.2
  * Visual Studio 2013
  * 64-bit build

Minimal code:

    
    
    #include <iostream>
    #include <boost/python.hpp>
    
    using namespace std;
    
    namespace legacy
    {
        void initialize() { cout << "legacy::initialize" << endl; }
        void shutdown() { cout << "legacy::shutdown" << endl; }
    
        class Test
        {
        public:
            Test();
            virtual ~Test();
        };
    
        Test::Test() { }
        Test::~Test() { cout << "legacy::Test::~Test" << endl; }
    }
    
    BOOST_PYTHON_MODULE(legacy)
    {
        using namespace boost::python;
        legacy::initialize();
        class_<legacy::Test>("Test");
        def("_finalize", &legacy::shutdown);
        object atexit = object(handle<>(PyImport_ImportModule("atexit")));
        object finalize = scope().attr("_finalize");
        atexit.attr("register")(finalize);
    }
    

Once compiled, this can be run using python with the following input and
outputs being displayed:

> >>> import legacy  
>  legacy::initialize  
>  >>> test = legacy.Test()  
>  >>> ^Z  
>  legacy::shutdown  
>  legacy::Test::~Test


Answer: In short, create a guard type that will initialize and shutdown the legacy
library in its constructor and destructor, then manage the guard via a smart
pointer in each exposed object.

* * *

There are some subtle details that can make getting the destruction process
correct difficult:

  * The order of destruction for objects and objects in modules in [`Py_Finalize()`](https://docs.python.org/2/c-api/init.html#c.Py_Finalize) is random.
  * There is no finalization of a module. In particular, dynamically loaded extension modules are not unloaded.
  * The legacy API should only be shutdown once all objects using it are destroyed. However, the objects themselves may not be aware of one another.

To accomplish this, the Boost.Python objects need to coordinate when to
initialize and shutdown the legacy API. These objects also need to have
ownership over the legacy object that uses the legacy API. Using the [single
responsibility
principle](https://www.wikiwand.com/en/Single_responsibility_principle), one
can divide the responsibilities into a few classes.

One can use the [resource acquisition is
initialization](https://www.wikiwand.com/en/Resource_Acquisition_Is_Initialization)
(RAII) idiom to initialize and shutdown the legacy AP. For example, with the
following `legacy_api_guard`, when a `legacy_api_guard` object is constructed,
it will initialize the legacy API. When the `legacy_api_guard` object is
destructed, it will shutdown the legacy API.

    
    
    /// @brief Guard that will initialize or shutdown the legacy API.
    struct legacy_api_guard
    {
      legacy_api_guard()  { legacy::initialize(); }
      ~legacy_api_guard() { legacy::shutdown();   }
    };
    

As multiple objects will need to share management over when to initialize and
shutdown the legacy API, one can use a smart pointer, such as
`std::shared_ptr`, to be responsible for managing the guard. The following
example lazily initializes and shutdown the legacy API:

    
    
    /// @brief Global shared guard for the legacy API.
    std::weak_ptr<legacy_api_guard> legacy_api_guard_;
    
    /// @brief Get (or create) guard for legacy API.
    std::shared_ptr<legacy_api_guard> get_api_guard()
    {
      auto shared = legacy_api_guard_.lock();
      if (!shared)
      {
        shared = std::make_shared<legacy_api_guard>();
        legacy_api_guard_ = shared;
      }
      return shared;
    }
    

Finally, the actual type that will be embedded into the Boost.Python object
needs to obtain a handle to the legacy API guard before creating an instance
of the legacy object. Additionally, upon destruction, the legacy API guard
should be released after the legacy object has been destroyed. One non-
intrusive way to accomplish this is to use provide a custom
[HeldType](http://www.boost.org/doc/libs/1_58_0/libs/python/doc/v2/class.html#HeldType)
when exposing the legacy types to Boost.Python. When exposing the type, the
default Boost.Python generated initializers need to be suppressed, as a custom
factory function will be used instead to provide control over object creation:

    
    
    /// @brief legacy_object_holder is a smart pointer that will hold
    ///        legacy types and help guarantee the legacy API is initialized
    ///        while these objects are alive.  This smart pointer will remain
    ///        transparent to the legacy library and the user-facing Python.
    template <typename T>
    class legacy_object_holder
    {
    public:
    
      typedef T element_type;
    
      template <typename... Args>
      legacy_object_holder(Args&&... args)
        : legacy_guard_(::get_api_guard()),
          ptr_(std::make_shared<T>(std::forward<Args>(args)...))
      {}
    
      legacy_object_holder(legacy_object_holder& rhs) = default;
    
      element_type* get() const { return ptr_.get(); }
    
    private:
    
      // Order of declaration is critical here.  The guard should be
      // allocated first, then the element.  This allows for the
      // element to be destroyed first, followed by the guard.
      std::shared_ptr<legacy_api_guard> legacy_guard_;
      std::shared_ptr<element_type> ptr_;
    };
    
    /// @brief Helper function used to extract the pointed to object from
    ///        an object_holder.  Boost.Python will use this through ADL.
    template <typename T>
    T* get_pointer(const legacy_object_holder<T>& holder)
    {
      return holder.get();
    }
    
    /// Auxiliary function to make exposing legacy objects easier.
    template <typename T, typename ...Args>
    legacy_object_holder<T>* make_legacy_object(Args&&... args)
    {
      return new legacy_object_holder<T>(std::forward<Args>(args)...);
    }
    
    BOOST_PYTHON_MODULE(example)
    {
      namespace python = boost::python;
      python::class_<
          legacy::Test, legacy_object_holder<legacy::Test>, 
          boost::noncopyable>("Test", python::no_init)
        .def("__init__", python::make_constructor(
          &make_legacy_object<legacy::Test>))
        ;
    }
    

* * *

Here is a complete example [demonstrating](http://coliru.stacked-
crooked.com/a/01389d3c0e229d55) using a custom HeldType to non-intrusively
lazily guard a resource with shared management:

    
    
    #include <iostream> // std::cout, std::endl
    #include <memory> // std::shared_ptr, std::weak_ptr
    #include <boost/python.hpp>
    
    /// @brief legacy namespace that cannot be changed.
    namespace legacy {
    
    void initialize() { std::cout << "legacy::initialize()" << std::endl; }
    void shutdown()   { std::cout << "legacy::shutdown()" << std::endl;   }
    
    class Test
    {
    public:
      Test()          { std::cout << "legacy::Test::Test()" << std::endl;  }
      virtual ~Test() { std::cout << "legacy::Test::~Test()" << std::endl; }
    };
    
    void use_test(Test&) {}
    
    } // namespace legacy
    
    namespace {
    
    /// @brief Guard that will initialize or shutdown the legacy API.
    struct legacy_api_guard
    {
      legacy_api_guard()  { legacy::initialize(); }
      ~legacy_api_guard() { legacy::shutdown();   }
    };
    
    /// @brief Global shared guard for the legacy API.
    std::weak_ptr<legacy_api_guard> legacy_api_guard_;
    
    /// @brief Get (or create) guard for legacy API.
    std::shared_ptr<legacy_api_guard> get_api_guard()
    {
      auto shared = legacy_api_guard_.lock();
      if (!shared)
      {
        shared = std::make_shared<legacy_api_guard>();
        legacy_api_guard_ = shared;
      }
      return shared;
    }
    
    } // namespace 
    
    /// @brief legacy_object_holder is a smart pointer that will hold
    ///        legacy types and help guarantee the legacy API is initialized
    ///        while these objects are alive.  This smart pointer will remain
    ///        transparent to the legacy library and the user-facing Python.
    template <typename T>
    class legacy_object_holder
    {
    public:
    
      typedef T element_type;
    
      template <typename... Args>
      legacy_object_holder(Args&&... args)
        : legacy_guard_(::get_api_guard()),
          ptr_(std::make_shared<T>(std::forward<Args>(args)...))
      {}
    
      legacy_object_holder(legacy_object_holder& rhs) = default;
    
      element_type* get() const { return ptr_.get(); }
    
    private:
    
      // Order of declaration is critical here.  The guard should be
      // allocated first, then the element.  This allows for the
      // element to be destroyed first, followed by the guard.
      std::shared_ptr<legacy_api_guard> legacy_guard_;
      std::shared_ptr<element_type> ptr_;
    };
    
    /// @brief Helper function used to extract the pointed to object from
    ///        an object_holder.  Boost.Python will use this through ADL.
    template <typename T>
    T* get_pointer(const legacy_object_holder<T>& holder)
    {
      return holder.get();
    }
    
    /// Auxiliary function to make exposing legacy objects easier.
    template <typename T, typename ...Args>
    legacy_object_holder<T>* make_legacy_object(Args&&... args)
    {
      return new legacy_object_holder<T>(std::forward<Args>(args)...);
    }
    
    // Wrap the legacy::use_test function, passing the managed object.
    void legacy_use_test_wrap(legacy_object_holder<legacy::Test>& holder)
    {
      return legacy::use_test(*holder.get());
    }
    
    BOOST_PYTHON_MODULE(example)
    {
      namespace python = boost::python;
      python::class_<
          legacy::Test, legacy_object_holder<legacy::Test>, 
          boost::noncopyable>("Test", python::no_init)
        .def("__init__", python::make_constructor(
          &make_legacy_object<legacy::Test>))
        ;
    
      python::def("use_test", &legacy_use_test_wrap);
    }
    

Interactive usage:

    
    
    >>> import example
    >>> test1 = example.Test()
    legacy::initialize()
    legacy::Test::Test()
    >>> test2 = example.Test()
    legacy::Test::Test()
    >>> test1 = None
    legacy::Test::~Test()
    >>> example.use_test(test2)
    >>> exit()
    legacy::Test::~Test()
    legacy::shutdown()
    

* * *

Note that the basic overall approach is also applicable to a non-lazy
solution, where the legacy API gets initialized upon importing the module. One
would need to use a `shared_ptr` instead of a `weak_ptr`, and register a
cleanup function with `atexit.register()`:

    
    
    /// @brief Global shared guard for the legacy API.
    std::shared_ptr<legacy_api_guard> legacy_api_guard_;
    
    /// @brief Get (or create) guard for legacy API.
    std::shared_ptr<legacy_api_guard> get_api_guard()
    {
      if (!legacy_api_guard_)
      {
        legacy_api_guard_ = std::make_shared<legacy_api_guard>();
      }
      return legacy_api_guard_;
    }
    
    void release_guard()
    {
      legacy_api_guard_.reset();
    }
    
    ...
    
    BOOST_PYTHON_MODULE(example)
    {
      // Boost.Python may throw an exception, so try/catch around
      // it to initialize and shutdown legacy API on failure.
      namespace python = boost::python;
      try
      {
        ::get_api_guard(); // Initialize.  
    
        ...
    
        // Register a cleanup function to run at exit.
        python::import("atexit").attr("register")(
          python::make_function(&::release_guard)
        );
      }
      // If an exception is thrown, perform cleanup and re-throw.
      catch (const python::error_already_set&)
      {
        ::release_guard();  
        throw;
      }
    }
    

See [here](http://coliru.stacked-crooked.com/a/589af70587f62042) for a
demonstration.

