Python extension (Boost.Python & Py++) and dlopen confusion
Question: I'm wrapping a C++ project with Py++/Boost.Python under Windows and Linux.
Everything in Windows is working fine, but I'm a bit confused over the
behavior in Linux. The C++ project is built into a single shared library
called libsimif, but I'd like to split it up into 3 separate extension
modules. For simplicity, I'll only discuss two of them, since the behavior for
the third is identical. The first, called storage contains definitions of data
structures. It has no dependencies on anything defined in either of the other
two extension modules. The second module, control, uses data structures that
are defined in storage. On the C++ side of things, the headers and source
files for storage and control are in entirely different directories. I've
tried a number of different configurations to build the extensions, but one
thing that has remained consistent is that for storage, I am only generating
Py++ wrappers for the headers included in the storage directory and only
building source files in that directory along with the Py++ generated sources.
Ditto for the control extension.

The current configuration that I am using that works passes in libsimif as a
library to the distutils.Extension constructor. Then before starting Python, I
need to ensure that libsimif is found in LD_LIBRARY_PATH. Then I can launch
Python and import either module (or from them) and everything works as-
expected. Here is some sample output from this working configuration:

    
    
    >>> import ast.simif.model_io.storage as storage
    >>> import ast.simif.model_io.control as control
    >>> dir(storage)
    ['DiscreteStore', 'PulseStore', 'RtStore', 'SerialStore', 'SharedMemoryBuilder', 'SharedMemoryDeleter', 'SpaceWireStore', '__doc__', '__file__', '__name__', '__package__']
    >>> dir(control)
    ['DiscreteController', 'ModelIoController', 'PulseController', 'RtController', 'SerialController', 'SpaceWireController', '__doc__', '__file__', '__name__', '__package__']
    >>> storage.__file__
    'ast/simif/model_io/storage.so'
    >>> control.__file__
    'ast/simif/model_io/control.so'
    

As you can see, both modules have their own shared library and unique set of
symbols. Now here is why I am confused. In Linux, we've always set the dlopen
flags to include RTLD_NOW and RTLD_GLOBAL. If I do that, this is what happens:

    
    
    >>> import sys
    >>> import DLFCN
    >>> sys.setdlopenflags(DLFCN.RTLD_NOW | DLFCN.RTLD_GLOBAL)
    >>> import ast.simif.model_io.storage as storage
    >>> import ast.simif.model_io.control as control
    __main__:1: RuntimeWarning: to-Python converter for DiscreteStore::FrameData already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for PulseStore::FrameData already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for RtStore::Link already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for RtStore::FrameData already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for RtStore::RtData already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for SerialStore::FrameData already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for SharedMemoryBuilder already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for SharedMemoryDeleter already registered; second conversion method ignored.
    >>> dir(storage)
    ['DiscreteStore', 'PulseStore', 'RtStore', 'SerialStore', 'SharedMemoryBuilder', 'SharedMemoryDeleter', 'SpaceWireStore', '__doc__', '__file__', '__name__', '__package__']
    >>> dir(control)
    ['DiscreteStore', 'PulseStore', 'RtStore', 'SerialStore', 'SharedMemoryBuilder', 'SharedMemoryDeleter', '__doc__', '__file__', '__name__', '__package__']
    >>> storage.__file__
    'ast/simif/model_io/storage.so'
    >>> control.__file__
    'ast/simif/model_io/control.so'
    

So, here storage imports ok, but control complains about a bunch of duplicate
registrations. Then when inspecting the modules, control is completely wrong.
It's like it tried to import storage twice even though **file** reports the
correct shared libraries. Perhaps not surprising, if I change the import order
and import control ahead of storage, this is what happens:

    
    
    >>> import sys
    >>> import DLFCN
    >>> sys.setdlopenflags(DLFCN.RTLD_NOW | DLFCN.RTLD_GLOBAL)
    >>> import ast.simif.model_io.control as control
    >>> dir(control)
    ['DiscreteController', 'ModelIoController', 'PulseController', 'RtController', 'SerialController', 'SpaceWireController', '__doc__', '__file__', '__name__', '__package__']
    >>> import ast.simif.model_io.storage as storage
    __main__:1: RuntimeWarning: to-Python converter for DiscreteController already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for PulseController already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for RtController already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for SerialController already registered; second conversion method ignored.
    __main__:1: RuntimeWarning: to-Python converter for SpaceWireController already registered; second conversion method ignored.
    >>> dir(storage)
    ['DiscreteController', 'ModelIoController', 'PulseController', 'RtController', 'SerialController', 'SpaceWireController', 'SpaceWireStore', '__doc__', '__file__', '__name__', '__package__']
    

Similar behavior, but now the storage import is FUBAR. Does anyone understand
what is going on here?

I'm using:

  * x64 Python 2.6.6 on x64 RHEL6. Gcc version 4.4.6 
  * x64 Python 2.6.5 on x64 RHEL5. Gcc version 4.1.2


Answer: Turns out this was actually due to a quirk with how Boost.Python registration
code is generated when using balanced_split_module in Py++.
balanced_split_module basically splits up all of the registration code into a
fixed number of source files, each with its own registration function. The
source files are named using the extension name plus the generated file number
(e.g. _.cpp, but the gotcha is that the actual functions they contain do not
contain the extension name and are just a simple register_1(), register_2(),
etc. This is find and dandy when you are only importing a single module or not
making making a module's symbols global. What happens in this case is that
when you set RTLD_GLOBAL the first module import successfully, but then all
subsequent modules will call the registration functions that were loaded in as
part of the initial module.

