Q: How to load a ctype library in multiple Python processes? I have a golang library which I need to call from Python. I build it to a .so library and load it in Python via cdll.LoadLibrary(), it works well.
Now I need to use the library in both my main process and in a subprocess established by multiprocessing.Process(). (I understand that if I use thread this won't be a problem but I need to use Process for some reason.) But the library cannot work well in the subprocess. It just halts from time to time. My guess is that the global variables in the lib are only initiated once (in the main process), so the subprocess cannot get access to those vars. I debugged this by printing some lines in the init() function in the golang lib, it indeed only prints once even if I called cdll.LoadLibrary() respectively in both processes.
I don't know how to fix this, i.e. how to load a whole new self-contained version of the library in the subprocess. (Alternatively if I can just load the library in the main process and let the subprocess access the lib that would work for me too, but that seems to be more complicated and I don't know how to do that either.)
Any suggestion would be very appreciated.
Simplified code sample:
from ctypes import cdll
from multiprocessing import Process
...

lib = cdll.LoadLibrary("./go_lib.so")
lib.do_some_thing() # <- works well
...

def worker():
    lib = cdll.LoadLibrary("./go_lib.so")
    lib.do_some_thing() # <- won't work correctly

p = Process(target=worker)
p.start()
p.join()


A: I'd suggest that the issue you're facing is due to the library not coping with being in a program that forks.  You have a few of obvious alternatives:


*

*only use the library in child processes, like my experiment

*only instantiate it in the parent process after forking off child processes. using a Pool might make this easier, as processes are all forked when the pool is created and you can run code in them when needed.  you can load the library in the initializer

*tell multiprocessing not to fork and spawn processes instead

