Title: How to parallelize a function that returns a lambda
Tags: python;lambda;python-multithreading
Question: I have a function that generates a lambda and places it into a dictionary, along these lines:
class Example:
def __init__(self):
self.functions = {}
def create_function(self, key):
(code that sets up parameters)
func = scipy.interpolate.interp2d(x, y, z)     # returns a function for interpolation
self.functions[key] = lambda a, b: func(b, a)  # axes need to be switched
In a different part of the class, this function is called in a loop that I'd like to parallelize, as it takes ~30 seconds to run in serial. I've tried using multiprocessing's Pool, but that can't write to the dictionary. I've also tried modifying the function to return the lambda in a couple ways, but it can't return a lambda, saying AttributeError(&quot;Can't pickle local object '(class).(function).<locals&gt;.<lambda&gt;'&quot;).
The below code segments would be in a method of Example, and are approximately what I tried.
pool = multiprocessing.Pool()
keys = []  # list of keys it needs to operate on
(code to set up keys)
# Doesn't work, won't write to dictionary (Effectively does nothing but take up time)
pool.map(self.create_function, keys)
# Doesn't work, can't return lambda (AttributeError)
new_dict = dict(pool.map(self.create_function, keys))
self.functions.update(new_dict)
Is there a way to parallelize this?
I was able to solve the problem as relevant to my project by separating the (code that sets up parameters) part and paralellizing that instead, but the question is still relevant - how can a function that returns a lambda be parallelized?
Comment: @MatthiasFripp I've updated it, it's scipy's `interp2d`.

Comment: @MatthiasFripp I've edited it again, hopefully this should help. Both code segments would be in methods of  `Example`.

Comment: Yes, it does. Thank you!

Comment: Can you show where `method_getting_function()` is defined and a minimal version of what it does? There's not quite enough here to reproduce your problem.

Comment: Thanks. I also should have asked, what is `self` here? Is your second block of code running inside some method of `Example`? It would help if you give a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example).

Comment: Does this answer your question? [Can Python pickle lambda functions?](https://stackoverflow.com/questions/25348532/can-python-pickle-lambda-functions)


Here is another answer: From this answer linked by @quamrana, using a package such as dill allows lambdas and other things to be pickled/serialized, and therefore returned in parallel.

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 6, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2020:00:07/24/20', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<python><lambda><python-multithreading>', 'Title': 'How to parallelize a function that returns a lambda', 'ViewCount': 118, 'id': 63079067, 'user_AboutMe': None, 'user_AccountId': 14506633.0, 'user_DisplayName': 'Alex Pickering', 'user_Reputation': 141.0, 'user_Views': 12.0, 'user_id': 10478247.0}}