Faster generation of list with multiple sublists I'm trying to improve the performance of some rather large code. One of the bottlenecks is the line below. It doesn't look like much but it is processed millions of times.

Is there any way to improve its performance at the code level (ie: no parallelization)? I'm perfectly fine with the final outcome being a numpy array instead of a list.

import numpy as np

def func(m1, m2):
    return -2.5 * np.log10(10 ** (-0.4 * m1) + 10 ** (-0.4 * m2))

# Generate data.
N1, N2 = 300, 200
aa = np.random.random((5, N1))
idxs = np.random.choice(N1, N2, replace=False)
bb = np.random.random((10, N2))

# I need to improve the performance of this line.
cc = [func(m[idxs], bb[i]) for (i, m) in enumerate(aa)]




Add

As Mibac suggested, the issue here might actually be the successive calls to func(), not the generation of the cc list itself.

If this is the case, then I'd need to improve the performance of the line:

-2.5 * np.log10(10 ** (-0.4 * m1) + 10 ** (-0.4 * m2))


or equivalently:

-2.5 * np.log10(c ** m1 + c ** m2)


where c=10**(-0.4).
aa = np.random.random((5, N1))
idxs = np.random.choice(N1, N2, replace=False)
bb = np.random.random((10, N2))

# I need to improve the performance of this line.
cc = [func(m[idxs], bb[i]) for (i, m) in enumerate(aa)]



Unless I'm misunderstanding something, half of bb is never used. Shouldn't the 5 in the instantiation of aa and the 10 in the instantiation of bb be the same value?




def func(m1, m2):
    return -2.5 * np.log10(10 ** (-0.4 * m1) + 10 ** (-0.4 * m2))



Algebraically, $$\log_{10}(10^x + 10^y) = \log_{10}(10^x (1 + 10^{y - x})) = x + \log_{10}(1 + 10^{y - x})$$ Since in this scenario \$-1 \le y - x \le 1\$ there doesn't seem to be any numerical-analytic reason not to simplify like this.

Depending on your error requirements, it may be possible to speed up the logarithm further by approximating it with a Taylor expansion $$\log_{10}(1 + 10^z) = \frac{\ln 2}{\ln 10} + \frac{z}{2} + \frac{\ln 10}{8} z^2 - \frac{\ln^3 10}{192} z^4 + \frac{\ln^5 10}{2880}z^6 - \ldots$$
or with a Padé approximant.