Acceleration techniques for CPU soft-shadow calculation in light mapper
You could try to optimize the calculation by first casting a smaller number of "feeler rays" (say 32) to check for penumbra, and cast the remaining rays if the result is in penumbra according to the rays. Of course this can give you false negatives, but you could then try to improve the heuristics by checking the results of nearby feeler rays.