Running parallel tasks until they're done
1) The library you are trying to use clearly does not fit your requirements. So why force it? Aren't you essentially just calling:

var tasks = Enumerable.Range(0, tasks).Select(i =&gt; DoWithRetries(action, retries, method));
await Task.WhenAll(tasks);


2) Fatal error is an error that your software cannot recover from. If you just swallow it in retry loop and keep going, then it is not fatal.

3) Delay duration should probably be a parameter and not a hardcoded value.

4) You can use nameof operator instead of MethodBase.GetCurrentMethod().Name.

P.S. I don't know what ThreadExecutor is and I didn't understand your use-case example, so I won't comment on those.