ectools/parallelization.py
def parallel_wrapper(fcn, iterable, cache_to=None, n_jobs=1, progress=False, verbose=0, force_recompute=False,
                     parallel_kwargs={}):
    """A function that just wraps the most common use case of joblib. Parallelizes the calculation of `fcn` on the elements of `iterable`.

    Args:
        fcn: the function to be calculated in parallel
        iterable: the iterable containing the argument for the function
        cache_to: if it is a valid file path, and the file does not exist the calculation will be pickled to this path. If the file exists, the calculation will be skipped, and the function will read the result from this path.
        n_jobs: the number of CPU cores to use
        progress: whether to show a progressbar using tqdm
        verbose: passed to the `verbose` keyword of joblib.Parallel
        force recompute: makes the wrapper ignore `cache_to`, and recomputes even if the file exists.
        parallel_kwargs: additional keyword arguments for joblib.Parallel

    Returns:
            results: a list containing the outputs of `fcn`
    """
    # if caching is active
    if cache_to is not None:
        # check directory exists
        directory_name = dirname(cache_to)
        if directory_name:
            if not exists(directory_name):
                raise FileNotFoundError('The directory specified for the caching of the computation does not exist.')
        # do computation
        if force_recompute or not exists(cache_to):
            results = _pw(fcn=fcn, iterable=iterable, n_jobs=n_jobs, progress=progress, verbose=verbose, parallel_kwargs=parallel_kwargs)
            # save result
            picklesave(results, cache_to)
        # if file exists:
        else:
            results = pickleload(cache_to)
    else:
        results = _pw(fcn=fcn, iterable=iterable, n_jobs=n_jobs, progress=progress, verbose=verbose, parallel_kwargs=parallel_kwargs)
    return results
A function that just wraps the most common use case of joblib. Parallelizes the calculation of `fcn` on the elements of `iterable`.

    Args:
        fcn: the function to be calculated in parallel
        iterable: the iterable containing the argument for the function
        cache_to: if it is a valid file path, and the file does not exist the calculation will be pickled to this path. If the file exists, the calculation will be skipped, and the function will read the result from this path.
        n_jobs: the number of CPU cores to use
        progress: whether to show a progressbar using tqdm
        verbose: passed to the `verbose` keyword of joblib.Parallel
        force recompute: makes the wrapper ignore `cache_to`, and recomputes even if the file exists.
        parallel_kwargs: additional keyword arguments for joblib.Parallel

    Returns:
            results: a list containing the outputs of `fcn`
    
A function that just wraps the most common use case of joblib. Parallelizes the calculation of `fcn` on the elements of `iterable`.
A function that just wraps the most common use case of joblib . Parallelizes the calculation of ` fcn ` on the elements of ` iterable ` .
def parallel_wrapper(fcn, iterable, cache_to=None, n_jobs=1, progress=False, verbose=0, force_recompute=False,
                     parallel_kwargs={}):
    if cache_to is not None:
        directory_name = dirname(cache_to)
        if directory_name:
            if not exists(directory_name):
                raise FileNotFoundError('The directory specified for the caching of the computation does not exist.')
        if force_recompute or not exists(cache_to):
            results = _pw(fcn=fcn, iterable=iterable, n_jobs=n_jobs, progress=progress, verbose=verbose, parallel_kwargs=parallel_kwargs)
            picklesave(results, cache_to)
        else:
            results = pickleload(cache_to)
    else:
        results = _pw(fcn=fcn, iterable=iterable, n_jobs=n_jobs, progress=progress, verbose=verbose, parallel_kwargs=parallel_kwargs)
    return results
def parallel_wrapper ( fcn , iterable , cache_to = None , n_jobs = 1 , progress = False , verbose = 0 , force_recompute = False , parallel_kwargs = { } ) : if cache_to is not None : directory_name = dirname ( cache_to ) if directory_name : if not exists ( directory_name ) : raise FileNotFoundError ( 'The directory specified for the caching of the computation does not exist.' ) if force_recompute or not exists ( cache_to ) : results = _pw ( fcn = fcn , iterable = iterable , n_jobs = n_jobs , progress = progress , verbose = verbose , parallel_kwargs = parallel_kwargs ) picklesave ( results , cache_to ) else : results = pickleload ( cache_to ) else : results = _pw ( fcn = fcn , iterable = iterable , n_jobs = n_jobs , progress = progress , verbose = verbose , parallel_kwargs = parallel_kwargs ) return results
A function that just wraps the most common use case of joblib.
A function that just wraps the most common use case of joblib .
"""A function that just wraps the most common use case of joblib. Parallelizes the calculation of `fcn` on the elements of `iterable`.

    Args:
        fcn: the function to be calculated in parallel
        iterable: the iterable containing the argument for the function
        cache_to: if it is a valid file path, and the file does not exist the calculation will be pickled to this path. If the file exists, the calculation will be skipped, and the function will read the result from this path.
        n_jobs: the number of CPU cores to use
        progress: whether to show a progressbar using tqdm
        verbose: passed to the `verbose` keyword of joblib.Parallel
        force recompute: makes the wrapper ignore `cache_to`, and recomputes even if the file exists.
        parallel_kwargs: additional keyword arguments for joblib.Parallel

    Returns:
            results: a list containing the outputs of `fcn`
    """ # if caching is active # check directory exists # do computation # save result # if file exists:
a list containing the outputs of `fcn`
a list containing the outputs of ` fcn `
results
fcn
the function to be calculated in parallel
the function to be calculated in parallel
iterable
the iterable containing the argument for the function
the iterable containing the argument for the function
cache_to
if it is a valid file path, and the file does not exist the calculation will be pickled to this path. If the file exists, the calculation will be skipped, and the function will read the result from this path.
if it is a valid file path and the file does not exist the calculation will be pickled to this path . If the file exists the calculation will be skipped and the function will read the result from this path .
n_jobs
the number of CPU cores to use
the number of CPU cores to use
progress
whether to show a progressbar using tqdm
whether to show a progressbar using tqdm
verbose
passed to the `verbose` keyword of joblib.Parallel
passed to the ` verbose ` keyword of joblib . Parallel
force_recompute
parallel_kwargs
additional keyword arguments for joblib.Parallel
additional keyword arguments for joblib . Parallel
force recompute
makes the wrapper ignore `cache_to`, and recomputes even if the file exists.
makes the wrapper ignore ` cache_to ` and recomputes even if the file exists .