text: Dask Distributed Runtime¶
  Dask Overview¶
  Source: Dask docs
  Dask collections and schedulers
  Dask DataFrame mimics Pandas¶()
  Dask.distributed
  ¶
  Dask.distributed is a lightweight library for distributed computing in Python. It extends both the concurrent.futures and dask APIs to moderate sized clusters.
  install.
  ~5X Faster with Dask¶
  Short example which demonstrates the power of Dask, in this notebook we will preform the following:
  Generate random text files
  Process the file by sorting and counting it’s content
  Compare run times
  Generate Random Text Files¶
  import random import string import os from collections import Counter from dask.distributed import Client import warnings warnings.filterwarnings('ignore')
  def generate_big_random_letters(filename, size): """ generate big random letters/alphabets to a file :param filename: the filename :param size: the size in bytes :return: void """ chars = ''.join([random.choice(string.ascii_letters) for i in range(size)]) #1 with open(filename, 'w') as f: f.write(chars) pass
  PATH = '/User/howto/dask/random_files' SIZE = 10000000 for i in range(100): generate_big_random_letters(filename = PATH + '/file_' + str(i) + '.txt', size = SIZE)
  Set Function for Benchmark¶
  def count_letters(path): """ count letters in text file :param path: path to file """ # open file in read mode file = open(path, "r") # read the content of file data = file.read() # sort file sorted_file = sorted(data) # count file number_of_characters = len(sorted_file) return number_of_characters
  def process_files(path): """ list file and count letters :param path: path to folder with files """ num_list = [] files = os.listdir(path) for file in files: cnt = count_letters(os.path.join(path, file)) num_list.append(cnt) l = num_list return print("done!")
  Sort & Count Number of Letters with Python¶
  %%time PATH = '/User/howto/dask/random_files/' process_files(PATH)
  done! CPU times: user 2min 19s, sys: 9.31 s, total: 2min 29s Wall time: 2min 32s
  Sort & Count Number of Letters with Dask¶
  # get the dask client address client = Client()
  # list all files in folder files = [PATH + x for x in os.listdir(PATH)]
  %%time # run the count_letter function on a list of files while using multiple workers a = client.map(count_letters, files)
  CPU times: user 13.2 ms, sys: 983 µs, total: 14.2 ms Wall time: 12.2 ms
  %%time # gather results l = client.gather(a)
  CPU times: user 3.39 s, sys: 533 ms, total: 3.92 s Wall time: 40 s
  Additional Topics¶
  - Running Dask on the cluster with mlrun
  - Pipelines Using Dask, Kubeflow and MLRun