looking under the hood how does jug workat the basic levelit is very simplea task is function plus its argument its arguments may be either values or other tasks if task takes other tasksthere is dependency between the two tasks (and the second one cannot be run until the results of the first task are availablebased on thisjug recursively computes hash for each task this hash value encodes the whole computation to get there when you run jug executethere is little loop as shown in the following code snippetfor in alltasksif has_not_run(and not backend_has_value( hash())value execute(save_to_backend(valuekey= hash()the real loop is much more complex because of locking issuesbut the general idea is the one that appears in the preceding code snippet the default backend writes the file to the disk (in this funny directory named jugfile jugdata/but another backend is available which uses redis database with proper lockingwhich jug takes care ofthis also allows for many processors to execute tasksthey will independently look at all the tasks and run the ones which have not run yet and then write them back to the shared backend this works on either the same machine (using multicore processorsor in multiple machines as long as they all have access to the same backend (for exampleusing network disk or the redis databasesin the next half of this we will discuss computer clustersbut for nowlet us focus on multiple cores you can also understand why it is able to memoize intermediate results if the backend already has the result of taskit is not run anymore on the other handif you change the taskeven in minute ways (by altering one of the parameters)its hash will change thereforeit will be recomputed furthermoreall tasks that depend on it will also have their hashes changed and they will be recomputed as well using jug for data analysis jug is generic frameworkbut it is ideally suited for medium-scale data analysis as you develop your analysis pipelineit is good to have intermediate results be saved if you already computed the preprocessing step before and are only changing the features you computeyou do not want to recompute the preprocessing if you already computed the features but want to try combining few new ones into the mixyou also do not want to recompute all your other features