your algorithms cannot assume that the entire data is in ram managing data becomes major task in itself using computer clusters or multicore machines becomes necessity and not luxury this will focus on this last piece of the puzzlehow to use multiple cores (either on the same machine or on separate machinesto speed up and organize your computations this will also be useful in other medium-sized data tasks using jug to break up your pipeline into tasks oftenwe have simple pipelinewe preprocess the initial datacompute featuresand then we need to call machine learning algorithm with the resulting features jug is package developed by luis pedro coelhoone of the authors of this book it is open source (using the liberal mit licenseand can be useful in many areas but was designed specifically around data analysis problems it simultaneously solves several problemsfor exampleit can memorize results to disk (or database)which means that if you ask it to compute something you have computed beforethe result is instead read from the disk it can use multiple cores or even multiple computers on cluster jug was also designed to work very well in batch computing environments that use queuing system such as portable batch system (pbs)the load sharing facility (lsf)or the oracle grid engine (ogeearlier known as sun grid enginethis will be used in the second half of the as we build online clusters and dispatch jobs to them about tasks tasks are the basic building block of jug task is just function and values for its argumentsfor exampledef double( )return *