within clusteryou might have individual executor tasks that are running these might be running on different computersor they might be running on different cores of the same computer they each have their own individual cache and their own individual tasks that they run the driver programthe spark context and the cluster manager work together to coordinate all this effort and return the final result back to you the beauty of it isall you have to do is write the initial little scriptthe driver programwhich uses spark context to describe at high level the processing you want to do on this data sparkworking together with the cluster manager that you're usingfigures out how to spread that out and distribute it so you don' have to worry about all those details wellif it doesn' workobviouslyyou might have to do some troubleshooting to figure out if you have enough resources available for the task at handbutin theoryit' all just magic it' fast what' the big deal about sparki meanthere are similar technologies like mapreduce that have been around longer spark is fast thoughand on the website they claim that spark is "up to faster than mapreduce when running job in memoryor times faster on disk of coursethe key words here are "up to,your mileage may vary don' think 've ever seen anythingactuallyrun that much faster than mapreduce some well-crafted mapreduce code can actually still be pretty darn efficient but will say that spark does make lot of common operations easier mapreduce forces you to really break things down into mappers and reducerswhereas spark is little bit higher level you don' have to always put as much thought into doing the right thing with spark part of that leads to another reason why spark is so fast it has dag enginea directed acyclic graph wowthat' another fancy word what does it meanthe way spark works isyou write script that describes how to process your dataand you might have an rdd that' basically like data frame you might do some sort of transformation on itor some sort of action on it but nothing actually happens until you actually perform an action on that data what happens at that point isspark will say "hmmok sothis is the end result you want on this data what are all the other things had to do to get up this pointand what' the optimal way to lay out the strategy for getting to that point?sounder the hoodit will figure out the best way to split up that processingand distribute that information to get the end result that you're looking for sothe key inside hereis that spark waits until you tell it to actually produce resultand only at that point does it actually go and figure out how to produce that result soit' kind of cool concept thereand that' the key to lot of its efficiency