as mentionedthe most fundamental piece of spark is called the resilient distributed datasetan rddand this is going to be the object that you use to actually load and transform and get the answers you want out of the data that you're trying to process it' very important thing to understand the final letter in rdd stands for datasetand at the end of the day that' all it isit' just bunch of rows of information that can contain pretty much anything but the key is the and the first resilientit is resilient in that spark makes sure that if you're running this on cluster and one of those clusters goes downit can automatically recover from that and retry nowthat resilience only goes so farmind you if you don' have enough resources available to the job that you're trying to runit will still failand you will have to add more resources to it there' only so many things it can recover fromthere is limit to how many times it will retry given task but it does make its best effort to make sure that in the face of an unstable cluster or an unstable network it will still continue to try its best to run through to completion distributedobviouslyit is distributed the whole point of using spark is that you can use it for big data problems where you can actually distribute the processing across the entire cpu and memory power of cluster of computers that can be distributed horizontallyso you can throw as many computers as you want to given problem the larger the problemthe more computersthere' really no upper bound to what you can do there the sparkcontext object you always start your spark scripts by getting sparkcontext objectand this is the object that embodies the guts of spark it is what is going to give you your rdds to process onso it is what generates the objects that you use in your processing you knowyou don' actually think about the sparkcontext very much when you're actually writing spark programsbut it is sort of the substrate that is running them for you under the hood if you're running in the spark shell interactivelyit has an td object already available for you that you can use to create rdds in standalone scripthoweveryou will have to create that sparkcontext explicitlyand you'll have to pay attention to the parameters that you use because you can actually tell the spark context how you want that to be distributed should take advantage of every core that have available to meshould be running on cluster or just standalone on my local computersothat' where you set up the fundamental settings of how spark will operate