as you seefor one thingwe're using nm instead of -mjcand that' because the new dataframe-based api is in there implementing linear regression in this examplewhat we're going to do is implement linear regressionand linear regression is just way of fitting line to set of data what we're going to do in this exercise is take bunch of fabricated data that we have in two dimensionsand try to fit line to it with linear model we're going to separate our data into two setsone for building the model and one for evaluating the modeland we'll compare how well this linear model does at actually predicting real values first of allin spark if you're going to be doing stuff with the qbsl interface and using datasetsyou've got to be using qbsl fttjpo object instead of qbsl$poufyu to set one upyou do the followingtqbsl qbsl fttjpocvjmefsdpogjh tqbsltrmxbsfipvtfejsgjmf ufnqbqq/bnf -jofbs fhsfttjpohfu $sfbuf note that the middle bit is only necessary on windows and in spark it kind of works around little bug that they haveto be honest soif you're on windowsmake sure you have ufnq folder if you want to run thisgo create that now if you need to if you're not on windowsyou can delete that whole middle section to leavetqbsl qbsl fttjpocvjmefsbqq/bnf -jofbs fhsfttjpohfu $sf buf okayso you can say tqbslgive it an bqq/bnf and hfu $sfbuf this is interestingbecause once you've created spark sessionif it terminates unexpectedlyyou can actually recover from that the next time that you run it soif we have checkpoint directoryit can actually restart where it left off using hfu $sfbuf nowwe're going to use this sfhsfttjpouyu file that have included with the course materialsjoqvu-joft tqbsltqbsl$poufyuufyu'jmf sfhsfttjpouyu