using the spark dataframe api for mllib this was originally produced for spark so let' talk about what' new in spark and what new capabilities exist in mllib now sothe main thing with spark is that they moved more and more toward dataframes and datasets datasets and dataframes are kind of used interchangeably sometimes technically dataframe is dataset of row objectsthey're kind of like rddsbut the only difference is thatwhereas an rdd just contains unstructured dataa dataset has defined schema to it dataset knows ahead of time exactly what columns of information exists in each rowand what types those are because it knows about the actual structure of that dataset ahead of timeit can optimize things more efficiently it also lets us think of the contents of this dataset as littlemini databasewellactuallya very big database if it' on cluster that means we can do things like issue sql queries on it this creates higher-level api with which we can query and analyze massive datasets on spark cluster it' pretty cool stuff it' fasterit has more opportunities for optimizationand it has higher-level api that' often easier to work with how spark mllib works going forward in spark mllib is pushing dataframes as its primary api this is the way of the futureso let' take look at how it works 've gone ahead and opened up the qbsl-jofbs fhsfttjpoqz file in canopyas shown in the following figureso let' walk through it little bit