in this we got hands-on introduction to the hadoop and spark frameworks we covered lot of groundbut be honestpython makes working with big data technologies dead easy in the next we'll dig deeper into the world of nosql databases and come into contact with more big data technologies summary in this you learned that hadoop is framework that enables you to store files and distribute calculations amongst many computers hadoop hides all the complexities of working with cluster of computers for you an ecosystem of applications surrounds hadoop and sparkranging from databases to access control spark adds shared memory structure to the hadoop framework that' better suited for data science work in the case study we used pyspark ( python libraryto communicate with hive and spark from python we used the pywebhdfs python library to work with the hadoop librarybut you could do as well using the os command line it' easy to connect bi tool such as qlik to hadoop