doing this at scale is the hard part if you want to do this over all of wikipediathen you're going to have to run this on cluster but for the sake of argumentwe are just going to run this on our own desktop for nowusing small sample of wikipedia data using tfidf how do we turn that into an actual search problemonce we have tf-idfwe have this measure of each word' relevancy to each document what do we do with itwellone thing you could do is compute tf-idf for every word that we encounter in the entire body of documents that we haveand thenlet' say we want to search for given terma given word let' say we want to search for "what wikipedia article in my set of wikipedia articles is most relevant to gettysburg? could sort all the documents by their tf-idf score for gettysburgand just take the top resultsand those are my search results for gettysburg that' it just take your search wordcompute tf-idftake the top results that' it obviouslyin the real world there' lot more to search than that google has armies of people working on this problem and it' way more complicated in practicebut this will actually give you working search engine algorithm that produces reasonable results let' go ahead and dive in and see how it all works searching wikipedia with spark mllib we're going to build an actual working search algorithm for piece of wikipedia using apache spark in mlliband we're going to do it all in less than lines of code this might be the coolest thing we do in this entire book