and finallywe computed the wssse metricit came out to in this example soif you want to play around with this little biti encourage you to do so you can see what happens to that error metric as you increase or decrease the values of kand think about why that may be you can also experiment with what happens if you don' normalize all the datadoes that actually affect your results in meaningful wayis that actually an important thing to doand you can also experiment with the nby*ufsbujpot parameter on the model itself and get good feel of what that actually does to the final resultsand how important it is sofeel free to mess around with it and experiment away that' kmeans clustering done with mllib and spark in scalable manner very cool stuff tf-idf soour final example of mllib is going to be using something called term frequency inverse document frequencyor tf-idfwhich is the fundamental building block of many search algorithms as usualit sounds complicatedbut it' not as bad as it sounds sofirstlet' talk about the concepts of tf-idfand how we might go about using that to solve search problem and what we're actually going to do with tf-idf is create rudimentary search engine for wikipedia using apache spark in mllib how awesome is thatlet' get started tf-idf stands for term frequency and inverse document frequencyand these are basically two metrics that are closely interrelated for doing search and figuring out the relevancy of given word to documentgiven larger body of documents sofor exampleevery article on wikipedia might have term frequency associated with itevery page on the internet could have term frequency associated with it for every word that appears in that document sounds fancybutas you'll seeit' fairly simple concept all term frequency means is how often given word occurs in given document sowithin one web pagewithin one wikipedia articlewithin one whateverhow common is given word within that documentyou knowwhat is the ratio of that word' occurrence rate throughout all the words in that documentthat' it that' all term frequency is document frequencyis the same ideabut this time it is the frequency of that word across the entire corpus of documents sohow often does this word occur throughout all of the documents that haveall the web pagesall of the articles on wikipediawhatever for examplecommon words like "aor "thewould have very high document frequencyand would expect them to also have very high term frequencybut that doesn' necessarily mean they're relevant to given document