this is actually represented as sparse vector at this point to save even more space sonot only have we converted all of our words to numbersbut we've also stripped out any missing data in the event that word does not appear in document where you're not storing the fact that word does not appear explicitlyit saves even more space computing the tf-idf score to actually compute the tf-idf score for each word in each documentwe first cache this ug rdd ugdbdif we do that because we're going to use it more than once nextwe use *%njo%pd'sfr meaning that we're going to ignore any word that doesn' appear at least twicejeg *%njo%pd'sfr gju ug we call gju on ugand then in the next line we call usbotgpsn on ugugjeg jegusbotgpsn ug what we end up with here is an rdd of the tf-idf score for each word in each document using the wikipedia search engine algorithm let' try and put the algorithm to use let' try to look up the best article for the word gettysburg if you're not familiar with us historythat' where abraham lincoln gave famous speech sowe can transform the word gettysburg into its hash value using the following codehfuuztcvsh ibtijoh 'usbotgpsn hfuuztcvsh)bti bmvf jou hfuuztcvsh 'joejdft we will then extract the tf-idf score for that hash value into new rdd for each documenthfuuztcvsh fmfwbodf ugjegnbq mbnceby