finallyi' going to map that datatake in each list of fieldsextract field number three ywhich happen to know is the body of the article itselfthe actual article textand ' in turn going to split that based on spacesepdvnfout gjfmetnbq mbnceby tqmju what does is extract the body of the text from each wikipedia articleand split it up into list of words my new epdvnfout rdd has one entry for every documentand every entry in that rdd contains list of words that appear in that document nowwe actually know what to call these documents later on when we're evaluating the results ' also going to create new rdd that stores the document namesepdvnfou/bnft gjfmetnbq mbnceby all that does is take that same gjfmet rdd and uses this nbq function to extract the document namewhich happen to know is in field number one soi now have two rddsepdvnfoutwhich contains lists of words that appear in each documentand epdvnfou/bnftwhich contains the name of each document also know that these are in the same orderso can actually combine these together later on to look up the name for given document creating and transforming hashingtf object nowthe magic happens the first thing we're going to do is create )btijoh objectand we're going to pass in parameter of , this means that ' going to hash every word into one of , numerical valuesibtijoh )btijoh instead of representing words internally as stringswhich is very inefficientit' going to try toas evenly as possibledistribute each word to unique hash value ' giving it up to , hash values to choose from basicallythis is mapping words to numbers at the end of the day nexti' going to call usbotgpsn on ibtijoh with my actual rdd of documentsug ibtijoh 'usbotgpsn epdvnfout that' going to take my list of words in every document and convert it to list of hash valuesa list of numbers that represent each word instead