you can kind of see where we're going with this solet' say we have very high term frequency and very low document frequency for given word the ratio of these two things can give me measure of the relevance of that word to the document soif see word that occurs very often in given documentbut not very often in the overall space of documentsthen know that this word probably conveys some special meaning to this particular document it might convey what this document is actually about sothat' tf-idf it just stands for term frequency inverse document frequencywhich is just fancy way of saying term frequency over document frequencywhich is just fancy way of saying how often does this word occur in this document compared to how often it occurs in the entire body of documentsit' that simple tf-idf in practice in practicethere are few little nuances to how we use this for examplewe use the actual log of the inverse document frequency instead of the raw valueand that' because word frequencies in reality tend to be distributed exponentially soby taking the logwe end up with slightly better weighting of wordsgiven their overall popularity there are some limitations to this approachobviouslyone is that we basically assume document is nothing more than bagful of wordswe assume there are no relationships between the words themselves andobviouslythat' not always the caseand actually parsing them out can be good part of the workbecause you have to deal with things like synonyms and various tenses of wordsabbreviationscapitalizationsmisspellingsand so on this gets back to the idea of cleaning your data being large part of your job as data scientistand it' especially true when you're dealing with natural language processing stuff fortunatelythere are some libraries out there that can help you with thisbut it is real problem and it will affect the quality of your results another implementation trick that we use with tf-idf isinstead of storing actual string words with their term frequencies and inverse document frequencyto save space and make things more efficientwe actually map every word to numerical valuea hash value we call it the idea is that we have function that can take any wordlook at its lettersand assign thatin some fairly well-distributed mannerto set of numbers in range that wayinstead of using the word "represented"we might assign that hash value of and we can then refer to the word "representedas " from now on nowif the space of your hash values isn' large enoughyou could end up with different words being represented by the same numberwhich sounds worse than it is butyou knowyou want to make sure that you have fairly large hash space so that is unlikely to happen those are called hash collisions they can cause issuesbutin realitythere' only so many words that people commonly use in the english language you can get away with , or so and be just fine