we will achieve this goal in this using clustering this is method of arranging items so that similar items are in one cluster and dissimilar items are in distinct ones the tricky thing that we have to tackle first is how to turn text into something on which we can calculate similarity with such measurement for similaritywe will then proceed to investigate how we can leverage that to quickly arrive at cluster that contains similar posts once therewe will only have to check out those documents that also belong to that cluster to achieve thiswe will introduce the marvelous scikit librarywhich comes with diverse machine-learning methods that we will also use in the following measuring the relatedness of posts from the machine learning point of viewraw text is useless only if we manage to transform it into meaningful numberscan we feed it into our machine-learning algorithms such as clustering the same is true for more mundane operations on textsuch as similarity measurement how not to do it one text similarity measure is the levenshtein distancewhich also goes by the name edit distance let' say we have two words"machineand "mchienethe similarity between them can be expressed as the minimum set of edits that are necessary to turn one word into the other in this casethe edit distance would be as we have to add an "aafter "mand delete the first "ethis algorithm ishoweverquite costlyas it is bound by the product of the lengths of the first and second words looking at our postswe could cheat by treating the whole word as characters and performing the edit distance calculation on the word level let' say we have two posts (let' concentrate on the title for the sake of simplicity)"how to format my hard diskand "hard disk format problems"we would have an edit distance of five (removing "how""to""format""my"and then adding "formatand "problemsat the endthereforeone could express the difference between two posts as the number of words that have to be added or deleted so that one text morphs into the other although we could speed up the overall approach quite bitthe time complexity stays the same even if it would have been fast enoughthere is another problem the post above the word "formataccounts for an edit distance of two (deleting it firstthen adding itso our distance doesn' seem to be robust enough to take word reordering into account