how to do it more robust than edit distance is the so-called bag-of-word approach it uses simple word counts as its basis for each word in the postits occurrence is counted and noted in vector not surprisinglythis step is also called vectorization the vector is typically huge as it contains as many elements as the words that occur in the whole dataset take for instance two example posts with the following word countsword occurrences in post occurrences in post disk format how hard my problems to the columns post and post can now be treated as simple vectors we could simply calculate the euclidean distance between the vectors of all posts and take the nearest one (too slowas we have just found outas suchwe can use them later in the form of feature vectors in the following clustering steps extract the salient features from each post and store it as vector per post compute clustering on the vectors determine the cluster for the post in question from this clusterfetch handful of posts that are different from the post in question this will increase diversity howeverthere is some more work to be done before we get thereand before we can do that workwe need some data to work on preprocessing similarity measured as similar number of common words as we have seen previouslythe bag-of-word approach is both fast and robust howeverit is not without challenges let' dive directly into them