wellit turns out there' perfectly reasonable explanationand this is good lesson in why you always need to examine your results when you're done with any sort of data science task-question the resultsbecause often there' something you missedthere might be something you need to clean in your datathere might be something you did wrong but you should also always look skeptically at your resultsdon' just take them on faithokayif you do soyou're going to get in troublebecause if were to actually present these as recommendations to people who liked star warsi would get fired don' get firedpay attention to your resultssolet' dive into what went wrong in our next section improving the results of movie similarities let' figure out what went wrong with our movie similarities there we went through all this exciting work to compute correlation scores between movies based on their user ratings vectorsand the results we got kind of sucked sojust to remind youwe looked for movies that are similar to star wars using that techniqueand we ended up with bunch of weird recommendations at the top that had perfect correlation andmost of them were very obscure movies sowhat do you think might be going on therewellone thing that might make sense islet' say we have lot of people watch star wars and some other obscure film we' end up with good correlation between these two movies because they're tied together by star warsbut at the end of the daydo we really want to base our recommendations on the behavior of one or two people that watch some obscure movieprobably noti mean yesthe two people in the worldor whatever it isthat watch the movie full speedand both liked it in addition to star warsmaybe that is good recommendation for thembut it' probably not good recommendation for the rest of the world we need to have some sort of confidence level in our similarities by enforcing minimum boundary of how many people watched given movie we can' make judgment that given movie is good just based on the behavior of one or two people solet' try to put that insight into action using the following codejnqpsuovnqzbtoq npwjf ubut sbujohthspvqcz npwjf ubutifbe ujumf bhh sbujoh what we're going to do is try to identify the movies that weren' actually rated by many people and we'll just throw them out and see what we get soto do that we're going to take our original ratings dataframe and we're going to say hspvqcz ujumf again pandas has all sorts of magic in it andthis will basically construct new dataframe that aggregates together all the rows for given title into one row