reg fit(xcy[train]- we need to perform the same normalization while testing xc, movie_norm( [test] np array(map(reg predictxc)ravel( ( + )- [testerr +np sum( *ewe did not explain the movie_norm function this function performs per-movie normalizationsome movies are just generally better and get higher average marksdef movie_norm( )xc copy(toarray(we cannot use xc mean( because we do not want to have the zeros counting for the mean we only want the mean of the ratings that were actually givenx np array([xi[xi mean(for xi in xc]in certain casesthere were no ratings and we got nan valueso we replace it with zeros using np nan_to_numwhich does exactly this taskx np nan_to_num( now we normalize the input by removing the mean value from the non-zero entriesfor in xrange(xc shape[ ])xc[ -(xc[ [iimplicitlythis also makes the movies that the user did not rate have value of zerowhich is average finallywe return the normalized array and the meansreturn , you might have noticed that we converted to regular (densearray this has the added advantage that it makes the optimization much fasterwhile scikit-learn works well with the sparse valuesthe dense arrays are much faster (if you can fit them in memorywhen you cannotyou are forced to use sparse arrayswhen compared with simply guessing the average value for that userthis approach is percent better the results are not spectacularbut it is start on one handthis is very hard problem and we cannot expect to be right with every predictionwe perform better when the users have given us more reviews on the other handregression is blunt tool for this job note how we learned completely separate model for each user in the next we will look at other methods that go beyond regression for approaching this problem in those modelswe integrate the information from all users and all movies in more intelligent manner