# Matrix Factorization via Singular Value Decomposition

Matrix factorization is the breaking down of one matrix in a product of multiple matrices. It's extremely well studied in mathematics, and it's highly useful. There are many different ways to factor matrices, but singular value decomposition is particularly useful for making recommendations.

So what is singular value decomposition (SVD)? At a high level, SVD is an algorithm that decomposes a matrix $R$ into the best lower rank (i.e. smaller/simpler) approximation of the original matrix $R$. Mathematically, it decomposes R into a two unitary matrices and a diagonal matrix:

$$\begin{equation}
R = U\Sigma V^{T}
\end{equation}$$

where R is users's ratings matrix, $U$ is the user "features" matrix, $\Sigma$ is the diagonal matrix of singular values (essentially weights), and $V^{T}$ is the movie "features" matrix. $U$ and $V^{T}$ are orthogonal, and represent different things. $U$ represents how much users "like" each feature and $V^{T}$ represents how relevant each feature is to each movie.

To get the lower rank approximation, we take these matrices and keep only the top $k$ features, which we think of as the underlying tastes and preferences vectors.



```python
import pandas as pd
import numpy as np

r_cols = ['user_id', 'movie_id', 'rating']
m_cols = ['movie_id', 'title']

ratings_df = pd.read_csv('u.data',sep='\t', names=r_cols, usecols = range(3), dtype = int)
movies_df = pd.read_csv('u.item', sep='|', names=m_cols, usecols=range(2))
movies_df['movie_id'] = movies_df['movie_id'].apply(pd.to_numeric)
```


```python
movies_df.head(3)
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>movie_id</th>
      <th>title</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>Toy Story (1995)</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>GoldenEye (1995)</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>Four Rooms (1995)</td>
    </tr>
  </tbody>
</table>
</div>




```python
ratings_df.head(3)
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user_id</th>
      <th>movie_id</th>
      <th>rating</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>196</td>
      <td>242</td>
      <td>3</td>
    </tr>
    <tr>
      <th>1</th>
      <td>186</td>
      <td>302</td>
      <td>3</td>
    </tr>
    <tr>
      <th>2</th>
      <td>22</td>
      <td>377</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>



These look good, but I want the format of my ratings matrix to be one row per user and one column per movie. I'll `pivot` `ratings_df` to get that and call the new variable `R`.


```python
R_df = ratings_df.pivot(index = 'user_id', columns ='movie_id', values = 'rating').fillna(0)
R_df.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>movie_id</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
      <th>5</th>
      <th>6</th>
      <th>7</th>
      <th>8</th>
      <th>9</th>
      <th>10</th>
      <th>...</th>
      <th>1673</th>
      <th>1674</th>
      <th>1675</th>
      <th>1676</th>
      <th>1677</th>
      <th>1678</th>
      <th>1679</th>
      <th>1680</th>
      <th>1681</th>
      <th>1682</th>
    </tr>
    <tr>
      <th>user_id</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>5.0</td>
      <td>3.0</td>
      <td>4.0</td>
      <td>3.0</td>
      <td>3.0</td>
      <td>5.0</td>
      <td>4.0</td>
      <td>1.0</td>
      <td>5.0</td>
      <td>3.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>4.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>2.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>4.0</td>
      <td>3.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>...</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 1682 columns</p>
</div>



The last thing I need to do is de-mean the data (normalize by each users mean) and convert it from a dataframe to a numpy array.


```python
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis = 1)
R_demeaned = R - user_ratings_mean.reshape(-1, 1)
```

# Singular Value Decomposition

Scipy and Numpy both have functions to do the singular value decomposition. I'm going to use the Scipy function `svds` because it let's me choose how many latent factors I want to use to approximate the original ratings matrix (instead of having to truncate it after).


```python
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k = 50)
```

Done. The function returns exactly what I detailed earlier in this post, except that the $\Sigma$ returned is just the values instead of a diagonal matrix. This is useful, but since I'm going to leverage matrix multiplication to get predictions I'll convert it to the diagonal matrix form.


```python
sigma = np.diag(sigma)
```

# Making Predictions from the Decomposed Matrices

I now have everything I need to make movie ratings predictions for every user. I can do it all at once by following the math and matrix multiply $U$, $\Sigma$, and $V^{T}$ back to get the rank $k=50$ approximation of $R$.

I also need to add the user means back to get the actual star ratings prediction.


```python
all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + user_ratings_mean.reshape(-1, 1)
```


```python
preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns)
preds_df.head()
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>movie_id</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
      <th>5</th>
      <th>6</th>
      <th>7</th>
      <th>8</th>
      <th>9</th>
      <th>10</th>
      <th>...</th>
      <th>1673</th>
      <th>1674</th>
      <th>1675</th>
      <th>1676</th>
      <th>1677</th>
      <th>1678</th>
      <th>1679</th>
      <th>1680</th>
      <th>1681</th>
      <th>1682</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>6.488436</td>
      <td>2.959503</td>
      <td>1.634987</td>
      <td>3.024467</td>
      <td>1.656526</td>
      <td>1.659506</td>
      <td>3.630469</td>
      <td>0.240669</td>
      <td>1.791518</td>
      <td>3.347816</td>
      <td>...</td>
      <td>0.011976</td>
      <td>-0.092017</td>
      <td>-0.074553</td>
      <td>-0.060985</td>
      <td>0.009427</td>
      <td>-0.035641</td>
      <td>-0.039227</td>
      <td>-0.037434</td>
      <td>-0.025552</td>
      <td>0.023513</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2.347262</td>
      <td>0.129689</td>
      <td>-0.098917</td>
      <td>0.328828</td>
      <td>0.159517</td>
      <td>0.481361</td>
      <td>0.213002</td>
      <td>0.097908</td>
      <td>1.892100</td>
      <td>0.671000</td>
      <td>...</td>
      <td>0.003943</td>
      <td>-0.026939</td>
      <td>-0.035460</td>
      <td>-0.029883</td>
      <td>-0.027153</td>
      <td>-0.015244</td>
      <td>-0.008277</td>
      <td>-0.011760</td>
      <td>0.011639</td>
      <td>-0.046924</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.291905</td>
      <td>-0.263830</td>
      <td>-0.151454</td>
      <td>-0.179289</td>
      <td>0.013462</td>
      <td>-0.088309</td>
      <td>-0.057624</td>
      <td>0.568764</td>
      <td>-0.018506</td>
      <td>0.280742</td>
      <td>...</td>
      <td>-0.028964</td>
      <td>-0.031622</td>
      <td>0.045513</td>
      <td>0.026089</td>
      <td>-0.021705</td>
      <td>0.002282</td>
      <td>0.032363</td>
      <td>0.017322</td>
      <td>-0.006644</td>
      <td>-0.009480</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.366410</td>
      <td>-0.443535</td>
      <td>0.041151</td>
      <td>-0.007616</td>
      <td>0.055373</td>
      <td>-0.080352</td>
      <td>0.299015</td>
      <td>-0.010882</td>
      <td>-0.160888</td>
      <td>-0.118834</td>
      <td>...</td>
      <td>0.020069</td>
      <td>0.015981</td>
      <td>-0.000182</td>
      <td>0.005593</td>
      <td>0.026634</td>
      <td>0.023562</td>
      <td>0.036405</td>
      <td>0.029984</td>
      <td>0.015612</td>
      <td>-0.008713</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4.263488</td>
      <td>1.937122</td>
      <td>0.052529</td>
      <td>1.049350</td>
      <td>0.652765</td>
      <td>0.002836</td>
      <td>1.730461</td>
      <td>0.870584</td>
      <td>0.341027</td>
      <td>0.569055</td>
      <td>...</td>
      <td>0.019973</td>
      <td>-0.053521</td>
      <td>-0.017242</td>
      <td>-0.007137</td>
      <td>-0.038987</td>
      <td>0.010338</td>
      <td>0.004869</td>
      <td>0.007603</td>
      <td>-0.020575</td>
      <td>0.003330</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 1682 columns</p>
</div>




```python
def recommend_movies(predictions_df, userID, movies_df, original_ratings_df, num_recommendations):
    
    # Get and sort the user's predictions
    user_row_number = userID - 1 # UserID starts at 1, not 0
    sorted_user_predictions = preds_df.iloc[user_row_number].sort_values(ascending=False) # UserID starts at 1
    
    # Get the user's data and merge in the movie information.
    user_data = original_ratings_df[original_ratings_df.user_id == (userID)]
    user_full = (user_data.merge(movies_df, how = 'left', left_on = 'movie_id', right_on = 'movie_id').
                     sort_values(['rating'], ascending=False)
                 )

    print('User {0} has already rated {1} movies.'.format(userID, user_full.shape[0]))
    print('Recommending highest {0} predicted ratings movies not already rated.'.format(num_recommendations))
    
    # Recommend the highest predicted rating movies that the user hasn't seen yet.
    recommendations = (movies_df[~movies_df['movie_id'].isin(user_full['movie_id'])].
         merge(pd.DataFrame(sorted_user_predictions).reset_index(), how = 'left',
               left_on = 'movie_id',
               right_on = 'movie_id').
         rename(columns = {user_row_number: 'Predictions'}).
         sort_values('Predictions', ascending = False).
                       iloc[:num_recommendations, :-1]
                      )

    return user_full, recommendations
```


```python
already_rated, predictions = recommend_movies(preds_df,276, movies_df, ratings_df, 10)
```

    User 276 has already rated 518 movies.
    Recommending highest 10 predicted ratings movies not already rated.



```python
predictions
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>movie_id</th>
      <th>title</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>25</th>
      <td>93</td>
      <td>Welcome to the Dollhouse (1995)</td>
    </tr>
    <tr>
      <th>295</th>
      <td>642</td>
      <td>Grifters, The (1990)</td>
    </tr>
    <tr>
      <th>215</th>
      <td>515</td>
      <td>Boot, Das (1981)</td>
    </tr>
    <tr>
      <th>24</th>
      <td>90</td>
      <td>So I Married an Axe Murderer (1993)</td>
    </tr>
    <tr>
      <th>10</th>
      <td>32</td>
      <td>Crumb (1994)</td>
    </tr>
    <tr>
      <th>209</th>
      <td>509</td>
      <td>My Left Foot (1989)</td>
    </tr>
    <tr>
      <th>15</th>
      <td>48</td>
      <td>Hoop Dreams (1994)</td>
    </tr>
    <tr>
      <th>43</th>
      <td>132</td>
      <td>Wizard of Oz, The (1939)</td>
    </tr>
    <tr>
      <th>406</th>
      <td>824</td>
      <td>Great White Hype, The (1996)</td>
    </tr>
    <tr>
      <th>183</th>
      <td>480</td>
      <td>North by Northwest (1959)</td>
    </tr>
  </tbody>
</table>
</div>




```python
already_rated.head(10)
```




<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user_id</th>
      <th>movie_id</th>
      <th>rating</th>
      <th>title</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>225</th>
      <td>276</td>
      <td>124</td>
      <td>5</td>
      <td>Lone Star (1996)</td>
    </tr>
    <tr>
      <th>416</th>
      <td>276</td>
      <td>173</td>
      <td>5</td>
      <td>Princess Bride, The (1987)</td>
    </tr>
    <tr>
      <th>207</th>
      <td>276</td>
      <td>204</td>
      <td>5</td>
      <td>Back to the Future (1985)</td>
    </tr>
    <tr>
      <th>209</th>
      <td>276</td>
      <td>223</td>
      <td>5</td>
      <td>Sling Blade (1996)</td>
    </tr>
    <tr>
      <th>95</th>
      <td>276</td>
      <td>272</td>
      <td>5</td>
      <td>Good Will Hunting (1997)</td>
    </tr>
    <tr>
      <th>293</th>
      <td>276</td>
      <td>100</td>
      <td>5</td>
      <td>Fargo (1996)</td>
    </tr>
    <tr>
      <th>213</th>
      <td>276</td>
      <td>423</td>
      <td>5</td>
      <td>E.T. the Extra-Terrestrial (1982)</td>
    </tr>
    <tr>
      <th>292</th>
      <td>276</td>
      <td>603</td>
      <td>5</td>
      <td>Rear Window (1954)</td>
    </tr>
    <tr>
      <th>387</th>
      <td>276</td>
      <td>174</td>
      <td>5</td>
      <td>Raiders of the Lost Ark (1981)</td>
    </tr>
    <tr>
      <th>83</th>
      <td>276</td>
      <td>853</td>
      <td>5</td>
      <td>Braindead (1992)</td>
    </tr>
  </tbody>
</table>
</div>



# Conclusion

We've seen that we can make good recommendations with raw data based collaborative filtering methods (neighborhood models) and latent features from low-rank matrix factorization methods (factorization models).

Low-dimensional matrix recommenders try to capture the underlying features driving the raw data (which we understand as tastes and preferences). From a theoretical perspective, if we want to make recommendations based on people's tastes, this seems like the better approach. This technique also scales **significantly** better to larger datasets.

However, we still likely lose some meaningful signals by using a lower-rank matrix. And though these factorization based techniques work extremely well, there's research being done on new methods. These efforts have resulted in various types probabilistic matrix factorization (which works and scales even better) and many other approaches.

f94e2708bd6c776fb98c07ca60f7391f16be9c08
ipynb
Jupyter Notebook
recommender-system/ml-100k/model.ipynb
ankitpandey2708/ml
2d32e91cb5a73dd47306cc32a8eedd379aaee032
MIT
recommender-system/ml-100k/model.ipynb
ankitpandey2708/ml
2d32e91cb5a73dd47306cc32a8eedd379aaee032
MIT
recommender-system/ml-100k/model.ipynb
ankitpandey2708/ml
2d32e91cb5a73dd47306cc32a8eedd379aaee032
MIT
Qwen/Qwen-72B
1. YES
2. YES