How to build a sparse matrix in PySpark?
Question: I am new to Spark. I would like to make a sparse matrix a user-id item-id
matrix specifically for a recommendation engine. I know how I would do this in
python. How does one do this in PySpark? Here is how I would have done it in
matrix. The table looks like this now.

    
    
    Session ID| Item ID | Rating
         1          2       1
         1          3       5
    
    
    
        import numpy as np
    
        data=df[['session_id','item_id','rating']].values
        data
    
        rows, row_pos = np.unique(data[:, 0], return_inverse=True)
        cols, col_pos = np.unique(data[:, 1], return_inverse=True)
    
        pivot_table = np.zeros((len(rows), len(cols)), dtype=data.dtype)
        pivot_table[row_pos, col_pos] = data[:, 2]
    


Answer: Like that:

    
    
    >>> from pyspark.mllib.linalg.distributed import CoordinateMatrix, MatrixEntry
    >>> table = sqlContext.createDataFrame(
    ...     sc.parallelize([[1, 2, 1], [1, 3, 5]])
    ... )
    >>> mat = CoordinateMatrix(table.rdd.map(lambda row: MatrixEntry(*row)))
    

