title: Multiplying two RDD in pyspark
body: <p>I am new to pyspark. I have been trying to multiply two sparse RDD. The code whichI have tried generates two sparse matrices and I have written a function to multiply the two RDD but I think this is not the solution as the computations does not occur in parallel. Can someone help me with it? How can I multiply the RDD in parallel? I tried out a lot of resources on the sites but could not come up with a solution.</p>
  
  <pre><code>import findspark
  findspark.init()
  import numpy as np
  import pyspark
  import random
  from scipy.sparse import rand
  sc = pyspark.SparkContext(appName="matrix")
  
  np.random.seed(42)
  n=4
  x = rand(n, n, density=0.25)
  y = rand(n, n, density=0.25)
  A = x.A
  B = y.A
  rdd_x = sc.parallelize(A)
  rdd_y = sc.parallelize(B)
  def multiply(r1, r2):
     A = r1.collect()
     B = r2.collect()
     result = []
     for i in range(len(B[0])): 
         total = 0
         for j in range(len(A)): 
             total += A[j] * B[j][i]
         result.append(total)
     return result
  C = multiply(rdd_x,rdd_x)
  print(C)
  sc.stop()
  </code></pre>