I have a SQLContext data frame derived from pandas data frame consisting of several numerical columns. I want to perform multivariate statistical analysis using the pyspark.mllib.stats package. The statistics function expects a RDD of vectors. I could not convert this data frame into RDD of vectors. Is there a way to convert the data frame? Code: rdd = sqlCtx.createDataFrame(df_new) summary = Statistics.colStats(rdd)I am getting df_new from  df_new = df.applymap(lambda s: dic.get(s) if s in dic else s) #df is a pandas dataframeI am getting a PY4JJava error at the summary line. The issue is with the format of rdd. 
How to convert a SQLContext Dataframe to RDD of vectors in Python?
statistics;apache spark
The Dataframe Python API exposes the RDD of a Dataframe by calling the following :df.rdd # you can save it, perform transformations of course, etc. df.rdd returns the content as an pyspark.RDD of Row.You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given. Note 1: dfis the variable define our Dataframe. Note 2: this function is available since Spark 1.3