Plotting in PySpark? I have been searching for methods to plot in PySpark. 

I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are:


through method collect() which brings data into 'local' Python session and plot
through method toPandas() which converts data to 'local' Pandas Dataframe.


The problem is that these both are very time-consuming functions. Is there any method by which we can plot data residing in Spark session directly (not importing it into the local session)?
No, there is no such method, I have found out.

The reason is, plotting libraries run on a single machine and expect a rather sample dataset. Data on Spark is distributed among its clusters and hence needs to be brought to a local session first, from where it can be plotted.

That is why methods such as collect(), toPandas() are needed.