Plot RDD data using a pyspark dataframe from csv file I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering specific rows, I would like to visualize it by plotting 2 columns (latitude and longitude) using matplotlib.

This is what I've done so far:

import packages:

from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
from pyspark.sql.functions import *
import matplotlib.pyplot as plt


build spark session:

spark = SparkSession.builder \
     .master("master:IP") \
     .appName("plot track") \
     .config(conf=SparkConf()) \
     .getOrCreate()


read .csv file:

df = spark.read.csv('data.csv')


filter rows according to values in a specific column ("_c1"):

one_ID = df.filter("_c1 == 1234")


save column data? (Latitude data is in column "_c3" and longitude data is in column "_c4")

latitude = one_ID.rdd.map(lambda y: y._c3)
longitude = one_ID.rdd.map(lambda x: x._c4)


plot:

plt.plot(longitude,latitude, 'o-')
plt.show()


I thought this would work but it does not. I get the error -
- 'TypeError: float() argument must be a string or a number'

Any help would be greatly appreciated. Thanks.
There is no need of two rdd values,

plotdf=df.select(_c3,_c4);
plotdf.plot( style=[])
plt.show()


If you still face the error, please follow the link and set index.