work on distinct elements of RDD-pyspark
So what i did was to define a function that checks if I have seen that name in the past and then use the .filter(myfunc) to only work with the names i want...
The problem now is that in each new streaming window the function is being applied from the beggining , so if i have seen the name John in the first window 7 times , i will keep it only once , but then if i have seen the name John in the second window 5 times i will keep it again once...
I want to keep the name John once for all the streaming application...
Any thoughts on that ?
I am receiving data from Kafka into a Spark Streaming application. It comes in the format of Transformed DStreams. I then keep only the features i want.
features=data.map(featurize)
which gives me the "name","age","whatever".
I then want to keep only the name of all the data
features=data.map(featurize).map(lambda Names: Names["name"]
Now, when i print this command, i get all the names coming from the streaming application, but i want to work on each one separately.
More specifically, I want to check each name and if I have already came across it in the past i want to apply a function on it. Otherwise i will just continue with my application. So I want each name to be a string so that I can insert it into my function that checks if one string has been seen in the past.
I know that foreach will give me each RDD , but still I want to work on each name of the RDDs separately.
Is there any way in pyspark to do so?