How to improve performance of toLocalIterator() in Pyspark
Using toLocalIterator() will eliminate parrallesim . It will work on one partition on driver node one by one. Based on use cases and size of data you can do collect or save file. If you add details can explore more options.
I am trying to convert a Pyspark dataframe to Python list with 140000 rows. I am doing it using toLocalIterator() but it is taking a lot of time. How do i fasten this process?