How to find pyspark dataframe memory usage?
You can persist dataframe in memory and take action as df.count(). You would be able to check the size under storage tab on spark web ui.. let me know if it works for you.
For python dataframe, info() function provides memory usage.
Is there any equivalent in pyspark ?
Thanks