PySpark: How do I specify dropna axis in PySpark transformation?
I  know this is a bit late, but I struggled with this also. This is my attempt at removing null columns from a Spark Dataframe.
from pyspark.sql.functions import when, isnull

colsthatarenull = df.select([(when(isnull(c), c)).alias(c) for c in df.columns]).first().asDict()
namesofnullcols = {key:val for key, val in colsthatarenull.items() if val != None}.values()
df = df.drop(*namesofnullcols)