Remove all columns where the entire column is null
[Updated]: Just realized it is about pyspark!
It is still simple! A concrete example (idea heavily borrowed from this answer):
Creating a dummy dataset
import pandas as pd
import numpy as np 
import pyspark.sql.functions as sqlf

main= pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

main[&quot;E&quot;]= np.NAN
main[&quot;F&quot;]= np.NAN

df = sqlContext.createDataFrame(main)

Function to drop Null columns
def drop_null_columns(df):
  
    &quot;&quot;&quot;
    This function drops columns containing all null values.
    :param df: A PySpark DataFrame
    &quot;&quot;&quot;
    
    null_counts = df.select([sqlf.count(sqlf.when(sqlf.col(c).isNull(), c)).alias(c) for c in df.columns]).collect()[0].asDict()
    to_drop = [k for k, v in null_counts.items() if v &gt;= df.count()]
    df = df.drop(*to_drop)
    
    return df

Outcome
df_dropped = drop_null_columns(df)