Pyspark: Filter dataframe based on separate specific conditions
You can use the filter method on Spark's DataFrame API:
df_filtered = df.filter(&quot;df.col1 = F&quot;).collect()
which also supports regex
pattern = r&quot;[a-zA-Z0-9]+&quot;
df_filtered_regex = df.filter([df_filtered.c.rlike(pattern) for c in df.columns]).collect()`