PySpark Data Frames when to use .select() Vs. .withColumn()?
@Robert Kossendey You can use select to chain multiple withColumn() statements without suffering the performance implications of using withColumn. Likewise, there are cases where you may want/need to parameterize the columns created. You could set variables for windows, conditions, values, etcetera to create your select statement.
I'm new to PySpark and I see there are two ways to select columns in PySpark, either with ".select()" or ".withColumn()".
From what I've heard ".withColumn()" is worse for performance but otherwise than that I'm confused as to why there are two ways to do the same thing.
So when am I supposed to use ".select()" instead of ".withColumn()"?
I've googled this question but I haven't found a clear explanation.