Do we must make a complex query in PySpark or a simple, and use .filter / .select?
The key points to mind is

Restrict/filter data to maximum possible level while loading into dataframe, so as only needed data resides in dataframe
for non file sources: filtering data at source by using native filter and fetching only needed columns (aim for minimum data transfer).
for file sources: restricting/modifying data in file source is not feasible. so the first operation is to filter data once loaded

In complex operations first perform narrow transformations (filters, selecting only needed columns) and then perform wide transformations(joins, ordering) which involves shuffle towards the end, so that less data will be shuffled between worker nodes.


The less the shuffles the faster your end dataframe would be.
I have a question. Suppose I run a python script on the server where my data are stored. What is the faster way to have a spark dataframe of my data between :

Make a complex query with lot of conditions but it return me the exact dataframe I need or
Make a simple query and make the dataframe I need with .filter / .select

You can also suppose that the dataframe I need is small enough to fit on my RAM.
Thanks