Do we must make a complex query in PySpark or a simple, and use .filter / .select?
First of all I think we should be careful when dealing with small data in our Spark programs. It was designed to give you parallel processing for big data.
Second, we have something like Catalyst query optimizer and lazy evaluation, which is a good tool for Spark to optimize everything what was put either in SQL query or API call transformations.
I have a question. Suppose I run a python script on the server where my data are stored. What is the faster way to have a spark dataframe of my data between :

Make a complex query with lot of conditions but it return me the exact dataframe I need or
Make a simple query and make the dataframe I need with .filter / .select

You can also suppose that the dataframe I need is small enough to fit on my RAM.
Thanks