How to get Python in Qubole to save CSV and TXT files to Azure data lake?
If I understand your question correctly,I believe you are unable to download the result of pyspark command output into text or CSVs while you are able to do so for the spark sql command command output in a nice tabular format.
Unfortunately, there is no direct field separator for the output text for a Python or Shell command outputs. You will need to get your output comma separated so you can download the raw output and save it as a csv.
If this is not what you meant, Please share more details as to what exactly are you trying to do along with screenshots details. As that will help us answer your question better.
I have Qubole connected to Azure data lake, and I can start a spark cluster, and run PySpark on it. However, I can't save any native Python output, like text files or CSVs. I can't save anything other than Spark SQL DataFrames.
What should I do to resolve this?
Thank you in advance!