How to convert a csv file to an avro file using PySpark?
You can read the csv file into a dataset/dataframe using spark and use databricks library to write it as avro. Something like:
dataset.write.format("com.databricks.spark.avro").save("your output path")
I am working in Google Cloud Platform and I am trying to use Pyspark to convert a csv file into an avro file. I have seen a lot of websites but I haven't been able to implment the solution. Thank you in advance. :)