PySpark - Pull all files into a dataframe based off filemask
If all account, customer, sales are under one date time partition, then you can use
accountdf = spark.read.parquet("wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/account*.parquet")
This is day 1 of my journey into python (day 0 was a right pita).
I have am using Azure DataBricks (Python/Pyspark) and ADLS Gen2 Storage container.
Within my container I have the below partition structure. Which is data stored post ADF Pipeline.
ARCHIVE/[YEAR]/[Month]/[Day]/[Time]/[approx 150 files].parquet (account.parquet, customer.parquet, sales.parquet etc)
What I would like to achieve is to be able to do is to traverse the container and for example any files where the filemask contains "account" send to the accountdf.
This would allow me to be able to compare the data frame with the data in the synapse pool to ensure there are no gaps within the data.