databricks: Add a column and insert rest of the data in a table
Create an empty dataframe having all the columns let say X. Then you can concatenate all the other dataframes into X. Then save X directly.
I am new to databricks and sql and want to add some data there.
I am using python notebook in azure databricks. I have created a very big empty delta table. Columns here:
Id| A| B| C| D| E| F| G| H| I| J| K (A,B,C.... are column names)
I will parse log files as they will appear them in blob and create dataframes. The dataframes could like this.
DF1
A| B| C| D|   (A,B,C.... are column names)
DF2
A| B| D| E|   (A,B,C.... are column names)
DF3
A| B| D| F|   (A,B,C.... are column names)
I want to insert all of these data frames in the delta table. In addition I will also need to add Id(log_file_id). Is there a way to insert data in this manner to the tables?