I am battling with a datatype and wondered if someone could assist me. I am doing a structured stream using Event Grid, Azure Databricks into Azure Synapse. The file in question is parquet format and I am using scala.

When I do a printschema on the file, there are two columns that come up as decimal (`6,2`) one of which is causing an issue. The column that is giving issues has null values, the other decimal column has integer values.

I create StructType and for these two fields, I use the type DoubleType for the two decimal fileds. When I do this, I get the following error message for the decimal column that contains nulls

```
Error while reading file wasbs:REDACTED_LOCAL_PART@storage.blob.core.windows.net/part-00000-xxx-xxx-xx-xxxx-xxx.c000.snappy.parquet. Parquet column cannot be converted. Column: [FlUsed], Expected: DoubleType, Found: INT32

```

Then when if I change the structure to an IntegerType as suggested in the error message, and run the stream I get the following error message.

```
java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary

```

If I convert the parquet to csv first then everything works perfectly but this is not a long term solution. From what I have read, this is clearly an issue with the datatype but I don't know what to do to solve it.

Any help would be greatly appreciated.