monotonically_increasing_id function behaviour explanation in Spark
This may have to do with integer overflow as monotonically_increasing_id returns a long datatype.
I have been using monotonically_increasing_id() for a long time and I just discovered a weird behaviour, I need explanations please.
So I have a df with 20 000 lines.
I added an Id column:
val df2 = df.withColumn("Id", monotonically_increasing_id().cast("int"))
And surprise, I didnt get monotonically increasing ids, I found Id=1 on 5 different rows, Id=2 on 2 rows ....
So I thought maybe it was because of Spark distributing my dataframe, to be sure I did the following:
val df2 = df.coalesce(1).withColumn("Id", monotonically_increasing_id().cast("int"))
And the weird behaviour disappeared.
Are my thoughts right ? Doesn't monotonically_increasing_id() manage dataframes repartitions automatically ?
Why didn't I encounter this behaviour previously, I always worked with much bigger dataframes and never did I have this error.
Thanks