Broadcast large array in pyspark (~ 8GB)
This is not the problem of PySpark, this is a limit of Spark implement. 
Spark use a scala array to store the broadcast elements, since the max Integer of Scala is 2*10^9, so the total string bytes is 2*2*10^9 = 4GB, you can view the Spark code.
In Pyspark, I am trying to broadcast a large numpy array of size around 8GB. But it fails with the error "OverflowError: cannot serialize a string larger than 4GiB". I have 15g in executor memory and 25g driver memory. I have tried using default and kyro serializer. Both didnot work and show same error. 
Can anyone suggest how to get rid of this error and the most efficient way to tackle large broadcast variables?