title: Cannot write dataframe into hive table after using UDF in Pyspark
body: <p>I am trying to extract first element of probability column (vector data type) using UDF in Pyspark. I was able to get the new dataframe with extracted values in probability column. I also checked the data type of probability column which has changed from vector to float. But I am not able to write the dataframe into hive table. I get numpy module not found error. Deploy mode is client. Is there a workaround other than installing numpy in all the worker nodes ?</p>
  <p>Code -</p>
  <pre><code>spark = (SparkSession
  .builder
  .appName(&quot;Model_Scoring&quot;)
  .master('yarn')
  .enableHiveSupport()
  .getOrCreate()
  )
  
  hive = HiveWarehouseSession.session(spark).build()
  hive.setDatabase(&quot;hc360_models&quot;)
  
  final_ads = spark.read.parquet(&quot;hdfs://DATAHUB/datahube/feature_engineering/final_ads.parquet&quot;)
  
  model = PipelineModel.load(&quot;/tmp/fitted_model_new/&quot;)
  
  first_element=udf(lambda v:float(v[0]),FloatType())
  
  out = model.transform(final_ads)
  
  out = out.withColumn(&quot;probability&quot;,first_element(&quot;probability&quot;)).drop('features').drop('rawPrediction')
  
  out.show(10)
  out.write.mode(&quot;append&quot;).format(HiveWarehouseSession.HIVE_WAREHOUSE_CONNECTOR).option(&quot;table&quot;,&quot;test_hypertension_table_final3&quot;).save()
  
  spark.stop()
  </code></pre>
  <p>Error -</p>
  <p>ImportError: ('No module named numpy', &lt;function _parse_datatype_json_string at 0x7f83370922a8&gt;, (u'{&quot;type&quot;:&quot;struct&quot;,&quot;fields&quot;:[{&quot;name&quot;:&quot;_0&quot;,&quot;type&quot;:{&quot;type&quot;:&quot;udt&quot;,&quot;class&quot;:&quot;org.apache.spark.ml.linalg.VectorUDT&quot;,&quot;pyClass&quot;:&quot;pyspark.ml.linalg.VectorUDT&quot;,&quot;sqlType&quot;:{&quot;type&quot;:&quot;struct&quot;,&quot;fields&quot;:[{&quot;name&quot;:&quot;type&quot;,&quot;type&quot;:&quot;byte&quot;,&quot;nullable&quot;:false,&quot;metadata&quot;:{}},{&quot;name&quot;:&quot;size&quot;,&quot;type&quot;:&quot;integer&quot;,&quot;nullable&quot;:true,&quot;metadata&quot;:{}},{&quot;name&quot;:&quot;indices&quot;,&quot;type&quot;:{&quot;type&quot;:&quot;array&quot;,&quot;elementType&quot;:&quot;integer&quot;,&quot;containsNull&quot;:false},&quot;nullable&quot;:true,&quot;metadata&quot;:{}},{&quot;name&quot;:&quot;values&quot;,&quot;type&quot;:{&quot;type&quot;:&quot;array&quot;,&quot;elementType&quot;:&quot;double&quot;,&quot;containsNull&quot;:false},&quot;nullable&quot;:true,&quot;metadata&quot;:{}}]}},&quot;nullable&quot;:true,&quot;metadata&quot;:{}},{&quot;name&quot;:&quot;_1&quot;,&quot;type&quot;:&quot;integer&quot;,&quot;nullable&quot;:true,&quot;metadata&quot;:{}}]}',))</p>
  <p>Sample data -
  <a href="https://i.stack.imgur.com/ChlS9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ChlS9.png" alt="enter image description here" /></a></p>
  <p>Schema -</p>
  <p>StructType(List(StructField(patient_id,IntegerType,true),
  StructField(carrier_operational_id,IntegerType,false),
  StructField(gender_cde,StringType,true),
  StructField(pre_fixed_mpr_qty,DecimalType(38,8),true),
  StructField(idx_days_in_gap,DecimalType(11,1),true),
  StructField(age,DecimalType(6,1),true),
  StructField(post_fixed_mpr_adh_ind,DecimalType(2,1),true),
  StructField(probability,FloatType,true),
  StructField(prediction,DoubleType,false),
  StructField(run_date,TimestampType,false)))</p>