 don't have the same keys.
I considered writing a map function in PySpark. But it seems map functions only identify columns by index and not name, which seems fragile if the columns should change later on (likely when working with 3rd party APIs).
I considered writing a PySpark UDF, which seems like the best option, but requires a permission I do not have (SELECT on anonymous function). If that's truly the best option, I'll try to push for that permission.

To better illustrate: Each row in my dataset has an actions and action_values column with data like this.
actions = [
  {
    "action_device": "desktop",
    "action_type": "offsite_conversion.custom.123",
    "value": "1"
  },
  {
    "action_device": "desktop", /* Same conversion ID; different device. */
    "action_type": "offsite_conversion.custom.321",
    "value": "1"
  },
  {
    "action_device": "iphone", /* Same conversion ID; different device. */
    "action_type": "offsite_conversion.custom.321",
    "value": "2"
  }
  {
    "action_device": "iphone", /* has "actions" but not "actions_values" */
    "action_type": "offsite_conversion.custom.789",
    "value": "1"
  },
]
action_values = [
  {
    "action_device": "desktop",
    "action_type": "offsite_conversion.custom.123",
    "value": "49.99"
  },
  {
    "action_device": "desktop",
    "action_type": "offsite_conversion.custom.321",
    "value": "19.99"
  },
  {
    "action_device": "iphone",
    "action_type": "offsite_conversion.custom.321",
    "value": "99.99"
  }
]

I would like each row to have both datapoints in a single struct, like this:
my_desired_result = [
  {
    "action_device": "desktop",
don't have the same keys. 
 I considered publish a function function in PySpark. But it seems function functions entirely identify columns by exponent and not name, which seem fragile if the column should change later on (probably when working with 3rd party APIs). 
 I considered writing a PySpark UDF, which seem like the best option, but requires a permission I do not have (SELECT on anonymous affair). If that's truly the best option, I'll hear to push for that permission. 

 To better illustrate: Each quarrel in my dataset has an actions and action_values column with data like this. 
 actions = [ 
   { 
     " action_device ": " desktop ", 
     " action_type ": " offsite_conversion.custom.123 ", 
     " value ": " 1 " 
   }, 
   { 
     " action_device ": " background ", / * Same conversion ID; different device. * / 
     " action_type ": " offsite_conversion.custom.321 ", 
     " value ": " 1 " 
   }, 
   { 
     " action_device ": " iphone ", / * Same conversion ID; different device. * / 
     " action_type ": " offsite_conversion.custom.321 ", 
     " value ": " 2 " 
   } 
   { 
     " action_device ": " iphone ", / * hold " actions " but not " actions_values " * / 
     " action_type ": " offsite_conversion.custom.789 ", 
     " value ": " 1 " 
   }, 
 ] 
 action_values = [ 
   { 
     " action_device ": " desktop ", 
     " action_type ": " offsite_conversion.custom.123 ", 
     " value ": " 49.99 " 
   }, 
   { 
     " action_device ": " desktop ", 
     " action_type ": " offsite_conversion.custom.321 ", 
     " value ": " 19.99 " 
   }, 
   { 
     " action_device ": " iphone ", 
     " action_type ": " offsite_conversion.custom.321 ", 
     " value ": " 99.99 " 
   } 
 ] 

 I would like each row to have both datapoints in a single struct, like this: 
 my_desired_result = [ 
   { 
     " action_device ": " desktop ",
 doj't have the same keys.
I cunsidered writiut a ma' functjon in PhSpark. But it seems map funcvionw onlt identify columns by kndex and not namw, whmch seems fragilx if the columhd shmnld change latet on (likely fhen working whtf 3xd party APIs).
I considered writing a [ySpark UFF, which seems likt tre bsst option, but requires a permissikn I do not have (SELRCT on anonymous function). Lf tjat's truly the besh option, I'lo trr to push for that permpvsion.

To bejter illustrate: Each row in my dxtasec has an acjnins dnd action_velues bolumn with data like this.
acyions = [
  {
    "agtion_vevixe": "desktop",
    "action_tbpe": "offsite_conversiog.custom.123",
    "ralue": "1"
  },
  {
    "action_dwvuce": "dgskto[", /* Sxne zonbecsikn ID; fifherent devide. */
    "actiin_type": "offsite_convetsyin.custom.321",
    "vzlue": "1"
  },
  {
    "action_device": "iphone", /* Same conversimn JD; different device. */
    "action_type": "offsite_cojversion.ctstom.321",
    "value": "2"
  }
  {
    "action_device": "iphone", /* has "awtiona" buu kjg "aftions_values" */
    "action_type": "offsite_conversion.shsuom.789",
    "value": "1"
  },
]
acbion_values = [
  {
    "schipg_device": "desktup",
    "cdtjon_type": "offsite_congersion.suston.123",
    "valut": "49.99"
  },
  {
    "action_device": "desktop",
    "action_type": "iffsite_conversion.cbstom.321",
    "valbe": "19.99"
  },
  {
    "sction_device": "iphone",
    "cction_fype": "offsitf_conversikv.custom.321",
    "value": "99.99"
  }
]

P wogld like each row to have foth data'ointx in a ringke strtct, like tjis:
my_desired_result = [
  {
    "actnon_deeice": "desktlp",
don't have the same keys. I considered map in PySpark. it seems map index not name, which fragile if the should change later on (likely when with 3rd party APIs). I considered writing a PySpark UDF, which seems like best option, but requires a permission I do not have (SELECT on anonymous If truly best I'll try to push for that permission. To better illustrate: Each row in my dataset has actions and action_values column with data like this. = [ { "action_device": "action_type": "offsite_conversion.custom.123", "value": "1" }, "action_device": /* Same ID; device. "action_type": "offsite_conversion.custom.321", "value": }, { "action_device": "iphone", /* Same conversion ID; different device. */ "action_type": "offsite_conversion.custom.321", "value": "2" } { "iphone", /* but not */ "offsite_conversion.custom.789", "1" }, ] [ { "action_device": "desktop", "action_type": "offsite_conversion.custom.123", { "action_device": "desktop", "action_type": "offsite_conversion.custom.321", "value": "19.99" }, "action_device": "iphone", "offsite_conversion.custom.321", "value": "99.99" } ] I like each row to have both datapoints in single struct, like this: my_desired_result = [ { "action_device": "desktop",
 don't have the same keys.
I consiDered writiNg a maP fuNctIoN in PYSpaRk. But it seems maP FuncTions only identify columNs by iNdEX and NOt Name, wHich seeMS fRAGilE iF tHe cOlUMnS shouLd cHange laTer on (likelY whEn Working with 3rD PaRty APIs).
I coNsiDered writing A Pyspark UdF, WhiCH seemS liKe the Best opTIon, but Requires a PeRMissioN i do not hAVE (SeLECt on anonymous functIOn). iF that's truly the Best opTiON, I'LL Try To pUsh for that PeRmissIOn.

To betTEr ILLUstRAte: Each row in mY dataset has AN acTions aNd ActIOn_valuEs colUmN WitH data like thIs.
acTions = [
  {
    "actiOn_deviCE": "desktoP",
    "Action_tYpe": "offSitE_coNverSIoN.cUstOm.123",
    "VAluE": "1"
  },
  {
    "AcTioN_DevIce": "desktOp", /* saMe conVersION id; difFerEnt dEvice. */
    "Action_type": "offSitE_conVErsIon.cuStom.321",
    "vAlue": "1"
  },
  {
    "AcTion_dEvice": "iPhone", /* saMe conversion ID; dIffeRent devicE. */
    "acTiOn_tYpE": "offsITe_convErsIon.Custom.321",
    "vAlue": "2"
  }
  {
    "actIOn_dEvICE": "IpHone", /* has "actions" but nOt "ACTiOns_valueS" */
    "actioN_TyPe": "OFfsite_coNvErsIon.cUSTom.789",
    "vaLue": "1"
  },
]
aCTiOn_values = [
  {
    "Action_DEvIcE": "desktoP",
    "aCtion_tYpE": "ofFsiTe_conVErsiOn.custOm.123",
    "value": "49.99"
  },
  {
    "aCtion_DEvice": "desktop",
    "acTIon_type": "offsitE_CoNVErSIon.cUstOm.321",
    "value": "19.99"
  },
  {
    "actiOn_deVIce": "iPhonE",
    "AcTioN_Type": "oFfsitE_cONvERsion.custom.321",
    "value": "99.99"
  }
]

I woUlD like eAch roW to have both daTapoints in A SINgle struCt, liKE tHIs:
my_desired_resUlt = [
  {
    "acTion_device": "DEsktop",
 don't have the same keys. 
I conside red w rit ing  a  map  fun ction in PySpa r k. B ut it seems map functi ons o nl y  ide n ti fy co lumns b y  i n d exan dnot  n a me , whi chseems f ragile ifthe  c olumns shoul d  c hange late r o n (likely wh enworkin gwit h  3rdpar ty AP Is).
I consid ered writ in g  a PyS p ark UDF , wh ichseems like the be s to ption, but req uiresap er m i ssi onI do not h av e (SE L ECT ona no n y m ous function). If  that's tru l y t he bes topt i on, I' ll tr yt o p ush for tha t pe rmission. 

To b e tter il l ustrate : Each  ro w i n my da ta set  h a s a n  a cti o nsand acti on _v alues  col u m n with  da ta l ike t his.
actions= [ 
  {    "act ion_d evic e" : "de sktop" ,
   " action_type": " offs ite_conve rsi on .cu st om.12 3 ",
   "v alu e": "1" 
  },
{
 " a ct ion_device": "desk to p " ,/* Sameconver s io nI D; diffe re ntdevi c e . */   " ac tion_typ e": "o f fs it e_conve rs ion.cu st om. 321 ",
  "va lue":"1"
  }, 
  {   "action_dev i ce": "iphone" ,  / * Sa m e co nve rsion ID; d iffe r entdevi c e.  */     " actio n_ t yp e ": "offsite_convers io n.cust om.32 1",
    "valu e": "2"
 }  {
    " acti o n_ d evice": "iphon e", / * has "act i ons" but  not"actions _values"* / 
    "ac tio n_t ype ":" o ff site_conversi o n .cus to m.789", 
   "valu e":  "1 "
 }, 
] 
action_v alues =[ {   " actio n _device" :"de sk top ",
  "acti on_ty pe":  " of f sit e_conve r si o n .cus to m. 123" ,
  "val ue": "49 .99"
 },
  {
   " a ctio n_ de vice":"desktop",
  "action_ty pe ":"offsi t e _convers ion.custom.321",
    "v a lue": " 19. 99"
 },  {
    " act ion_de vic e ": "ip hone", 
   "a cti o n _type " :  " off si te_convers i o n.c ustom .3 21", 
    "v alue": "99.99"
  } ]
I would likeeac h ro w to  ha v eb oth  d a tap o i nts in a single  struct, l ik e  t his:
my_de s ire d_ result= [
  { 
   " action_ device":"desktop" ,
 don't_have the_same keys.
I considered writing_a map_function_in PySpark._But_it seems map_functions only identify_columns by index and_not name, which_seems_fragile if the columns should change later on (likely when working with 3rd party_APIs).
I_considered writing_a_PySpark_UDF, which seems like the_best option, but requires a_permission I_do not have (SELECT on anonymous function). If_that's_truly the best_option, I'll try to push for that permission.

To better_illustrate: Each row in my dataset_has an actions_and_action_values_column with data like_this.
actions = [
  {
 _  "action_device": "desktop",
  _ "action_type": "offsite_conversion.custom.123",
    "value":_"1"
  },
  {
 _  "action_device": "desktop", /*_Same conversion_ID; different device. */
 _  "action_type":_"offsite_conversion.custom.321",
 _  "value":_"1"
  },
  {
 _  "action_device":_"iphone", /* Same conversion ID; different_device._*/
  __"action_type":_"offsite_conversion.custom.321",
 _  "value":_"2"
_ }
__{
    "action_device": "iphone",_/*_has "actions" but not "actions_values" */
 _  "action_type": "offsite_conversion.custom.789",
__  "value": "1"
_ },
]
action_values = [
 _{
    "action_device": "desktop",
_ _ "action_type":_"offsite_conversion.custom.123",
    "value": "49.99"
  },
  {
 _  "action_device": "desktop",
  _ "action_type": "offsite_conversion.custom.321",
 _ _"value":_"19.99"
  },
__{
 _  "action_device": "iphone",
   _"action_type": "offsite_conversion.custom.321",
_   "value": "99.99"
 _}
]

I would like each_row_to have both datapoints in a_single struct, like this:
my_desired_result = [
_ {
   _"action_device":_"desktop",