title: iterate over pyspark Dataframe and then for each row interact with mongoDB
body: <p>I have a pyspark Dataframe and now i want to iterate over each row and insert/update to mongoDB collection. </p>
  
  <pre><code> #Did every required imports 
   #dataframe
   +---+----+
   |age|name|
   +---+----+
   | 30|   c|
   |  5|   e|
   |  6|   f|
   +---+----+
    db = mongodbclient['mydatabase']
    collection = db['mycollection']
    #created below function to insert/update
    def customFunction(row):
        key = {'name':row.name}
        data = dict(zip(columns,[row.x for x in columns]))
        collection.update(key, data, {upsert:true})
        #return a_flag #commented it as of now, a_flag can be 0 or 1
  </code></pre>
  
  <p>If a name exist in the mongoDB collection 'mycollection' it should update that row/record else insert that new record.</p>
  
  <p>i am getting following error when tried to map this function over spark-dataframe</p>
  
  <pre><code> result = my_dataframe.rdd.map(customFunction)
   #.....TypeError: can't pickle _thread.lock objects....
   #AttributeError: 'TypeError' object has no attribute 'message'
  </code></pre>
  
  <p>Can anybody please figure out 'what is wrong here in that function and/or anywhere else' or please suggest if any other alternative is there to this type of task.</p>
  
  <p>Basically iterate each row(without a collect call is that even possible??)</p>
  
  <p>And, on each row apply  a function to run outside-spark work.</p>
  
  <p>Please suggest, Thanks in Advance..:)</p>
  
  <p>My data in mongoDB</p>
  
  <pre><code>name  age
   a    1
   b    2
   c    3 #new update should make age as 30 and 2 more new recs should inserted
  </code></pre>