I have data which keeps on changing in source. I am pulling that data through sqoop but as the volume is large i cannot keep it as daily truncate load. I want to append the data but the logic should be update and insert. If a record is updating in source by deleting the previous same record, same should be done in hive i.e. old record should be deleted and new one should be inserted/updated.
Below is one such example.

[![enter image description here](https://i.stack.imgur.com/P1iRw.png)](https://i.stack.imgur.com/P1iRw.png)

After some time say 30 mins, the data is updated like this:

[![enter image description here](https://i.stack.imgur.com/BYJdk.png)](https://i.stack.imgur.com/BYJdk.png)

Now, my hive table picked up original record and after some time picked the updated record but inserted it as a different row.

[![enter image description here](https://i.stack.imgur.com/rIPBH.png)](https://i.stack.imgur.com/rIPBH.png)

I want the data to be reflected same as that it is in source without overwriting my table.
(Pyspark code is recommended)

Please help. Thank You.