Hostage taker and player death
It's removed from the game if you own it. When a player loses the game, they leave the game and every card they own does as well. You'll take back the physical card, and they lose control of whatever it was because it's vanished out of the game. This happens no matter where it is (in exile, on the stack, on the battlefield, etc). The rules that handle this don't leave behind a copy for them to use — the thing is gone.
There's a pragmatic reason for this: the cards are your legal property, and once you're done in that game you're entitled to leave and take your stuff with you. The rules won't presume they can leave someone else in possession of your stuff, so those rules just say your stuff goes away.
In 1v1, the game would already be over when you lose, so I presume you're not actually playing 1v1. In multiplayer, we go by these rules:

104.5. If a player loses the game, he or she leaves the game. If the game is a draw for a player, he or she leaves the game. The multiplayer rules handle what happens when a player leaves the game; see rule 800.4.
800.4a When a player leaves the game, all objects (see rule 109) owned by that player leave the game and any effects which give that player control of any objects or players end. [...] This is not a state-based action. It happens as soon as the player leaves the game.