The only way to get the villager back is for them to decide to move from the new town. Once they're packed in boxes, you can ask them to come back to your town. This will only work if you've moved out 16 other villagers in the meantime.