Auto block contact but unblock it after a couple days
Try Root Call Blocker, but your phone has to be rooted. Root Call Blocker has many options you can set (im not sure if timing of auto block and auto-unblock is available though). Even while the person is calling, there is a swipe button right next to the swipe green button, and you can end the call and block the person then and there.

If your phone is not rooted, then you can use AFirewall

This is also feature rich with options, check to see if there are any auto-block and unblock timers.