Why do potential reviewers take a lot of time to reject the request for review?
In addition to the Kerkyras answer, here is what happens all the time when I receive an invitation:


The potential reviewer needs to find the time to skim the paper to form a well founded decision to accept or decline to review.


I want to accept a review when I am sure that I have the expertise and time to do a good review. It takes about 2 minutes to decide if I have the time for a review (usually my conclusion is "no"), but to decide whether I have the expertise is more complex (and if the answer here is "yes", I often agree to review, even if the answer to the former question was "no"…). Hence, a basic skimming of the article is needed and this may take some more minutes. Some days (or even weeks) are extremely busy such that finding just 10 minutes to skim a paper is hard.