Why does Drupal use 403 Forbidden?
As stated in your quote, 401 Unauthorized must be combined with HTTP authentication, which is not what Drupal is using, so this is not an option.

So 403 is IMHO a perfectly valid response because clients must not repeat their request but might eventually fill out a login form wich whill result in a different request.

Special cases like the shortlink function you posted aren't so clear. I guess the reason for using access denied there is that it is checking the token which is kinda an authentication. But it could just as well use 400 or 404. I guess the reason for using 403 instead of 400 is simply because there is no helper function for returning a 400.

There also has been a discussion regarding the Privatemsg module (that I'm maintaing) whether trying a private message as anonymous user should result in a 403 or a 404 (for privacy reasons). It used to be 404 but has been changed to 403 because users often click on direct links in mail notifications and a 403 page can easily be turned into a login form (as there is a module that does this out of the box). So this is related to the last two sentences of the 403 description.