Why does Drupal use 403 Forbidden? RFC 2616, the document that defines HTTP/1.1, lays out the definitions of all the HTTP status codes. For 403 Forbidden, it says:


  The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity.  If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.


But that's not how Drupal uses it: depending on the case, it either uses it for the default condition in callbacks (e.g. shortcut_link_add_inline())—essentially making 403 Forbidden equivalent to "Drupal doesn't understand the request"—or it uses it to inform the client it doesn't have access to the page (i.e. due to permissions like being an anonymous user).

In the case of the former ("the request isn't understood"), the more appropriate status code is 400 Bad Request:


  The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.


And in the case of the latter ("the client does not have access to the page due to permissions"), it should either be 401 Unauthorized (if Drupal wants to give away that it's just a permission problem): 


  The request requires user authentication. The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource. The client MAY repeat the request with a suitable Authorization header field (section 14.8). If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials.


or 404 File Not Found (if it doesn't):


  The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.


Emphasis mine (see also the final sentence in the definition for 403 Forbidden).

I realize it's been like this for many years, but what was the original rationale for making drupal_access_denied()—and by extension 403 Forbidden—the only response Drupal sends when it gets a request it doesn't like?
As stated in your quote, 401 Unauthorized must be combined with HTTP authentication, which is not what Drupal is using, so this is not an option.

So 403 is IMHO a perfectly valid response because clients must not repeat their request but might eventually fill out a login form wich whill result in a different request.

Special cases like the shortlink function you posted aren't so clear. I guess the reason for using access denied there is that it is checking the token which is kinda an authentication. But it could just as well use 400 or 404. I guess the reason for using 403 instead of 400 is simply because there is no helper function for returning a 400.

There also has been a discussion regarding the Privatemsg module (that I'm maintaing) whether trying a private message as anonymous user should result in a 403 or a 404 (for privacy reasons). It used to be 404 but has been changed to 403 because users often click on direct links in mail notifications and a 403 page can easily be turned into a login form (as there is a module that does this out of the box). So this is related to the last two sentences of the 403 description.