What is the purpose of implementing cryptographic algorithms in a token? In particular, in PKCS #11. Not a lot of information on the web; I found this explanations. Ignoring the empty marketing talk like “heightened security” and “centralized and secure management”, I managed to extract the following benefits:

secrets do not leave the token,
tamper-resistant audit of the usage of secrets.

In practice, “secrets do not leave the token” is not benefitial. In order to use a secret, a user needs to upload that secret to a program with a user interface (UI). For example, a banking application. This program is not able to execute on a token since tokens have insufficient UI. Hence this program will execute on a UI device to which the token is attached.

Suppose that the UI device is breached.



If cryptographic algorithms are implemented in the token, the attacker can execute any operation that the user can as long as the token is attached.




If cryptographic algorithms are implemented in the UI device, the attacker posesses the user's secrets.



In practice, option 0 is as bad as option 1. For example, for a banking application, the attacker can sign a payment in either case.

Suppose that the UI device is not breached. Then we have equal security independently of where cryptographic algorithms are implemented.


Implementing cryptographic algorithms in the token has disadvantages:

AFAIK, PKCS #11 does not allow backups. If you store all your secrets in a token, it will be valuable not only to attackers, but to you too.
Restricted set of cryptographic algorithms. Only algorithms implemented in a token.
More complex hardware interface and API. [Update 2021-02-15. Here I am comparing to USB mass storage. If a token does not implement cryptographic algorithms, it is essentially an information storage device.]

Am I right?
In practice, “secrets do not leave the token” is not beneficial.

Ouch, I think I would rather like to know where my keys are. This is about the same as saying that you don't care if your house keys got copied or not.

For example, a banking application. This program is not able to execute on a token since tokens have insufficient UI. Hence this program will execute on a UI device to which the token is attached.

In PKCS#11 you can still have key specific PIN or passwords. This is especially important for transactions which is what you are referring to here.
Sometimes additional security measures such as counters are also present. If those are not depleted they can at least be used to check if the key was used unexpectedly. In software the attacker could just reset the counter at will. Cars also have odometer's that are deliberately hard to compromise.
That all said, yes, if an attacker has full access to the keys within the token over extended periods of time then the token is virtually useless.

If cryptographic algorithms are implemented in the token, the attacker can execute any operation that the user can as long as the token is attached.

Absolutely, and there is a balance between usability and security here, which is hard to maintain. E.g. you can setup a HSM in such a way that a person has to login into each session. Then again, you might have some trouble finding a human in the middle of the night that is able to do this upon a sudden restart of a server. This would be fine for personal tokens and smart cards.
Login procedures can also be protected by additional counter measures such as number of logins or delays between password verification checks.

If cryptographic algorithms are implemented in the UI device, the attacker posesses the user's secrets.

There are things like white box cryptography of course. And then there are certainly levels of access. Even in software you can have a secure system run at a different level than the user process. However, in that case you cannot easily protect against side channel attacks on processor usage. So this actually advocates for a hardware device such as a token.

AFAIK, PKCS #11 does not allow backups. If you store all your secrets in a token, it will be valuable not only to attackers, but to you too.

PKCS#11 is a generic token interface. However, most if not all HSM's have proprietary backup mechanisms and / or extensions of the PKCS#11 interface to deal with backups.
Other tokens such as smart cards basically require you to request a new smart card or token - which is fine for signature generation (authentication, non-repudiation). It is bit more tricky when it comes to decryption for obvious reasons.

Restricted set of cryptographic algorithms. Only algorithms implemented in a token.

Absolutely, and this can be a nuisance. And if you enable FIPS mode then only FIPS algorithms will be available. HSM's can receive (signed) updates though. And generally we don't use tokens for research, we tend to use them for standardized algorithms.
Some HSM's actually allow you to run restricted programs as well - although this may have interesting effects on certification of the hardware.

More complex hardware interface and API.

That's debatable. I think the PKCS#11 interface is well specified and it is about the only standardized, generic crypto API out there. It may not suite your language / runtime model perfectly, but the fact that you can use it from a myriad of languages and runtimes has a lot going for it. Many languages contain wrapper libraries around PKCS#11, which are then wrapped object oriented wrappers where required.