Using security keys similar to the use of TouchID for removing need of passwords
It is not currently possible to get the system dialogs for TouchID to also work with a security key, such as for example the popular Yubikey product. Third party applications can however provide support for products such as the Yubikey through similar popups windows.

One method of using security keys in a manner that resembles TouchID is to set your system up to them as a kind of "virtual smart card". This allows you to use for example a Yubikey as a means of logging in to your Mac, and for authenticating with Keychain, etc.

The specific method of achieving that depends on the type of security key you have. In general you need to configure the system and/or install drivers to let the security key function as a "smart card".

For the Yubikey product you can follow this guide from the manufacturer:

https://support.yubico.com/support/solutions/articles/15000006468-using-your-yubikey-as-a-smart-card-in-macos