The ssh-add flags -D and -d (remove all identities and remove one identity, respectively), are not pkcs11-aware. They will remove pkcs11-backed identities, but will not unload the corresponding pkcs11 provider from ssh-agent. This puts the user into a broken state, where the identity cannot be re-added, because ssh-agent will refuse subsequent requests to load the provider. Steps to reproduce: 1. Configure a smart card (e.g. Yubikey in PIV mode) as an SSH key. 2. Add that key to ssh-agent. 3. Remove that key from ssh-agent (using either ssh-add -D or -d) 4. Add that key to ssh-agent. Expected results: Key is successfully added to ssh-agent. Actual results: ssh-add fails with "agent refused operation". I've looked at the code, and it appears that register_pkcs11_provider (https://github.com/openssh/openssh-portable/blob/master/ssh-pkcs11.c#L1470) fails if a PKCS#11 provider already exists. However, PKCS#11 providers are never unloaded. There is a pkcs11_del_provider but it is never called. Environments reproduced on: Ubuntu 19.10, Fedora Version of OpenSSH: git commit b2491c28, latest at time of writing. Example output demonstrating the problem (with a Yubikey in PIV mode inserted): $ SSH_AUTH_SOCK=/tmp/ssh-dhfNCpXwSk8B/agent.21022; export SSH_AUTH_SOCK; $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so Enter passphrase for PKCS#11: Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so": agent refused operation $ SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export SSH_AUTH_SOCK; $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so Enter passphrase for PKCS#11: Card added: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so $ ssh-add -D All identities removed. $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so Enter passphrase for PKCS#11: Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so": agent refused operation In a separate terminal: $ ./ssh-agent -d SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export SSH_AUTH_SOCK; echo Agent pid 21116; debug2: fd 3 setting O_NONBLOCK debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 20 debug1: process_add_smartcard_key: add /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so debug1: pkcs11_start_helper: starting /usr/local/libexec/ssh-pkcs11-helper -vvv debug1: process_add debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.19 debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot 0: label <SSH key> manufacturerID <piv_II> model <PKCS#15 emulate> serial <00000000> flags 0x40d debug1: have 1 keys debug1: pkcs11_k11_free: parent 0x5598092d2f30 ptr 0x5598092d2b10 idx 1 debug1: pkcs11_provider_unref: 0x559809258df0 refcount 2 debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 19 debug1: process_message: socket 1 (fd=4) type 9 debug2: fd 4 setting O_NONBLOCK debug1: process_message: socket 1 (fd=4) type 20 debug1: process_add_smartcard_key: add /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so debug1: process_add debug1: check 0x559809258df0 /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so debug1: pkcs11_register_provider: provider already registered: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so Also discussed on the mailing list at https://marc.info/?l=openssh-unix-dev&m=158234415817939&w=2