When you physically remove and re-insert your smartcard, you must re-initialize your ssh-agent with: ssh-add -e /usr/lib64/opensc-pkcs11.so ssh-add -s /usr/lib64/opensc-pkcs11.so It would be nice to be able to just ask it to prompt for your PIN again to reload access to the keys. Or better yet, when trying to connect, instead of: # ssh host sign_and_send_pubkey: signing failed: agent refused operation it could prompt for the PIN. See more details in https://bugzilla.redhat.com/show_bug.cgi?id=1609055 See a proposed patch in https://bugzilla.mindrot.org/show_bug.cgi?id=2890