I'm using SSH with a token storing my keys. Trying to limit the tried indentities with IdentitiesOnly and IdentityFiles works as long as I'm using the ssg-agent. But if I instead configure the token with PKCS11Provider, ssh ignores IdentitiesOnly and tries all identities present on the token.

My config includes

Host *
IdentitiesOnly yes
IdentityFile ~/.ssh/token.pub
PKCS11Provider /usr/lib/libeTPkcs11.so

There are no host specific entries for any of these.

If I now try to ssh to any system, the output of ssh -vvv shows that it loads the provider and counts the keys. Later it prints

debug1: Will attempt key: /usr/lib64/libeTPkcs11.so RSA SHA256:... token
debug1: Will attempt key: /usr/lib64/libeTPkcs11.so RSA SHA256:... token
debug1: Will attempt key: /usr/lib64/libeTPkcs11.so RSA SHA256:... token

for all identities on the token. The hashes are distinct, so it's not printing the same multiple times.

Doing the same with adding the provider to ssh-agent and removing the PKCS11Provider key, shows only one key as attempted.

I may be way off here, but the code in sshconnect2:pubkey_prepare around line 1592 seems to iterate the files list limiting itself to external keys. For each external key it iterates the files list again using the same filter. As it finds the same key again it compares the two and thus adds it to the preferred list. Running this in gdb shows the two keys id and id2 pointing to the same address when adding the key to the preferred list. It looks like it might be iterating the external keys and trying to find a matching IdentityFile to apply the IdentitiesOnly filter but fails to do so. But as I said, that might be a misinterpretation and not related.