Many smartcards are capable of storing multiple PINs and multiple RSA keys.  Some users may also have more than one smartcard in active use at a given time (though this seems less likely than 2 or more IDs on a card).

The current smartcard implementation appears to be capable of dealing with only a single PIN on a single card.  While this makes sense for a single instance of ssh, a long-running ssh-agent connection might reasonably want to deal with multiple identities or multiple cards.

Also problematic with the agent is that it doesn't associate any given identity with any particular card or reader.  So if a second card or reader is inserted in the local host (even if it's not used by the agent), there's a potential for dangerous things like sending the cached PIN to the wrong card.

I'm afraid i don't have a fix for this behavior at the moment, but i wanted to raise the issue and create a place for discussion about it.

I think that the right thing would be to adjust the agent (if compiled with smartcard support) to associate each hardware-based identity with a specific card and a specific PIN.