Bug 3188

Summary: Problems creating a second ecdsa-sk key for a second Yubikey
Product: Portable OpenSSH Reporter: David Walker <David>
Component: ssh-keygenAssignee: Assigned to nobody <unassigned-bugs>
Status: CLOSED WORKSFORME    
Severity: normal CC: djm, ismail, tyson
Priority: P5    
Version: 8.3p1   
Hardware: Other   
OS: Linux   
Attachments:
Description Flags
ssh-keygen with Yubikey 5c Nano
none
ssh-keygen with Yubikey 5 NFC none

Description David Walker 2020-06-27 03:42:33 AEST 
I've been playing around with ecdsa-sk type keys since 8.3p1 came out in a recent openSUSE Tumbleweed snapshot. It works fine for me, except when I try to add a second Yubikey. My first key (a Yubikey 5c Nano) was set up with "ssh-keygen -t ecdsa-sk" using the default key files (~/.ssh/id_ecdsa_sk*), but when I try to do the same for a second key (a Yubikey 5 NFC, using USB), the light doesn't flash on the Yubikey when I'm prompted to press the Yubikey's button, so pressing the button has no effect, I'm not prompted for where to store the new key pair, and no key is created.  I have to use ctrl-C to get out of ssh-keygen.

After this happens, the first key will not work for ssh authentication for a while (a few hours to a couple of days), even if I reboot the system. Both keys continue to work with a browser (Vivaldi), though.

Any ideas of how to diagnose what's going on?
Comment 1 Damien Miller 2020-06-29 13:29:07 AEST 
Please attach debug logs for a successful case and a failed case - use "ssh-keygen -vvv ..." to increase the verbosity.
Comment 2 David Walker 2020-06-30 02:56:44 AEST 
Created attachment 3422 [details]
ssh-keygen with Yubikey 5c Nano
Comment 3 David Walker 2020-06-30 02:57:33 AEST 
Created attachment 3423 [details]
ssh-keygen with Yubikey 5 NFC
Comment 4 David Walker 2020-06-30 03:07:43 AEST 
(In reply to Damien Miller from comment #1)
> Please attach debug logs for a successful case and a failed case -
> use "ssh-keygen -vvv ..." to increase the verbosity.

I've attached two logs; here's how I created them:

1. Using my "first" Yubikey (the 5c Nano), I generated a test key with "ssh-keygen -vvv -t ecdsa-sk". The log is in the "ssh-keygen log with Yubikey 5c Nano" attachment.

2. I removed the "first" key and inserted the "second" key (the 5 NFC).

3. I tried to generate a test key with "ssh-keygen -vvv -t ecdsa-sk"; the log is in the "ssh-keygen log with Yubikey 5 NFC" attachment. The light on the 5 NFC never started flashing, and when I pressed its button anyway, it appears to have sent an HOTP string.
Comment 5 Damien Miller 2020-06-30 11:37:09 AEST 
Thanks, it looks like libfido is unable to communicate with your yk5nfc, but by the OTP it dumped into your keyboard buffer it does seem to be attached as far as the system is concerned.

Can you try to talk to the card using a tool like ykman? E.g.

$ ykman info 
Device type: YubiKey 5 Nano
Serial number: 8331229
Firmware version: 5.1.2
Form factor: Nano (USB-A)
Enabled USB interfaces: FIDO+CCID

Applications
OTP             Disabled
FIDO U2F        Enabled 
OpenPGP         Enabled 
PIV             Enabled 
OATH            Disabled
FIDO2           Enabled
Comment 6 David Walker 2020-06-30 11:59:26 AEST 
(In reply to Damien Miller from comment #5)

> Can you try to talk to the card using a tool like ykman? E.g.

Here it is with the 5 NFC inserted:

  > ykman info
  Device type: YubiKey 5 NFC
  Serial number: 13377198
  Firmware version: 5.2.6
  Form factor: Keychain (USB-A)
  Enabled USB interfaces: OTP+FIDO+CCID
  NFC interface is enabled.

  Applications	USB    	NFC    
  OTP     	Enabled	Enabled	
  FIDO U2F	Enabled	Enabled	
  OpenPGP 	Enabled	Enabled	
  PIV     	Enabled	Enabled	
  OATH    	Enabled	Enabled	
  FIDO2   	Enabled	Enabled	

And here's the 5c Nano:

  > ykman info
  Device type: YubiKey 5C Nano
  Serial number: 11541414
  Firmware version: 5.2.4
  Form factor: Nano (USB-C)
  Enabled USB interfaces: OTP+FIDO+CCID

  Applications
  OTP     	Enabled	
  FIDO U2F	Enabled	
  OpenPGP 	Enabled	
  PIV     	Enabled	
  OATH    	Enabled	
  FIDO2   	Enabled

Note that *neither* Yubikey works with ssh (and its associated tools) for a period of time after the ssh-keygen failure, but both continue to work with browsers (Vivaldi, in particular). Does ssh-sk-helper have some kind of cache? The fact that things start working after a period of time is suspicious to me.
Comment 7 Damien Miller 2020-07-03 17:12:32 AEST 
No, ssh-keygen doesn't cache anything. The persistent failure could be explained by the USB bus getting confused or something hogging it.

I think the next thing to try is to put libfido2 in debugging mode. It might be possible to do this by setting the FIDO_DEBUG environment variable, but it's also possible to do it by editing sk-usbhid.c in the OpenSSH source and uncommenting the "/* #define SK_DEBUG 1 */" line.
Comment 8 David Walker 2020-07-05 17:45:14 AEST 
(In reply to Damien Miller from comment #7)
> I think the next thing to try is to put libfido2 in debugging mode.
> It might be possible to do this by setting the FIDO_DEBUG
> environment variable, but it's also possible to do it by editing
> sk-usbhid.c in the OpenSSH source and uncommenting the "/* #define
> SK_DEBUG 1 */" line.

Since my previous comment, I've installed an openSUSE Tumbleweed patched version of libfido2 that doesn't have this problem. (See https://github.com/Yubico/libfido2/issues/190.) Would you still like me to test? I could downgrade libfido2 and do that, if it's still useful now that the libfido2 project is working on the issue.

By the way, now that I'm able to generate keys for multiple Yubikeys, I've run into another issue. I'll open another bug for that.
Comment 9 Damien Miller 2020-07-17 14:00:15 AEST 
no need to retest, it looks like problem is in libfido2
Comment 10 Damien Miller 2021-03-04 09:54:02 AEDT 
close bugs that were resolved in OpenSSH 8.5 release cycle