Created attachment 3627 [details] verbose output of ssh attempt attempting to ssh with two identity files fails with `Permission denied (publickey)` for example: ``` ssh -i signed-cert.pub -i ~/.ssh/id_rsa username@10.0.23.5 ``` only able to resolve by downgrading to 8.6. this is a fairly common process to ssh with two `-i` identity files, for example if you're using vault as a certificate authority. as described here: https://developer.hashicorp.com/vault/docs/secrets/ssh/signed-ssh-certificates#client-ssh-authentication full ssh with -vvv attached
Comment on attachment 3627 [details] verbose output of ssh attempt The handling of -i hasn't changed as far as I know. [...] >debug1: identity file /Users/steve/.config/zaccess/penguin.randomhostname.com.cert type 4 >debug1: identity file /Users/steve/.config/zaccess/penguin.randomhostname.com.cert-cert type -1 >debug1: identity file /Users/steve/.ssh/vault type 0 >debug1: identity file /Users/steve/.ssh/vault-cert type -1 This doesn't exactly match the example invocation, but it indicates that two keys were loaded. [...] >debug1: Offering public key: /Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit >debug1: send_pubkey_test: no mutual signature algorithm I think this is your problem: ssh-rsa was disabled by default in 8.8 (https://www.openssh.com/releasenotes.html#8.8). You can test this by adding "-oPubkeyAcceptedAlgorithms=+ssh-rsa" to your command line. I'm not sure why it didn't try one of the stronger RSA SHA256/512 variants. >debug1: Offering public key: /Users/steve/.ssh/vault RSA SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit >debug3: send packet: type 50 >debug2: we sent a publickey packet, wait for reply >debug3: receive packet: type 51 >debug1: Authentications that can continue: publickey This key is not in the server's authorized_keys.
Tne problem is here: > debug1: Offering public key: /Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit > debug1: send_pubkey_test: no mutual signature algorithm What does `ssh -G username@10.0.23.5 | grep ^pubkeyacceptedalgorithms` report?
``` pubkeyacceptedalgorithms ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa ``` > Tne problem is here: > > > debug1: Offering public key: /Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit > > debug1: send_pubkey_test: no mutual signature algorithm > > What does `ssh -G username@10.0.23.5 | grep > ^pubkeyacceptedalgorithms` report?
(In reply to Darren Tucker from comment #1) > Comment on attachment 3627 [details] > verbose output of ssh attempt > > The handling of -i hasn't changed as far as I know. > > [...] > >debug1: identity file /Users/steve/.config/zaccess/penguin.randomhostname.com.cert type 4 > >debug1: identity file /Users/steve/.config/zaccess/penguin.randomhostname.com.cert-cert type -1 > >debug1: identity file /Users/steve/.ssh/vault type 0 > >debug1: identity file /Users/steve/.ssh/vault-cert type -1 > > This doesn't exactly match the example invocation, but it indicates > that two keys were loaded. > > [...] > >debug1: Offering public key: /Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit > >debug1: send_pubkey_test: no mutual signature algorithm > > I think this is your problem: ssh-rsa was disabled by default in 8.8 > (https://www.openssh.com/releasenotes.html#8.8). You can test this > by adding "-oPubkeyAcceptedAlgorithms=+ssh-rsa" to your command > line. I'm not sure why it didn't try one of the stronger RSA > SHA256/512 variants. > > >debug1: Offering public key: /Users/steve/.ssh/vault RSA SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit > >debug3: send packet: type 50 > >debug2: we sent a publickey packet, wait for reply > >debug3: receive packet: type 51 > >debug1: Authentications that can continue: publickey > > This key is not in the server's authorized_keys. TBC the example command is from the vault docs. the command we are running is: ssh -i /Users/steve/.config/zaccess/penguin.randomhostname.com.cert -i /Users/steve/.ssh/vault ubuntu@penguin.randomhostname.com the pub key for the CA is in `TrustedUserCAKeys` in `/etc/ssh/sshd_config`. we sign a cert using `/Users/steve/.ssh/vault.pub` and then ssh with the cert and it's private key. with -oPubkeyAcceptedAlgorithms=+ssh-rsa also fails.
> with -oPubkeyAcceptedAlgorithms=+ssh-rsa also fails. oh, it's a cert not a plain key. Try CASignatureAlgorithms=+ssh-rsa instead. (same goes for Damien's suggestion: what does "ssh -G username@10.0.23.5 | grep ^casignaturealgoritms" output?)
I think the problem is that OpenSSH 7.6 is marked as having SSH_BUG_SIGTYPE: https://github.com/openssh/openssh-portable/blob/f84b9cffd52c9c5c359a54a1929f9948e803ab1d/compat.c#L74 and so falls in to this special-case: https://github.com/openssh/openssh-portable/blob/f84b9cffd52c9c5c359a54a1929f9948e803ab1d/sshconnect2.c#L1200 That was introduced in https://github.com/openssh/openssh-portable/commit/4ba0d54794814ec0de1ec87987d0c3b89379b436 I don't remember exactly which bug(s) SSH_BUG_SIGTYPE was added to cover, but I do note that it was subsequently split into SSH_BUG_SIGTYPE and SSH_BUG_SIGTYPE74. Maybe 7.6 could be made to work with some more compat glue?
Here's an excerpt from an review email on that commit: > This re-adds the agent signature type checking fallback, but also moves > forward with the foreshadowed stricter signature type checking. sshd > will check that the type of a pubkey signature is consistent with the > key type advertised in the USERAUTH_REQUEST packet. > > Because OpenSSH >= 7.6 were lax with this in some bases (legacy agents > being one), I've put this stricter checking behind a compat flag and > cranked the version number. > > This also switches the various key algorithm knobs > (PubkeyAcceptedKeyTypes, etc.) to check the pktype as is present in the > userauth packet and not the type of the deserialised key blob as they > currently do. The effect of this is to allow users to ban ssh-rsa if > they want. More users have been asking to be able to do this to prevent > the use of SHA-1 but still use RSA keys (via the rsa-sha2-* sigtypes). > > This tweak to semantics has some possibility of breaking existing > configurations, e.g. if they had "PubkeyAcceptedKeyTypes=ssh-rsa" in > sshd_config. It might be able to fix this risk with some creative > filtering of proposal strings. So the bug that is being avoided (badly as far as certs are concerned) is that <=7.6 would use an incorrect "public key algorithm name" field in the SSH2_USERAUTH_REQUEST message (https://www.rfc-editor.org/rfc/rfc4252#section-7) - it should actually be the _signature_ algorithm being used and not the type of the public key (this is arguably an error in the RFC) Making this work for back-compat would require arranging for sshconnect2.c:key_sig_algorithm() to return two strings: the actual signing algorithm identifier as it does currently plus the algorithm identifier to use in when the SSH2_AUTH_REQUEST is assembled in sign_and_send_pubkey(). However. it's highly likely that additional compat problems would remain. IIRC ssh-agent had problems wrt RSA/SHA2 signatures around then too. None of this helps you right now unfortunately. If you need something sooner than someone getting motivated to fix this (which, honestly, may never happen) than you options are 1) update the server to a version >=7.8 or change to a different key type - ECDSA or ED25519 are fine choices.
(In reply to Darren Tucker from comment #5) > > with -oPubkeyAcceptedAlgorithms=+ssh-rsa also fails. > > oh, it's a cert not a plain key. Try CASignatureAlgorithms=+ssh-rsa > instead. (same goes for Damien's suggestion: what does "ssh -G > username@10.0.23.5 | grep ^casignaturealgoritms" output?) hey sorry for the late response. `-oCASignatureAlgorithms=+ssh-rsa` doesnt work either, and `grep ^casignaturealgoritms` returns nothing.
(In reply to Damien Miller from comment #7) > Here's an excerpt from an review email on that commit: > > > This re-adds the agent signature type checking fallback, but also moves > > forward with the foreshadowed stricter signature type checking. sshd > > will check that the type of a pubkey signature is consistent with the > > key type advertised in the USERAUTH_REQUEST packet. > > > > Because OpenSSH >= 7.6 were lax with this in some bases (legacy agents > > being one), I've put this stricter checking behind a compat flag and > > cranked the version number. > > > > This also switches the various key algorithm knobs > > (PubkeyAcceptedKeyTypes, etc.) to check the pktype as is present in the > > userauth packet and not the type of the deserialised key blob as they > > currently do. The effect of this is to allow users to ban ssh-rsa if > > they want. More users have been asking to be able to do this to prevent > > the use of SHA-1 but still use RSA keys (via the rsa-sha2-* sigtypes). > > > > This tweak to semantics has some possibility of breaking existing > > configurations, e.g. if they had "PubkeyAcceptedKeyTypes=ssh-rsa" in > > sshd_config. It might be able to fix this risk with some creative > > filtering of proposal strings. > > So the bug that is being avoided (badly as far as certs are > concerned) is that <=7.6 would use an incorrect "public key > algorithm name" field in the SSH2_USERAUTH_REQUEST message > (https://www.rfc-editor.org/rfc/rfc4252#section-7) - it should > actually be the _signature_ algorithm being used and not the type of > the public key (this is arguably an error in the RFC) > > Making this work for back-compat would require arranging for > sshconnect2.c:key_sig_algorithm() to return two strings: the actual > signing algorithm identifier as it does currently plus the algorithm > identifier to use in when the SSH2_AUTH_REQUEST is assembled in > sign_and_send_pubkey(). > > However. it's highly likely that additional compat problems would > remain. IIRC ssh-agent had problems wrt RSA/SHA2 signatures around > then too. > > None of this helps you right now unfortunately. If you need > something sooner than someone getting motivated to fix this (which, > honestly, may never happen) than you options are 1) update the > server to a version >=7.8 or change to a different key type - ECDSA > or ED25519 are fine choices. we confirmed we can ssh to a server running `OpenSSH_8.2p1` thanks.