Bug 3490

I already reported this in bugzilla.redhat.com (https://bugzilla.redhat.com/show_bug.cgi?id=2127970), but  Dmitry Belyavskiy told me to raise the issue here as well. 
For completeness sake: I'm on CentOS 9 Stream, fully updated.

I'll copy my bug report from the other bugzilla:

Description of problem:
Keys explicitly provided by "-i /path/to/keyfile" is ignored when -J <jumphost> is on the command line

Version-Release number of selected component (if applicable):

openssh-8.7p1-22.el9.x86_64

How reproducible:
always

Steps to Reproduce:
1. ssh -i /path/to/ssh-key -J user@jumphost user@targethost

Actual results:
ssk-key is ignored, login fails

Expected results:
ssh-key should be used, login succeeds

Additional info:
Detailed description and observations. 

In the process of setting up a jumphost I noticed what I would classify as a bug in ssh.

I set up a jumphost which knows userA, which then passes the connection though to userA@targethost. The jumphost does not allow password-login of userA, just key-login. A special key was generated for this connection chain, which is stored in /path/to/ssh-key. The public key was put into the authorized_keys-file of userA on both jumphost and targethost.

I can login into the jumphost by

ssh -i /path/to/ssh-key userA@jumphost

I can also directly login to the targethost:

ssh -i /path/to/ssh-key userA@jumphost

This will be disallowed later, hence the requirement for a jumphost. The jumphost will also disallow direct logins of users later; direct login is only allowed for debugging right now. Now for the weird part:

ssh -i /path/to/ssh-key -J userA@jumphost userA@targethost

fails with 

userA@jumphost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

checking with "ssh -v", the key provided by the -i option is not even read (keys replaced by #### for privacy):

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:####  agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA SHA256:#### agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
d
(...)

If I place /path/to/ssh-key into the .ssh-folder of localuser, the key -is- offered automatically (no -i option this time), and login to targethost works

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:####  agent
debug1: Will attempt key: user ECDSA SHA256:#### agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA SHA256:#### agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: user ECDSA SHA256:#### agent
debug1: Server accepts key: user ECDSA SHA256:#### agent
debug1: Offering public key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
d
(...)

If I remove the key from localuser's .ssh-folder again, but add it to ssh-agent: (ssh-add /path/to/ssh-key), login again works (no -i option):

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:####  agent
debug1: Will attempt key: user ECDSA SHA256:#### agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA SHA256:#### agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /path/to/ssh-key ECDSA SHA256:#### agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### agent
(...)

At one point I had forgotten to remove the -i from the command line after adding the key to ssh-agent, and the last to lines changed to:

(...)
debug1: Offering public key: /path/to/ssh-kedebug1: Offering public key: /path/to/ssh-key ECDSA SHA256:#### agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### agent
y ECDSA SHA256:#### explicit agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### explicit agent
(...)

So it seems that the -i option -is- evaluated, but not used in all cases. Very strange, and IMHO inconsistent behaviour indeed.