Bug 3233

Summary: Error loading key ".ssh/id_rsa": error in libcrypto
Product: Portable OpenSSH Reporter: Tomasz Paweł Gajc <tpgxyz>
Component: ssh-addAssignee: Assigned to nobody <unassigned-bugs>
Status: CLOSED FIXED    
Severity: normal CC: djm, dtucker, gl041188
Priority: P5    
Version: 8.4p1   
Hardware: ix86   
OS: Linux   
Attachments:
Description Flags
print libcrypto errors in ssh-add
none
more debugging none

Description Tomasz Paweł Gajc 2020-11-20 01:27:03 AEDT
Hi,

i have an issue with latest openssh-8.4p1.
My RSA ssh key which is quite old and can't be handled with ssh tools.

[tpg@tpg-virtualbox .ssh]$ ssh-add
Error loading key "/home/tpg/.ssh/id_rsa": error in libcrypto

At first i thought culprit may be OpenSSL3 so i filled issue upstream https://github.com/openssl/openssl/issues/13443, during the discussion it looks like it may be on OpenSSH side. I was trying to track some potential commits but did not found anything relevant.

Ofcourse ssh-keygen -t rsa works an with that new key ssh-add also works.

My system information:
Distribution: OpenMandriva Cooker
kernel-5.9.8
glibc-2.32
libxcrypt-4.4.17
LLVM-11.x
By default in this distro everything is compiled with LLVM/clang and linke with LLVM/lld

[tpg@tpg-virtualbox .ssh]$ rpm -qa | grep openssl
openssl-3.0.0-0.alpha8.1.x86_64
[tpg@tpg-virtualbox .ssh]$ rpm -qa | grep openssh
openssh-8.4p1-2.x86_64

RPM Sources:
https://github.com/OpenMandrivaAssociation/openssh
https://github.com/OpenMandrivaAssociation/openssl
Comment 1 Damien Miller 2020-11-20 14:38:42 AEDT
From the github issue it looks like your key is in the OpenSSH format.

If you have an older OpenSSH around, then could you try converting your key to the OpenSSL PEM format and see if that works?

cp id_rsa id_rsa_pem
ssh-keygen -p -m PEM -N '' -f id_rsa_pem

You'll also be able to check the key using OpenSSL's tools, e.g. "openssl rsa -check -noout -in id_rsa_pem"
Comment 2 Tomasz Paweł Gajc 2020-11-20 20:28:25 AEDT
(In reply to Damien Miller from comment #1)
> From the github issue it looks like your key is in the OpenSSH
> format.
> 
> If you have an older OpenSSH around, then could you try converting
> your key to the OpenSSL PEM format and see if that works?
> 

Yes, my ssh key works with older openssh/openssl


[live@localhost .ssh]$ ssh-keygen -p -m PEM -N ''  -f id_rsa_pem 
Enter old passphrase: 
Your identification has been saved with the new passphrase.
[live@localhost .ssh]$ openssl rsa -check -noout -in id
id_rsa      id_rsa_pem  id_rsa.pub  
[live@localhost .ssh]$ openssl rsa -check -noout -in id_rsa_pem 
RSA key ok

[live@localhost .ssh]$ rpm -qa | grep openssl
openssl-1.1.1c-1.x86_64
lib64qca2-plugin-openssl-2.1.4-2.x86_64
[live@localhost .ssh]$ rpm -qa | grep openssh
openssh-8.1p1-1.x86_64
openssh-server-8.1p1-1.x86_64
openssh-askpass-common-8.1p1-1.x86_64
openssh-clients-8.1p1-1.x86_64
Comment 3 Damien Miller 2020-11-23 09:56:37 AEDT
Created attachment 3457 [details]
print libcrypto errors in ssh-add

okay, can you load the PEM key on the openssl-3.x host?

Also, you could try building OpenSSH from git with the attached patch applied to see why openssl-3.x is refusing the key.
Comment 4 Tomasz Paweł Gajc 2021-02-04 07:04:21 AEDT
Sorry for long delay in my response.

I've applied your patch, recompiled opnssh, installed it and i executed ssh-add on my old key, and here is the output:

[root@tpg-virtualbox home]# ssh-add -vvv
Error loading key "/root/.ssh/id_rsa": error in libcrypt
Comment 5 Damien Miller 2021-02-04 09:23:03 AEDT
Created attachment 3469 [details]
more debugging

(In reply to Tomasz Paweł Gajc from comment #4)
> Sorry for long delay in my response.

No problem - thanks for helping debug this problem!

> I've applied your patch, recompiled opnssh, installed it and i
> executed ssh-add on my old key, and here is the output:
> 
> [root@tpg-virtualbox home]# ssh-add -vvv
> Error loading key "/root/.ssh/id_rsa": error in libcrypt

I don't understand how the patch has failed to print more information. I have attached a patch that should show a little more - can you please try that?
Comment 6 Tomasz Paweł Gajc 2021-02-05 22:58:15 AEDT
Hi, i did apply your second patch (https://github.com/OpenMandrivaAssociation/openssh/blob/errors/openssh-8.4p1-catch-openssl-errors.patch), here is the output:

[root@tpg-virtualbox home]# ssh-add -vvv
306: Error loading key "/root/.ssh/id_rsa": error in libcrypto


Here are the build logs
https://abf.openmandriva.org/build_lists/981796
Comment 7 Tomasz Paweł Gajc 2021-02-06 00:16:35 AEDT
Output of a strace ssh-add -vvv :

newfstatat(AT_FDCWD, "/foo.ssh/id_rsa", {st_mode=S_IFREG|0600, st_size=3326, ...}, 0) = 0
openat(AT_FDCWD, "/foo/.ssh/id_rsa", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...}, AT_EMPTY_PATH) = 0
getuid()                                = 0
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...}, AT_EMPTY_PATH) = 0
read(4, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 3326
read(4, "", 770)                        = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
futex(0x7f29f67cf20c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde50, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde58, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "306: Error loading key \"/foo/.s"..., 63306: Error loading key "/foo/.ssh/id_rsa": error in libcrypto
) = 63
Comment 8 Tomasz Paweł Gajc 2021-02-06 00:43:15 AEDT
I am more and more convinced this is openssl3 internal issue with reading keys.
Comment 9 Tomasz Paweł Gajc 2021-06-02 08:17:26 AEST
Issue has been fixed with openssl-3.0.0-alpha16
Comment 10 Damien Miller 2022-02-25 13:59:57 AEDT
closing bugs resolved before openssh-8.9