Bug 3233 - Error loading key ".ssh/id_rsa": error in libcrypto
Summary: Error loading key ".ssh/id_rsa": error in libcrypto
Status: CLOSED FIXED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: ssh-add (show other bugs)
Version: 8.4p1
Hardware: ix86 Linux
: P5 normal
Assignee: Assigned to nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-11-20 01:27 AEDT by Tomasz Paweł Gajc
Modified: 2022-02-25 13:59 AEDT (History)
3 users (show)

See Also:


Attachments
print libcrypto errors in ssh-add (629 bytes, patch)
2020-11-23 09:56 AEDT, Damien Miller
no flags Details | Diff
more debugging (2.00 KB, patch)
2021-02-04 09:23 AEDT, Damien Miller
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tomasz Paweł Gajc 2020-11-20 01:27:03 AEDT
Hi,

i have an issue with latest openssh-8.4p1.
My RSA ssh key which is quite old and can't be handled with ssh tools.

[tpg@tpg-virtualbox .ssh]$ ssh-add
Error loading key "/home/tpg/.ssh/id_rsa": error in libcrypto

At first i thought culprit may be OpenSSL3 so i filled issue upstream https://github.com/openssl/openssl/issues/13443, during the discussion it looks like it may be on OpenSSH side. I was trying to track some potential commits but did not found anything relevant.

Ofcourse ssh-keygen -t rsa works an with that new key ssh-add also works.

My system information:
Distribution: OpenMandriva Cooker
kernel-5.9.8
glibc-2.32
libxcrypt-4.4.17
LLVM-11.x
By default in this distro everything is compiled with LLVM/clang and linke with LLVM/lld

[tpg@tpg-virtualbox .ssh]$ rpm -qa | grep openssl
openssl-3.0.0-0.alpha8.1.x86_64
[tpg@tpg-virtualbox .ssh]$ rpm -qa | grep openssh
openssh-8.4p1-2.x86_64

RPM Sources:
https://github.com/OpenMandrivaAssociation/openssh
https://github.com/OpenMandrivaAssociation/openssl
Comment 1 Damien Miller 2020-11-20 14:38:42 AEDT
From the github issue it looks like your key is in the OpenSSH format.

If you have an older OpenSSH around, then could you try converting your key to the OpenSSL PEM format and see if that works?

cp id_rsa id_rsa_pem
ssh-keygen -p -m PEM -N '' -f id_rsa_pem

You'll also be able to check the key using OpenSSL's tools, e.g. "openssl rsa -check -noout -in id_rsa_pem"
Comment 2 Tomasz Paweł Gajc 2020-11-20 20:28:25 AEDT
(In reply to Damien Miller from comment #1)
> From the github issue it looks like your key is in the OpenSSH
> format.
> 
> If you have an older OpenSSH around, then could you try converting
> your key to the OpenSSL PEM format and see if that works?
> 

Yes, my ssh key works with older openssh/openssl


[live@localhost .ssh]$ ssh-keygen -p -m PEM -N ''  -f id_rsa_pem 
Enter old passphrase: 
Your identification has been saved with the new passphrase.
[live@localhost .ssh]$ openssl rsa -check -noout -in id
id_rsa      id_rsa_pem  id_rsa.pub  
[live@localhost .ssh]$ openssl rsa -check -noout -in id_rsa_pem 
RSA key ok

[live@localhost .ssh]$ rpm -qa | grep openssl
openssl-1.1.1c-1.x86_64
lib64qca2-plugin-openssl-2.1.4-2.x86_64
[live@localhost .ssh]$ rpm -qa | grep openssh
openssh-8.1p1-1.x86_64
openssh-server-8.1p1-1.x86_64
openssh-askpass-common-8.1p1-1.x86_64
openssh-clients-8.1p1-1.x86_64
Comment 3 Damien Miller 2020-11-23 09:56:37 AEDT
Created attachment 3457 [details]
print libcrypto errors in ssh-add

okay, can you load the PEM key on the openssl-3.x host?

Also, you could try building OpenSSH from git with the attached patch applied to see why openssl-3.x is refusing the key.
Comment 4 Tomasz Paweł Gajc 2021-02-04 07:04:21 AEDT
Sorry for long delay in my response.

I've applied your patch, recompiled opnssh, installed it and i executed ssh-add on my old key, and here is the output:

[root@tpg-virtualbox home]# ssh-add -vvv
Error loading key "/root/.ssh/id_rsa": error in libcrypt
Comment 5 Damien Miller 2021-02-04 09:23:03 AEDT
Created attachment 3469 [details]
more debugging

(In reply to Tomasz Paweł Gajc from comment #4)
> Sorry for long delay in my response.

No problem - thanks for helping debug this problem!

> I've applied your patch, recompiled opnssh, installed it and i
> executed ssh-add on my old key, and here is the output:
> 
> [root@tpg-virtualbox home]# ssh-add -vvv
> Error loading key "/root/.ssh/id_rsa": error in libcrypt

I don't understand how the patch has failed to print more information. I have attached a patch that should show a little more - can you please try that?
Comment 6 Tomasz Paweł Gajc 2021-02-05 22:58:15 AEDT
Hi, i did apply your second patch (https://github.com/OpenMandrivaAssociation/openssh/blob/errors/openssh-8.4p1-catch-openssl-errors.patch), here is the output:

[root@tpg-virtualbox home]# ssh-add -vvv
306: Error loading key "/root/.ssh/id_rsa": error in libcrypto


Here are the build logs
https://abf.openmandriva.org/build_lists/981796
Comment 7 Tomasz Paweł Gajc 2021-02-06 00:16:35 AEDT
Output of a strace ssh-add -vvv :

newfstatat(AT_FDCWD, "/foo.ssh/id_rsa", {st_mode=S_IFREG|0600, st_size=3326, ...}, 0) = 0
openat(AT_FDCWD, "/foo/.ssh/id_rsa", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...}, AT_EMPTY_PATH) = 0
getuid()                                = 0
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...}, AT_EMPTY_PATH) = 0
read(4, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 3326
read(4, "", 770)                        = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
futex(0x7f29f67cf20c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde50, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde58, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "306: Error loading key \"/foo/.s"..., 63306: Error loading key "/foo/.ssh/id_rsa": error in libcrypto
) = 63
Comment 8 Tomasz Paweł Gajc 2021-02-06 00:43:15 AEDT
I am more and more convinced this is openssl3 internal issue with reading keys.
Comment 9 Tomasz Paweł Gajc 2021-06-02 08:17:26 AEST
Issue has been fixed with openssl-3.0.0-alpha16
Comment 10 Damien Miller 2022-02-25 13:59:57 AEDT
closing bugs resolved before openssh-8.9