Bug 3063 - ssh connecting to IP address -- ssh_dispatch_run_fatal: unexpected internal error
Summary: ssh connecting to IP address -- ssh_dispatch_run_fatal: unexpected internal e...
Status: CLOSED FIXED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: ssh (show other bugs)
Version: -current
Hardware: ix86 Mac OS X
: P5 trivial
Assignee: Assigned to nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-09-05 00:18 AEST by Warren Lavallee
Modified: 2021-04-23 15:10 AEST (History)
2 users (show)

See Also:


Attachments
Config.log (893.39 KB, text/plain)
2019-09-05 00:18 AEST, Warren Lavallee
no flags Details
ssh -vvv log (4.52 KB, text/plain)
2019-09-05 13:50 AEST, Warren Lavallee
no flags Details
This may help, its my build script (490 bytes, text/plain)
2019-09-05 13:59 AEST, Warren Lavallee
no flags Details
ssh -vvv working (11.46 KB, text/plain)
2019-09-05 14:41 AEST, Warren Lavallee
no flags Details
more kex debugging (8.03 KB, patch)
2019-09-05 16:09 AEST, Damien Miller
dtucker: ok+
Details | Diff
ssh -vvv logs -- patched source code (12.56 KB, application/zip)
2019-09-05 19:21 AEST, Warren Lavallee
no flags Details
Output from configure script (25.94 KB, text/plain)
2019-09-05 19:50 AEST, Warren Lavallee
no flags Details
config.log (893.39 KB, text/plain)
2019-09-05 19:50 AEST, Warren Lavallee
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Warren Lavallee 2019-09-05 00:18:06 AEST
Created attachment 3317 [details]
Config.log

When connecting via the latest SSH from git, compiled on a Mac against libressl, connecting to an IP address with ssh commandline throws an error:

./ssh 192.168.61.230
ssh_dispatch_run_fatal: Connection to 192.168.61.230 port 22: unexpected internal error

But the FQDN works fine:
509 bash$ ./ssh ntp
Linux ntp 4.19.50+ #1234 Thu Jun 13 10:47:38 BST 2019 armv6l
....


sftp does work however --

510 bash$ ./sftp 192.168.61.230
*********************************************************************
This resource, including all related equipment, networks and network
devices, are provided for authorized use. Computer systems may be
monitored for all lawful purposes, including to ensure authorized
use, for management of the system, to facilitate protection against
unauthorized access and to verify security procedures and operational
procedures.  The monitoring on this system may include audits by
authorized personnel to test or verify the validity, security and
survivability of this system. During monitoring information may be
examined, recorded, copied and used for authorized purposes. All
information placed on or sent to this system may be subject to such
monitoring procedures.  Use of this computer system, authorized or
unauthorized, constitutes consent to this policy and the policies and
procedures set forth by System Owner.  Evidence of unauthorized use
collected during monitoring may be used for criminal prosecution
by System staff, legal counsel and law enforcement agencies.
*********************************************************************
Connected to 192.168.61.230.
sftp>

This bug has existed for several weeks.

511 bash$ ./ssh -V
OpenSSH_8.0p1, LibreSSL 3.0.0
Comment 1 Darren Tucker 2019-09-05 09:37:43 AEST
I have not been able to reproduce on:
Darwin osx-highsierra 17.7.0 Darwin Kernel Version 17.7.0: Sun Jun  2 20:31:42 PDT 2019; root:xnu-4570.71.46~1/RELEASE_X86_64 x86_64

What version of OS X are you using?  What configure flags did you give when you built libressl and openssh?  Could you please attach to this bug the debug output from ssh -vvv for both the working and non-working examples?
Comment 2 Darren Tucker 2019-09-05 09:53:29 AEST
From you config.log:
> Darwin Kernel Version 18.7.0:

presumably that's Mojave?

> ./configure --with-ssl-dir=/usr/local --with-pam

--with-pam doesn't change the behaviour of the client, so other than that it should be a default build.

It's possible the problem you're seeing is specific to a particular crypto method, which should show up in the comparison of the two debug logs.
Comment 3 Warren Lavallee 2019-09-05 13:49:55 AEST
(In reply to Darren Tucker from comment #1)
> I have not been able to reproduce on:
> Darwin osx-highsierra 17.7.0 Darwin Kernel Version 17.7.0: Sun Jun 
> 2 20:31:42 PDT 2019; root:xnu-4570.71.46~1/RELEASE_X86_64 x86_64
> 
> What version of OS X are you using?  What configure flags did you
> give when you built libressl and openssh?  Could you please attach
> to this bug the debug output from ssh -vvv for both the working and
> non-working examples?

Good evening & Hello, how are you tonight?

macOS Mojave 10.14.6 -- latest and greatest.

Configure line confirmed.

Attached -vvv log as requested.
I can try it without the pam to see if that is the cause.
Comment 4 Warren Lavallee 2019-09-05 13:50:34 AEST
Created attachment 3320 [details]
ssh -vvv log
Comment 5 Warren Lavallee 2019-09-05 13:52:11 AEST
(In reply to Darren Tucker from comment #2)
> It's possible the problem you're seeing is specific to a particular
> crypto method, which should show up in the comparison of the two
> debug logs.

Are you saying the crypto method changes between:

ssh foo.bar.com

and

ssh 1.1.1.1

invocation methods?  The first one works, the second one doesn't.

(I haven't coded in C for 20+ years, I am not an expert!)
Comment 6 Warren Lavallee 2019-09-05 13:53:40 AEST
(In reply to Warren Lavallee from comment #3)
> (In reply to Darren Tucker from comment #1)
> > I have not been able to reproduce on:
> > Darwin osx-highsierra 17.7.0 Darwin Kernel Version 17.7.0: Sun Jun 
> > 2 20:31:42 PDT 2019; root:xnu-4570.71.46~1/RELEASE_X86_64 x86_64
> > 
> > What version of OS X are you using?  What configure flags did you
> > give when you built libressl and openssh?  Could you please attach
> > to this bug the debug output from ssh -vvv for both the working and
> > non-working examples?
> 
> Good evening & Hello, how are you tonight?
> 
> macOS Mojave 10.14.6 -- latest and greatest.
> 
> Configure line confirmed.
> 
> Attached -vvv log as requested.
> I can try it without the pam to see if that is the cause.

I recompiled without the "with-pam" and it has the same behavior.
Comment 7 Warren Lavallee 2019-09-05 13:59:08 AEST
Created attachment 3321 [details]
This may help, its my build script

Don't know if this is all still necessary, but when I started building from git, the source wouldn't build, and I had to manually add some defines to config.sh before the build.

I think just:
#define HAVE_GETLINE 1
#define HAVE_BZERO 1

Not sure why configure didn't find them.

back to bed now !
Comment 8 Warren Lavallee 2019-09-05 14:11:22 AEST
If I download and build the latest release version -- "openssh-8.0p1" from your website, using the same build script, it works.

So that should remove everything on my system as a possible issue.
Comment 9 Darren Tucker 2019-09-05 14:16:03 AEST
(In reply to Warren Lavallee from comment #5) 
> Are you saying the crypto method changes between:
> "ssh foo.bar.com" and "ssh 1.1.1.1"
> invocation methods?

It's possible, eg if you have a Host entry for foo.bar.com in ~/ssh/config that sets things like Ciphers, MACs or HostKeyAlgorithms, but do not have the equivalent for 1.1.1.1.
Comment 10 Darren Tucker 2019-09-05 14:18:51 AEST
(In reply to Warren Lavallee from comment #4)
> Created attachment 3320 [details]
> ssh -vvv log

That's only the non-working one.  Please include the working one too, so that we can see what's different.
Comment 11 Warren Lavallee 2019-09-05 14:41:31 AEST
Created attachment 3322 [details]
ssh -vvv working
Comment 12 Warren Lavallee 2019-09-05 14:49:51 AEST
Don't look at that -- it's not the latest build -- it was the release version from your website.

I don't know what I did, but now neither method works --

00:45 warren@WJL-MBP-15:~/src/openssh-portable
536 bash$ ./ssh goliath.openfinance.com
ssh_dispatch_run_fatal: Connection to 192.168.61.230 port 22: unexpected internal error
00:45 warren@WJL-MBP-15:~/src/openssh-portable
537 bash$ ./ssh 192.168.61.230
ssh_dispatch_run_fatal: Connection to 192.168.61.230 port 22: unexpected internal error
00:46 warren@WJL-MBP-15:~/src/openssh-portable
538 bash$


(In reply to Warren Lavallee from comment #11)
> Created attachment 3322 [details]
> ssh -vvv working
Comment 13 Darren Tucker 2019-09-05 15:11:53 AEST
diffie-hellman-group-exchange-sha256 (which both sets of logs show) involve selecting a Diffie-Hellman group (ie a set of numbers) from the moduli file on the server.  It's possible that the behaviour varies depending on the specific one selected.

I suggest:
 - repeating each test a number of times to see if the behaviour changes
 - force use of the static group ("ssh -o kexalgorithms=diffie-hellman-group14-sha1 ...").
 - try replacing the moduli file on the server with the one from the current distribution (keep the old one, in case it's relevant :-)
Comment 14 Damien Miller 2019-09-05 16:09:38 AEST
Created attachment 3323 [details]
more kex debugging

This puts a bunch of more-proximal error messages in the key exchange code. You'll need to apply it to git HEAD and recompile, but it may give a better indication of where this is blowing up.
Comment 15 Warren Lavallee 2019-09-05 19:21:46 AEST
Created attachment 3324 [details]
ssh -vvv logs -- patched source code

I applied your patch, recompiled, and ran ssh -vvv 9 times against the same host.  The logfiles are in the ZIP file.
Comment 16 Warren Lavallee 2019-09-05 19:24:19 AEST
(In reply to Darren Tucker from comment #13)
> diffie-hellman-group-exchange-sha256 (which both sets of logs show)
> involve selecting a Diffie-Hellman group (ie a set of numbers) from
> the moduli file on the server.  It's possible that the behaviour
> varies depending on the specific one selected.
> 
> I suggest:
>  - repeating each test a number of times to see if the behaviour
> changes
>  - force use of the static group ("ssh -o
> kexalgorithms=diffie-hellman-group14-sha1 ...").
>  - try replacing the moduli file on the server with the one from the
> current distribution (keep the old one, in case it's relevant :-)

ssh -o kexalgorithms=diffie-hellman-group14-sha1
works

Perhaps its a compatibility issue with LibreSSL 3.0.0
The production ssh release does work with LibreSSL 3.0.0 however.
I compile LibreSSL 3.0.0 separately and install into /usr/local -- that is the reason for the with-SSL I configure.

macOS seems to come with LibreSSL 2.7.3 out of the box.
Comment 17 Damien Miller 2019-09-05 19:32:17 AEST
Here's the error:

choose_mac: unsupported MAC hmac-sha2-512

Could you please attach the output of configure and config.log (separate attachments please)
Comment 18 Warren Lavallee 2019-09-05 19:50:06 AEST
Created attachment 3325 [details]
Output from configure script
Comment 19 Warren Lavallee 2019-09-05 19:50:48 AEST
Created attachment 3326 [details]
config.log
Comment 20 Damien Miller 2019-09-05 20:27:30 AEST
The problem is that you're not regenerating configure/config.h when you are updating the source and rebuilding. Your build script needs a call to autoreconf between the git pull and configure invocations.
Comment 21 Warren Lavallee 2019-09-05 20:42:01 AEST
(embarrassed) that’s it!  I did an autoreconf and it works now!  I realize you work for free. Sorry for waiting so much of your valuable time with a non-issue.  Check your email.
Comment 22 Damien Miller 2021-04-23 15:10:44 AEST
closing resolved bugs as of 8.6p1 release