Starting with version 6.2p1, ssh client connections to Nortel/Avaya ERS 5600 series switches fail. Connections with 6.1p1 and earlier do not exhibit this problem. This is observed with the following switch models and release versions: * Avaya Ethernet Routing switch (ERS) 5698 and 5698-PoE * OS version 6.2.1.003 with boot firmware 6.0.0.10 * OS version 6.3.0.013 with boot firmware 6.0.0.15 * http://www.avaya.com/usa/product/ethernet-routing-switch-5000-series These switches actually run Mocana SSH server software, so it's possible that other embedded devices using Mocana SSH are also affected. It is unclear which version of Mocana these switches are running, or if the different OS/FW firmware versions have different versions of Mocana. Mocana SSH is described at: * https://www.mocana.com/for-device-manufacturers/nanossh/ Affected OpenSSH versions observed: * 6.2p1, 6.2p2, 6.2-SNAP-20130604 Unaffected OpenSSH versions observed: * 5.9p1, 6.0p1, 6.1p1 Client operating systems tested: * Linux r3239 3.2.0-44-generic #69-Ubuntu SMP Thu May 16 18:27:54 UTC 2013 i686 i686 i386 GNU/Linux * CYGWIN_NT-6.1-WOW64 L3313 1.7.18(0.263/5/3) 2013-04-19 10:39 i686 Cygwin I first noticed this problem under Cygwin. However, I have verified it on Ubuntu by compiling versions 5.9p1, 6.0p1, 6.1p1, 6.2p1, and 6.2p2, and the daily snapshot from 20130604 from source with no special configuration options. Running ssh 6.2p1 and later with -v shows that the connection gets as far as expecting SSH2_MSG_KEXDH_REPLY, then the far end closes the connection. Versions 6.1p1 and earlier work normally. Running ssh with additional -v flags, as well as running it when compiled with the various DEBUG macros, does not yield any additional information that is meaningful to me. I will attached "ssh -vvv" output from 6.1p1 and 6.2p1 in the hope that it will be helpful. If you do not have Nortel/Avaya or other hardware running Mocana SSH at your disposal, I am willing to assist with testing of alternate configurations or patches. I've marked this as "major" because I've been unable to identify any workaround with the affected versions. Rolling back to 6.1p1 is the only fix I've found.
Created attachment 2293 [details] Output from "ssh -vvv" on version 6.1p1
Created attachment 2294 [details] Output from "ssh -vvv" on version 6.2p1
The only noticeable difference in the debug output is that openssh is offering the hmac-md5-etm@openssh.com MACs although the since the server doesn't support them they're not selected. Please try disabling this via "ssh -o MACs=hmac-md5 yourswitch". The debug trace looks like the server is crashing. The default value for this in 6.1 was "hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-sha1-96,hmac-md5-96" which is 107 bytes. The default value in 6.2 is "hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com, hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-sha1-96,hmac-md5-96" which is 377 bytes. It would not surprise me if the server is overrunning a fixed size buffer, I would guess 256 bytes. If this is the case, it's quite possibly exploitable. I'm marking this bug private until the vendor has had a chance to be notified.
Reducing the list of MACs offered fixes the problem, thanks! I've just added the affected switches to my ~/.ssh/config file with a shorter MACs line for now as a workaround. Given that this doesn't appear to be an OpenSSH bug at all, I leave it to you as to whether you want to keep it open. For what it's worth, all that my Avaya switches log when this happens is "#5 Failed login from IP address: 10.10.64.130". I've since learned that an update (6.3.1.039) for these switches is available. There is no mention of any SSH issues resolved in this release, but I'll test it on a spare unit anyway and report back any change in behavior.
depending on how prevalent this is, maybe we should add a compatibility fallback for these servers. I'd like to give the vendor a chance to look at this before doing anything further, including marking this bug public. Could you please let us know what they say?
I see the same behavior with the latest code for our Avaya switches. I believe I've narrowed down the behavior. There are two aspects to it: 1. The magic number seems to be 287. Any list of MACs exceeding 287 bytes results in the observed behavior. However... 2. This particular version of Mocana only supports hmac-md5, hmac-md5-96, hmac-sha1, and hmac-sha1-96. Any MAC negotiation not containing one of those will be rejected with a specific message from the server, regardless of length. I agree with Darren that there is probably a fixed-size buffer being overrun somewhere, and is resulting in undesirable behavior. It can be worked around by manually configuring the MACs option for known Mocana servers, but ultimately I think it's their bug to fix. I'll open a trouble ticket with Avaya for this, and hopefully they'll push it upstream to Mocana. I will followup with any details they provide.
Thanks. The MACs are part of the KEXINIT packet (see http://openssh.com/txt/rfc4253.txt section 7.1) so your magic value may vary depending on what values you have in other fields, eg Ciphers and HostKeyAlgorithms.
Did you ever get any kind of response from the vendor? The banner does not provide any kind of version information, so any compat hacks we do would have to be an all-or-nothing thing: debug1: Remote protocol version 2.0, remote software version Mocana SSH
There's nothing we can do short of blacklisting every device reporting that version string and I'm not willing to do that without info from the vendor on what proportion of devices that might be. Please reopen if that information ever becomes available.
Close all resolved bugs after 7.3p1 release