When building OpenSSH 4.1p1 against OpenSSL 0.9.7g, 'make tests' indicates that the arcfour cipher fails in regress/try-ciphers.sh. All other ciphers are successful. The ssh client can successfully use arcfour when talking to previous builds of OpenSSH compiled against OpenSSL 0.9.7e. However, when connecting to the newly built sshd with either the newly built ssh or the previous (OpwnSSH 3.9p1) build, the connection is broken immediately after authentication with a "Bad packet length" message. Note that OpenSSL 0.9.7f introduced assembler code for the RC4 algorithm on the IA64 architecture. Building OpenSSL with the 'no_asm' flag eliminates the issue, but also kills performance. Running OpenSSL's 'make test' shows no problem with the RC4 algorithm. Further, comparing the output from 'make test' for version 0.9.7g against 0.9.7e shows no difference in regards to RC4. A slightly more elegant workaround for this issue is to set the RC4_CHAR flage _and_ edit crypto/rc4/asm/rc4-ia64.S; changing SZ=4 to SZ=1. Since this problem seems to only affect sshd and only occurs after authentication, I'm wondering if there could be an issue with the key exchange that appears to occur after authentication.
Have you filed a bug with the OpenSSL developers. If so, could you please provide a link or tracking number?
The issue has also been filed with openssl.org as bug #1114.
Correction: The two workarounds mentioned in the initial bug report were erroneous. I had temporarily removed the reference to arcfour in try-ciphers.sh in order to see if there were any issues with any of the other regression tests. Apparently, I forgot to clean up after myself. I started over from scratch and tried the 'no-asm' workaround and the RC4_CHAR workaround. Both failed. Even using the no-asm flag and setting RC4_CHAR when building OpenSSL does not fix the issue.OB
Bug resolved by Andy Polyakov at openssl.org: Summary can be found at http://cvs.openssl.org/chngview?cn=14145. Point is that I assumed that RC4_KEY structure initialized by RC4_set_key is passed down to RC4 verbatim in its original memory location, while OpenSSH takes freedom to swap the structures initialized in different locations. One can argue that the latter is inappropriate design choice, but it works on too many other platforms to argue. And so IA64 was "reduced" to common denominator. Case dismissed. A.
Since this has been addressed in OpenSSL, closing this bug.
Change all RESOLVED bug to CLOSED with the exception of the ones fixed post-4.4.