I built both a aix 5.2 32 bit version of the 6.0p1 release and a aix 6.1 64 bit version. The aix 6.1 64 bit version works correctly, but the 32 bit aix 5.2 version will not let putty connect to the aix server. While testing I also discovered that when I run this command: ssh-keygen -t dsa -f /usr/local/etc/ssh-host-dsa.key it just hangs and never builds the keys again the 64 bit aix 6.1 version does correctly create keys. sox:/home/apple/dean> ssh -V OpenSSH_6.0p1, OpenSSL 1.0.1b 26 Apr 2012 I usually build these using a script that creates a installation RPM. just to make sure that wasn't causing the problem, I did a manual build using configure and build with the same results on the 32 bit aix 5.2 box I don't see any errors during the build, but maybe I am not looking in the right places to see errors.
A couple of possibilities: 1) the key generation code in openssl was miscompiled. Does openssl s "make tests" pass? which compiler did you use. 2) from memory, AIX 5.2 has a /dev/random device. It's possible that it's blocking due to lack of entropy. I'd suggest stracing (or equivalent) ssh-keygen and seeing what it's doing. Another thing to try is to get openssl to generate a DSA key and see if it works ("openssl dsaparam -genkey 1024 -out /dev/null" and "openssl dsaparam -genkey 1024 -rand /dev/urandom -out /dev/null". I have a machine I can test on but it's not currently set up.
Also, by "can't connect" do you mean "sshd doesn't start and I get a tcp connection refused" or "I can make a tcp connection but the login never completes"? Another thing to try is ssh'ing out from the problem machine (with -vvv) and attaching the output here. If this also fails (which I suspect it will) it'll give an indication of where the problem lies.
the putty problem is what what caused me to start looking at the new openssh, I was able to do the usual ssh into the aix system just as normal, but when one of our engineers tried to use putty to get into the aix system it failed and timed out. What I noticed is that it never got to the point where it would make an entry for ssh in the windows registry. The compiles of both the 5.2 32 bit and the 6.1 64 bit have been working fine for quite a while. The last ones I made successfully were bach:/opt/freeware/src/packages/SOURCES/openssh-6.0p1> ssh -V OpenSSH_5.9p1, OpenSSL 1.0.0h 12 Mar 2012 the compiler that I use is xlc_r bach:/opt/freeware/src/packages/SOURCES/openssh-6.0p1> lslpp -La |grep -i xlc xlC.adt.include 9.0.0.0 C F C Set ++ Application xlC.aix50.rte 9.0.0.0 C F XL C/C++ Runtime for AIX 5.2 xlC.cpp 5.0.2.0 C F C for AIX Preprocessor xlC.rte 9.0.0.0 C F XL C/C++ Runtime the make tests gets to the ssh-keygen section and hangs next I re-installed the openssl 1.0.0h, recompiled the latest openssh. </opt/freeware/src/packages/SOURCES/openssh-6.0p1> ./ssh -V OpenSSH_6.0p1, OpenSSL 1.0.0h 12 Mar 2012 I ran the make tests and it completed with a "all tests passed". I am now running the make test on the openssl and when it gets to the Generating a 2048 bit RSA private key it seems to go into a endless loop of some sort. So I am thinking my problem is with the latest openssl not openssh. I compiled the same openssl code on our aix 6.1/64 system and it ran the make test resulting with a "ALL TESTS SUCCESSFUL". so I think you can close this call and i will talk to the openssl people.
one thing that's probably worth a try is building openssl without assembler optimizations ("./config no-asm"). anyway thanks for the update, closing bug.
one last note: building the openssl using the config no-asm did work. The openssl does pass the make test with no errors
Set all RESOLVED bugs to CLOSED with release of OpenSSH 7.1