Using Mandrake Linux 6.1, I decided to go to OpenSSH 3.0p1 instad of the insecure 1.2.27 that Mandrake had available. I downloaded the sources, configured, and built, but it would just give me "Failed password" log messages when attempting to log in. I tried with and without PAM support in a few different configurations, and pulled it up in GDB to figure out why password didn't work. Sure 'nuff, crypt() returned a different encrypted password string than what was in my /etc/shadow. I couldn't find any previous instances of this problem, except a thread on the openssh-unix-dev mailing list which ended with no resolution, which looked like the same problem. Well, to make a long story short, forcing md5-passwords in config fixed the problem, but it took me a Sunday afternoon to get there. I suggest a fix which would check the length of passwords in /etc/shadow and turn on md5 passwords when the entries are long enough, although there might be a better way to detect this that someone RedHat-studlier than me would know about.
we should be able to detect md5 password configurations at configure time without the user specifying --with-md5-passwords. i think it could also might be handled at runtime.
You can't go reading /etc/shadow to determine what format your passwords are in - it is quite possible (preferred even) that you build as a non-root user. The system's crypt() function is the place to implement MD5 password hashing. Unfortunately this is often overriden by libcrypto's DES-only function of the same name. I hear that future OpenSSL releases will remove this. IIRC Redhat patches OpenSSL to remove the function. BTW on Mandrake you should be using PAM anyway :)
Mass change of RESOLVED bugs to CLOSED