Bug 806 - openssh after 3.6.1p1 can not authenticate via public rsa2 key
Summary: openssh after 3.6.1p1 can not authenticate via public rsa2 key
Status: CLOSED WORKSFORME
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: ssh (show other bugs)
Version: 3.8p1
Hardware: HPPA HP-UX
: P2 major
Assignee: OpenSSH Bugzilla mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-03-04 02:25 AEDT by Peter Kielbasiewicz
Modified: 2006-10-07 11:36 AEST (History)
0 users

See Also:


Attachments
debug output from ssh 3.8p1 connections (5.59 KB, text/plain)
2004-03-04 02:37 AEDT, Peter Kielbasiewicz
no flags Details
config.h of openssh3.6.1p1 with buffer.adv patch (25.91 KB, text/plain)
2004-03-04 20:32 AEDT, Peter Kielbasiewicz
no flags Details
make log of openssh3.6.1p1 with buffer.adv patch (172.06 KB, text/plain)
2004-03-04 20:33 AEDT, Peter Kielbasiewicz
no flags Details
config.h of openssh3.8p1 (28.51 KB, text/plain)
2004-03-04 20:33 AEDT, Peter Kielbasiewicz
no flags Details
make log of openssh3.8p1 (178.00 KB, text/plain)
2004-03-04 20:36 AEDT, Peter Kielbasiewicz
no flags Details
openssh compile and build options (3.62 KB, text/plain)
2004-03-04 20:41 AEDT, Peter Kielbasiewicz
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Kielbasiewicz 2004-03-04 02:25:51 AEDT
My compilation of openssh 3.7.1p2 and 3.8p1 can not authenticate from HPUX 10.20
via rsa2 or dsa public key.
The sshd daemon side works OK. I can use public key authentication from Linux or
Windows TO HPUX without problems.
I am using rsa2 keys and the keys as well as the access rights of my directory
structure are ok. I can connect to the sshd on HPUX from other platforms with my
key pair using public key authentication without problems.
When I try ssh  FROM  HPUX  to other hosts or even to myself sshd always asks
for a password.
It seems that the ssh client skips the public key authentication step as can be
seen from the debug output below.
The openssh version 3.6.1p1 does not show the described effect, i.e. I can
connect from HP-UX using my rsa2 public key authentication without problems.

As HP-UX does not support PAM I did not use the with-pam flag for compilation.
The compile flags were the same for all revisions and as follows:
              CFLAGS="+O3 +ESlit +Optrs_strongly_typed
-I$SRC/tcp_wrappers/$TCP_WRAPver" \
              LDFLAGS="-L$SRC/tcp_wrappers/$TCP_WRAPver" \
              ./configure --prefix=/opt/$VER \
                          --sysconfdir=/etc/opt/openssh \
                          --with-default-path="/usr/bin:/usr/sbin:/opt/$VER/bin" \
                          --with-ssl-dir=$SRC/openssl/$OPENSSLver \
                          --with-zlib=$SRC/zlib/$ZLIBver \
                          --with-prngd-socket=/var/run/egd-pool \
                          --with-tcp-wrappers \
                          --without-shadow \
                          --disable-suid-ssh

I compiled against
    TCP_WRAPver=tcp_wrappers_7.6-ipv6.3
    OPENSSLver=openssl-0.9.7c
    ZLIBver=zlib-1.2.1
    PRNGDver=prngd-0.9.27

Parts from debug output:
    debug1: identity file /home/peterk/.ssh/identity type -1
    debug2: key_type_from_name: unknown key type '-----BEGIN'
    debug2: key_type_from_name: unknown key type '-----END'
    debug1: identity file /home/peterk/.ssh/id_rsa type 1
    debug1: identity file /home/peterk/.ssh/id_dsa type -1
    debug1: Remote protocol version 1.99, remote software version OpenSSH_3.8p1
   ...
    debug1: Authentications that can continue:
publickey,password,keyboard-interactive
    debug1: Next authentication method: publickey
    debug2: we did not send a packet, disable method
    debug1: Next authentication method: keyboard-interactive
    debug2: userauth_kbdint
    debug2: we sent a keyboard-interactive packet, wait for reply
    debug1: Authentications that can continue:
publickey,password,keyboard-interactive
    debug2: we did not send a packet, disable method
Comment 1 Peter Kielbasiewicz 2004-03-04 02:37:27 AEDT
Created attachment 559 [details]
debug output  from ssh 3.8p1 connections
Comment 2 Darren Tucker 2004-03-04 12:38:18 AEDT
I just tried 3.7.1p2 on my 11.00 box and it worked ok:
debug1: Offering public key: /home/dtucker/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
[...]

Peter, could you please create attachments of config.h after running configure,
for both 3.6.1p2 and 3.8p1?
Comment 3 Peter Kielbasiewicz 2004-03-04 20:32:18 AEDT
Created attachment 561 [details]
config.h of openssh3.6.1p1 with buffer.adv patch
Comment 4 Peter Kielbasiewicz 2004-03-04 20:33:13 AEDT
Created attachment 562 [details]
make log of openssh3.6.1p1 with buffer.adv patch
Comment 5 Peter Kielbasiewicz 2004-03-04 20:33:48 AEDT
Created attachment 563 [details]
config.h of openssh3.8p1
Comment 6 Peter Kielbasiewicz 2004-03-04 20:36:05 AEDT
Created attachment 564 [details]
make log of openssh3.8p1
Comment 7 Peter Kielbasiewicz 2004-03-04 20:41:33 AEDT
Created attachment 565 [details]
openssh compile and build options
Comment 8 Darren Tucker 2004-03-30 13:25:36 AEST
There have been reports[1][2] of OpenSSL builds not working properly on HP-UX. 
Does "make tests" in the openssl directory pass all of its tests?  It looks like
you are using the HP compiler?

See if openssl can read the keys itself:
openssl rsa -check -noout <$HOME/.ssh/id_rsa
openssl rsa -modulus -noout <$HOME/.ssh/id_rsa

[1] http://marc.theaimsgroup.com/?l=openssh-unix-dev&m=108012097630716
[2] http://marc.theaimsgroup.com/?l=openssh-unix-dev&m=107969091332191
Comment 9 Barrie Spence 2004-05-11 20:08:58 AEST
I get exactly the same problems on 10.20, 11.00 and 11i with OpenSSH-3.8.1p1 
and OpenSSL-0.9.7d. 

It appears to be due to the use the HP C compiler with "+Optrs_strongly_typed" 
to build OpenSSH.

Compiler versions used:

10.20: fileset B.10.20.09; what strings A.10.32.30
11.00: fileset B.11.11.04 + PHSS_26792; what strings B.11.11.26792.GP 
111:   fileset B.11.11.06; what strings B.11.11.06

OpenSSL was also built with the HP compiler with stock options from "Configure 
hpux-parisc-cc".

Comment 10 Damien Miller 2005-04-21 15:56:45 AEST
Does removing +Optrs_strongly_typed from your CFLAGS help?
Comment 11 Peter Kielbasiewicz 2005-04-21 20:47:02 AEST
I finally compiled it with gcc 3.0.1 which was the latest version for HPUX10.20
which I could find.
openssh now works almost perfect with the exception that the sshd daemon does
not accept Ctrl-C interrupts, when it is started at boot time.
If I invoke sshd from a terminal it inherits the intr key setting from the
terminal and everything is OK.
At boot time there is no tty though and I did not succeed to get the daemon to
accept ^C.
Comment 12 Darren Tucker 2005-04-21 21:04:20 AEST
Doesn't it use the default HP-UX interrupt char (ctrl-backspace)?  An easy
workaround would be to put something like this into your .profile:

if [ ! -z `tty` ]; then
        stty intr ^C
fi

Where ^C is produced by typing CTRL-V CTRL-C.
Comment 13 Damien Miller 2005-06-21 13:07:42 AEST
Compiler problems, not a bug in OpenSSH itself.
Comment 14 Darren Tucker 2006-10-07 11:36:10 AEST
Change all RESOLVED bug to CLOSED with the exception of the ones fixed post-4.4.