Created attachment 1899 [details]
test program that shows what kind of ttys are accepted

I downloaded openssh 5.5p1 for Solaris 8 x86 from Sunfreeware at ftp://ftp.sunfreeware.com/pub/freeware/intel/8/openssh-5.5p1-sol8-x86-local.gz .

If I set the UsePAM setting to yes, then no user is able to login. When the server is running with debugging, the client shows:
$ ssh testuser@10.100.3.175 
Password: 
Last login: Fri Jul 16 19:17:14 2010 from kyle-vmserver.c
Sun Microsystems Inc.   SunOS 5.8       Generic February 2000
debug3: PAM session not opened, exiting
Connection to 10.100.3.175 closed.
$ 

The error on the server is:
debug3: PAM: opening session
PAM: pam_open_session(): Can not make/remove entry for session

I have determined the problem is that pam_unix.so.1 does not like the value of PAM_TTY. I see this in the source code:
#ifdef PAM_TTY_KLUDGE
	/*
	 * Some silly PAM modules (e.g. pam_time) require a TTY to operate.
	 * sshd doesn't set the tty until too late in the auth process and
	 * may not even set one (for tty-less connections)
	 */
	debug("PAM: setting PAM_TTY to \"ssh\"");
	sshpam_err = pam_set_item(sshpam_handle, PAM_TTY, "ssh");
	if (sshpam_err != PAM_SUCCESS) {
		pam_end(sshpam_handle, sshpam_err);
		sshpam_handle = NULL;
		return (-1);
	}
#endif


I can tell that Sunfreeware did define PAM_TTY_KLUDGE, because I see this in the server's debug output:
debug1: PAM: setting PAM_TTY to "ssh"

I wrote a test program to show that Solaris 8 does not like this value (attached). Here is the output from the program:
bash-2.03# gcc -Wall -Werror -lpam try_tty.c -o try_tty
bash-2.03# ./try_tty  
Trying without PAM_TTY set
opensession: Can not make/remove entry for session
Trying with PAM_TTY set to 'ssh'
opensession: Can not make/remove entry for session
Trying with PAM_TTY set to '/dev/ssh'
opensession: Success
bash-2.03# ls -l /dev/ssh
/dev/ssh: No such file or directory
bash-2.03# cat /etc/release 
                       Solaris 8 2/02 s28x_u7wos_08a INTEL
           Copyright 2002 Sun Microsystems, Inc.  All Rights Reserved.
                           Assembled 18 December 2001