The early call to setpcred() in do_setusercontext() seems to drop the euid to the user's uid on AIX5.1. This stops the future call to initgroups() from working if setpcred() doesn't get the supplementary group list right. Which it doesn't with PAM. The symptoms are a 'successful' login, but the session exits immediately, with sshd logging "initgroups: Permission denied". setpcred() must still be called at some stage to correctly set up the process rlimits and auditing class. I found that moving it to the end of do_setusercontext() works.
I just found a similar kind of setpcred problem fixed in http://archives.neohapsis.com/archives/aix/2002-q3/0003.html: | A call to initgroups failed after a call to | setpcred. Changed order of calls so initgroups | is called first.
I'll take a look at this. One thought off the top of my head: what if some module in the PAM stack causes pam_setcred to drop the privs setpcred needs?
someone here pointed out the aix system I was using is at maintennance level (oslevel -r) 5100-02. I've upgraded to 5100-07 and the problem has gone away!! It seems to be an AIX bug, so I'm marking this bug invalid.
Change all RESOLVED bug to CLOSED with the exception of the ones fixed post-4.4.