> PAM: pam_setcred(): Failure setting user credentials

So pam_setcred() is failing. This is provided by your PAM "auth" module and sshd has no visibility into what is going wrong there beyond the error code/message reported above. You should check you PAM auth stack to see what is causing it, or see if you can enable extra logging if it doesn't tell you why it is failing by default.

I have spent some time looking at this issue from the Linux PAM stack end. It looks like the problem is actually caused by openssh calling PAM from different threads/processes.

Here is the rough sequence of what happens, with the thread id inside {} at the start of each line:

{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(108)] called.
{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(176)] user=root, password=[]
{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(182)] recording return code for next time [7]
{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(182)] done. [Authentication failure]
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(108)] called.
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(176)] user=root, password=[1]
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(182)] recording return code for next time [0]
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_authenticate(182)] done. [Success]
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_acct.c:pam_sm_acct_mgmt(196)] called.
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_acct.c:pam_sm_acct_mgmt(202)] user = `root'
{12574}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_acct.c:pam_sm_acct_mgmt(307)] all done
{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_setcred(203)] called.
{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_setcred(209)] recovering return code from auth call
{12568}[../../../Linux-PAM-1.3.0/modules/pam_unix/pam_unix_auth.c:pam_sm_setcred(217)] recovered data indicates that old retval was 7

This translates to:

Thread 1 tries to authenticate with empty password
This fails and the failure is cached in thread 1 context
Thread 2 tries to authenticate again, this time with the password that the user supplied
This succeeds and the success is cached in thread 2 context
Thread 1 tries to set credentials, which uses the cached failure from thread 1

The call to pam_sm_setcred() should have come from the same thread that performed the successful call to pam_sm_authenticate()

It looks like thread 2 has been forked from thread 1, because when it's called, it knows about the previous auth failure and it correctly replaces the previous (failure) return code with the new (success) return code. However, the parent process can never see this updated result code and instead only sees the previous failure.

I have not had the chance to explore the openssh code yet, but it looks like the kind of thing that PrivSep would be involved in.

Yes, calling pam_setcred in different process than pam_authenticate is wrong. Even worse that pam_authenticate with empty password is called in the original process that later calls the pam_setcred.

So this is really a bug in openssh.

Adding to this bug, since it seems related (PAM set_cred error seems to match). Likely another set of steps to reproduce.

Observed on various platforms with various OpenSSH versions, with server configured with PasswordAuthentication=yes, UsePAM=yes, ChallengeResponseAuthentication=yes.

When the client fails password authentication, and progresses to keyboard-interactive (ChallengeResponse), there seems to be some tainted state in the PAM module that causes the server to abruptly drop the transport connection very soon after acknowledging the (successful) authentication.

With server configuration options as above (allowing PasswordAuthencation and keyboard-interactive), run "ssh localhost -o NumberOfPasswordPrompts=1 -o PreferredAuthentications=password,keyboard-interactive -v " to reproduce. An empty password on the first (password) attempt will not result in reproducing the error, but any non-blank incorrect password that causes the followup keyboard-interactive attempt (using correct password) triggers the failure:

---
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: password
paul@localhost's password: 
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: keyboard-interactive
Password: 
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to localhost ([::1]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
packet_write_wait: Connection to ::1 port 22: Broken pipe
---

Also note, reversing the client preferred order (failing the keyboard-interactive attempt, then enter the correct password on the password authentication attempt) does not result in abrupt disconnect. The scenario seems to strictly be a password failed followed by keyboard-interactive success.