Bug 2655 - AuthorizedKeysCommand with large output can deadlock
Summary: AuthorizedKeysCommand with large output can deadlock
Status: CLOSED FIXED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: sshd (show other bugs)
Version: 7.2p2
Hardware: All Linux
: P5 normal
Assignee: Damien Miller
URL:
Keywords:
Depends on:
Blocks: V_7_5
  Show dependency treegraph
 
Reported: 2016-12-30 17:42 AEDT by Josiah Boning
Modified: 2018-04-06 12:26 AEST (History)
3 users (show)

See Also:


Attachments
consume entire output of keys/principals on success (1.47 KB, patch)
2017-01-27 12:04 AEDT, Damien Miller
dtucker: ok+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Josiah Boning 2016-12-30 17:42:56 AEDT
If an AuthorizedKeysCommand produces a large amount of output, a deadlock can result. The relevant code is in auth2-pubkey.c, beginning at line 1041:

	if ((pid = subprocess("AuthorizedKeysCommand", pw, command,
	    ac, av, &f)) == 0)
		goto out;

	uid_swapped = 1;
	temporarily_use_uid(pw);

	ok = check_authkeys_file(f, options.authorized_keys_command, key, pw);

	if (exited_cleanly(pid, "AuthorizedKeysCommand", command) != 0)
		goto out;

Upon finding the correct public key in the command's output, we immediately wait() for the command to exit. However, the command may continue to generate output; if the subsequent output is large enough to fill up the pipe's buffer, the command will block on write() and never exit, resulting in deadlock.

I believe adding "fclose(f); f = NULL;" after the check_authkeys_file() call will fix this. (There was indeed an fclose() after the check_authkeys_file() call prior to v1.50 of auth2-pubkey.c)
Comment 1 Damien Miller 2016-12-31 09:09:55 AEDT
Nice catch. The same problem exists with AuthorizedPrincipalsCommand too.

I've committed your fix for both cases.
Comment 2 Josiah Boning 2016-12-31 11:16:27 AEDT
Great, thank you!
Comment 3 Damien Miller 2017-01-27 12:00:54 AEDT
The fix that was committed causes a new problem: helper commands may now receive SIGPIPE when their output is closed early.
Comment 4 Damien Miller 2017-01-27 12:04:12 AEDT
Created attachment 2931 [details]
consume entire output of keys/principals on success

This eats the remaining output of the keys/principals file when we successfully match.
Comment 5 Damien Miller 2017-01-30 12:03:25 AEDT
fix committed - thanks.
Comment 6 Jim Knoble 2017-01-31 05:27:07 AEDT
If an sshd process dies while reading the output from Authorized{Keys,Pricipals}Command, wouldn't it also die with SIGPIPE? Wouldn't it be more resilient to require the command to handle SIGPIPE appropriately ... or even to set up an appropriate handler before spawning the command?

Continuing to consume unneeded output seems like the wrong thing to do here.
Comment 7 Damien Miller 2017-01-31 10:00:00 AEDT
(In reply to Jim Knoble from comment #6)

> Continuing to consume unneeded output seems like the wrong thing to
> do here.

Why?
Comment 8 Damien Miller 2017-01-31 10:24:57 AEDT
BTW setting a SIGPIPE handler isn't possible as signal handlers are reset on exec
Comment 9 Jim Knoble 2017-01-31 17:17:22 AEDT
(In reply to Damien Miller from comment #7)
> (In reply to Jim Knoble from comment #6)
> 
> > Continuing to consume unneeded output seems like the wrong thing to
> > do here.
> 
> Why?

For large amounts of additional output, it introduces lag in the processing of authorized keys; which in turn could cause issues in scalability.

It also seems to "promise" that the output will always be read, whereas calling out up front that it may not all be read encourages the writers of AuthorizedKeysCommand's to write code that handles such failures resiliently. 


(In reply to Damien Miller from comment #8)
> BTW setting a SIGPIPE handler isn't possible as signal handlers are
> reset on exec

You're right, of course. I was thinking of *ignoring* SIGPIPE when I wrote this, which, although it would survive execve(2), is clearly not the right thing to do here. Chalk this one up to "it's been too long since I've done this". Perhaps I should stop armchair quarterbacking....
Comment 10 Damien Miller 2018-04-06 12:26:33 AEST
Close all resolved bugs after release of OpenSSH 7.7.