Bug 3022

Summary: "dup() in/out/err failed" is output when stdin, stdout, or stderr is a TCP/IP socket
Product: Portable OpenSSH Reporter: minidino
Component: sshAssignee: Assigned to nobody <unassigned-bugs>
Status: CLOSED WORKSFORME    
Severity: normal CC: djm
Priority: P5    
Version: 8.0p1   
Hardware: ix86   
OS: Windows 10   

Description minidino 2019-06-26 02:08:58 AEST
Due to limitations in Windows, the CreateProcess() API accepts anonymous pipes for the stdin, stdout, and stderr handles.  However, buried in the MSDN docs is information that states that anonymous pipes cannot be made non-blocking.  This creates a situation where two processes can deadlock (e.g. one process is blocking on stdin while another process is blocking on stdout or stderr).

There are two known fixes for this on Windows:  Use multiple threads (one thread for each pipe) or use a little-known mode of CreateProcess() and pass connected TCP/IP socket handles for each of the pipes to the target process instead of anonymous pipes.  Not all languages support multithreading and using a thread per pipe is kind of wasteful of system resources.  Of course, establishing a TCP/IP connection per pipe is also wasteful but enables single-threaded, non-blocking code to work without deadlocking both processes.  Most processes work fine with TCP/IP stdout/stderr (TCP/IP stdin is a bit more iffy and depends on how the underlying C library works).

OpenSSH calls dup() in ssh.c and then bails out with the error message "dup() in/out/err failed" when it can't duplicate socket handle(s).  The connection then terminates.
Comment 1 Damien Miller 2019-09-05 22:11:00 AEST
Is this issue with the Cygwin support in portable OpenSSH or the Microsoft OpenSSH port?
Comment 2 Damien Miller 2022-01-14 15:44:06 AEDT
closing for lack of followup
Comment 3 Damien Miller 2022-02-25 13:59:56 AEDT
closing bugs resolved before openssh-8.9