Bug 1651

Summary: Possible race condition using local port forwarding with short lived connections
Product: Portable OpenSSH Reporter: Michael Kreitzer <mrgrim>
Component: sshAssignee: Assigned to nobody <unassigned-bugs>
Status: CLOSED INVALID    
Severity: normal CC: djm
Priority: P2    
Version: 5.2p1   
Hardware: ix86   
OS: Linux   

Description Michael Kreitzer 2009-09-17 14:07:00 AEST 
This may be a regression of bug #1054, but I've discovered symptoms outside its scope as well.

A simple test case is:

ssh -vv -L 5000:localhost:5000 -- nc -l -w 30 -p 5000 \> test.txt
cat test.txt | netcat -c -w 2 localhost 5000

If test.txt is very small (e.g. a single line of text) the following happens:

debug1: Connection to port 5000 forwarding to localhost port 5000 requested.                                                                                                            
debug2: fd 6 setting TCP_NODELAY                                                                                                                                                        
debug2: fd 6 setting O_NONBLOCK                                                                                                                                                         
debug1: channel 3: new [direct-tcpip]                                                                                                                                                   
debug2: channel 3: open confirm rwindow 2097152 rmax 32768
debug2: channel 3: read<=0 rfd 6 len 0
debug2: channel 3: read failed
debug2: channel 3: close_read
debug2: channel 3: input open -> drain
debug2: channel 3: ibuf empty
debug2: channel 3: send eof
debug2: channel 3: input drain -> closed
debug2: channel 3: rcvd eof
debug2: channel 3: output open -> drain
debug2: channel 3: obuf empty
debug2: channel 3: close_write
debug2: channel 3: chan_shutdown_write: shutdown() failed for fd 6: Transport endpoint is not connected
debug2: channel 3: output drain -> closed
debug2: channel 3: rcvd close
debug2: channel 3: send close
debug2: channel 3: is dead
debug2: channel 3: garbage collecting
debug1: channel 3: free: direct-tcpip: listening port 5000 for localhost port 5000, connect from 127.0.0.1 port 50076, nchannels 4
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: close_write
debug2: channel 2: output drain -> closed
debug2: channel 2: rcvd close
debug2: channel 2: almost dead
debug2: channel 2: gc: notify user
debug2: channel 2: gc: user detached
debug2: channel 2: send close
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug1: channel 0: free: port listener, nchannels 2
debug1: channel 1: free: port listener, nchannels 1
Transferred: sent 2368, received 2264 bytes, in 10.1 seconds
Bytes per second: sent 234.4, received 224.1
debug1: Exit status 1

In this case the file is transferred in full, but the exit status indicates the error shown in the output.

However, if test.txt is moderate in size (my test file was 66,870 bytes, but I imagine to reproduce you'd have to fiddle around depending on your connection to get the timing right) you get what is mentioned in bug 1054 (sorry, I'm having trouble making it happen now that I'm writing this bug so I'm pasting -v output of a previous run instead of -vv):

debug1: Connection to port 5000 forwarding to localhost port 5000 requested.                                                                                                
debug1: channel 3: new [direct-tcpip]                                                                                                                                       
debug1: getpeername failed: Transport endpoint is not connected                                                                                                             
debug1: channel 3: free: direct-tcpip: listening port 5000 for localhost port 5000, connect from UNKNOWN port -1, nchannels 4                            
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0                                                                                                       
debug1: channel 2: free: client-session, nchannels 3                                                                                                                        
debug1: channel 0: free: port listener, nchannels 2                                                                                                                         
debug1: channel 1: free: port listener, nchannels 1                                                                                                                         
Transferred: sent 61856, received 2264 bytes, in 10.3 seconds                                                                                                               
Bytes per second: sent 5990.7, received 219.3                                                                                                                               
debug1: Exit status 1                                                                                                                                                       

Sometimes the 66k file causes the former error as well. In both instances the transfer does not complete (the remote file is smaller). Using netcat directly successfully completes the transfer.

If I attempt a larger file that takes several seconds to complete everything works as expected.
Comment 1 Damien Miller 2009-09-17 14:23:10 AEST 
That looks like a normal shutdown to me. Perhaps the exit status is coming from netcat? Are both ends Linux? You seem to be using different netcat/nc commands for the client and server.
Comment 2 Michael Kreitzer 2009-09-17 23:45:04 AEST 
Ah, yes the use of nc vs netcat might be confusing I did not catch that.

Last night I was able to do this successfully using netcat by itself, but now I'm getting the same problem sans ssh port forwarding. It appears this is a gnu netcat issue.

I apologize for wasting your time.
Comment 3 Damien Miller 2009-10-06 15:01:57 AEDT 
Mass move of RESOLVED bugs to CLOSED now that 5.3 is out.