When I connect to an OpenSSH server using PuTTY with X11 forwarding, and run a makefile I use which invokes a large number of X applications in rapid succession, I find PuTTY terminates the session with a message along the lines of "Disconnected: Received SSH2_MSG_CHANNEL_FAILURE for nonexistent channel 257" (the channel number may vary). I told PuTTY to produce a full session log, which I attach here. (Sorry it's a bit large.) You can indeed see that PuTTY receives the final SSH2_MSG_CHANNEL_FAILURE at a point when it has both sent and received SSH2_MSG_CHANNEL_CLOSE on that channel. RFC 4254 section 5.3 states that when a party has both sent and received SSH2_MSG_CHANNEL_CLOSE for a channel, that channel is considered closed for that party and it may reuse the channel number; PuTTY's indignation at receiving the CHANNEL_FAILURE at that time therefore seems reasonable to me. I think what must have happened is that OpenSSHD sent its CHANNEL_CLOSE before it received the CHANNEL_REQUEST and CHANNEL_CLOSE from PuTTY; so when _it_ received the CHANNEL_REQUEST, it had sent but not received CHANNEL_CLOSE, and hence it did not yet consider the channel closed and felt free to respond to the request. However, although that sounds reasonable, I think it is in fact wrong behaviour. I think a direct logical consequence of section 5.3 is that after sending CHANNEL_CLOSE on a channel a party MUST NOT send any subsequent responses to channel events - because it can't be sure the other party hasn't just sent its own CHANNEL_CLOSE, causing exactly this circumstance. Conversely, on receiving CHANNEL_CLOSE, a party must discard any outstanding channel requests for which it was expecting to receive responses, and assume those requests were not received by the server until after it sent its close. I can't think of any other interpretation of RFC 4254 devoid of race conditions. (The channel request type is "winadj@putty.projects.tartarus.org": a phony channel request designed to elicit a CHANNEL_FAILURE which PuTTY uses to tune its window adjustment policy. I don't think that detail is important except inasmuch as it has the want_reply flag set.) I report this bug against 5.1p1, because that's the version in Debian stable. However, I have reproduced the same behaviour with openssh-SNAP-20100914 downloaded by following the links from http://www.openssh.com/portable.html, so I don't think this is an already-fixed issue.
Created attachment 1925 [details] putty.log : SSH packet log of failing connection (redacted)
Sorry about that; my packet log was apparently too big for Bugzilla to accept. I've attached a redacted version, produced using the Perl one-liner perl -ne 'if (/^\S/) { splice @lines, 2, $#lines-3, " ...\n" if $#lines >= 5; print @lines; @lines = (); print; } else { push @lines, $_; }' and hopefully that still shows the sequence of events (from PuTTY's viewpoint) and all important numbers and details without the huge wodges of irrelevant channel data.
Created attachment 1926 [details] openssh-channel-failure.diff : proposed patch Here's a trivial patch which seems to work for me: after I apply this (against last month's snapshot, not 5.1p1) my complex makefile runs to completion without any issues at the SSH level. As I discussed in my comments earlier, it simply suppresses generation of SSH2_MSG_CHANNEL_{SUCCESS,FAILURE} for channels on which SSH2_MSG_CHANNEL_CLOSE has already been sent. (It lacks comments and rationale, though; it's merely a proof of concept.)
Thanks for the detailed report. I don't see anything in RFC 4254 section 5.3 indicating that a channel shouldn't send notifications while it is half-closed. Could you explain how you arrived at this interpretation?
Because if you send anything that arrives at the other side when _it_ thinks the channel is fully closed, that's definitely in violation of 5.3. So if you send stuff after you've sent CLOSE, then it _might_ cross in transit with the other side's CLOSE, in which case it would arrive at the other side when that side had already both sent and received CLOSE. I discussed this last night with a friend, and he pointed out that there is an alternative protocol fix which also works, but it's more clearly contradictory of explicit text in the RFC. Instead of ruling that requests received after we send CLOSE may not be responded to, we could instead rule that all requests are responded to and modify section 5.3 to state that a channel number may be reused after you have both sent and received CLOSE _and_ received replies to all outstanding channel requests. However, it's clear that the current situation leads to a problem, so _something_ needs fixing one way or the other. If you don't agree with my analysis, should we take this to ietf-secsh and see if we can get a consensus?
Retarget unclosed bugs from 5.7=>5.8
Retarget unresolved bugs/features to 6.0 release
Retarget unresolved bugs/features to 6.0 release (try again - bugzilla's "change several" isn't)
Retarget from 6.0 to 6.1
Retarget 6.0 => 6.1
I've now worked around this issue on the PuTTY side (at least on the current development trunk), by implementing a more robust handling of SSH2_MSG_CHANNEL_CLOSE: _we_ now do not send CHANNEL_CLOSE until we have seen replies to all our outstanding channel requests, and therefore the question of which order the server's close and the request replies appear in is moot.
Retarget uncompleted bugs from 6.1 => 6.2
Retarget bugs from 6.1 => 6.2
retarget to openssh-6.3
Retarget to openssh-6.4
Retarget 6.3 -> 6.4
Retarget incomplete bugs / feature requests to 6.6 release
Retarget to 6.7 release, since 6.6 was mostly bugfixing.
Remove from 6.6 tracking bug
Created attachment 2431 [details] improved fix handling both client and server This issue has come up again recently and is currently under discussion on ietf-ssh. The general consensus seems to be that RFC 4254 has an ambiguity in it and that we should issue a clarification saying that responses to any outstanding or future channel requests should be implied by CHANNEL_CLOSE, and hence no actual reply messages should be sent after sending CHANNEL_CLOSE. OpenSSH is the only implementation I know of which does not do this (and PuTTY's window size tuning mechanism does cause us to notice...), so could I request higher priority on fixing it, please? I've attached a patch that I think is right. (I notice that I've attached a patch before, but it only dealt with the server side; this does the same thing on the client side too. I've only been able to test the server-side fix, but the client-side fix matches it as far as I can see so it should be right.)
Patch applied - this will be in OpenSSH 6.7
Close all bugs left open from 6.6 and 6.7 releases.
[spam removed]