I have the same issue as encountered in bug 1254. When launching thousands of SSH connections via a script (the open source taktuk/kanif) using ssh-agent to forward keys, occasionally I will see ssh-agent hang and consume 100% of one CPU. This does not happen every time, but around 1 out of every 3 runs. I have compiled 5.2p1 which also exhibits the same issue. strace at the time of the hang reports an EAGAIN error on a read call. A few printfs isolated the code in question to be the same as mentioned in bug 1254, but the suggested workaround (add a usleep before trying the read again) does not work in any case. This issue is also reported at http://www.plug.org/pipermail/plug/2009-April/033800.html +++ This bug was initially created as a clone of Bug #1254 +++ In function after_select(), case AUTH_CONNECTION, the do-loop which handles socket reads will peg my CPU at close to 100% when errno is EAGAIN. I'm running FreeBSD 6.2 pre-release, with OpenSSH built from the ports collection (security/openssh-portable). The problem only occurs for me while running an automation script that sends commands through ssh to about a hundred servers at at time, and I have not been successful in identifying which server causes the problem. But the bottom line is that the read fails with errno EAGAIN, and continues to fail in a very tight loop until a timeout occurs at some point. My work-around was to introduce a tiny sleep before the continue statement in that loop, which is apparently enough to allow some data to become available for reading, and makes the problem go away. I will attach my work-around as a patch, realizing that usleep() is probably not available on all platforms.
Created attachment 1670 [details] fall back to select() on read/write interruptions Could you try to reproduce the problem with this patch applied?
... and here is a theory on how it occurs: on a heavily loaded ssh-agent, we can create a new socket in the ssh-agent.c:after_select() loop, via the AUTH_SOCKET case calling new_socket(). This might increase sockets_alloc past the value it had when execution enters after_select(). The for() loop in after_select() can therefore progress into sockets that did not exist when select() and, critically, prepare_select() was called. prepare_select() sizes and clears the fd_sets that select() subsequently populates and after_select() tests. So a new AUTH_CONNECTION socket whose creation increments sockets_alloc can cause after_select to test past the end of the allocated fd_sets and might (depending on what it finds) treat them as ready for reading.
Created attachment 1671 [details] fix the root cause of the problem too
Patch applied to the ssh-agent.c in openssh-5.2p1 (RCS revision 1.159). I have now successfully run our scripts against 6000 hosts for the first time, so it appears to have solved the issue. I will be soak-testing over the next 48 hours and will update after that. (and thanks for the very quick response!)
Have you been able to reproduce the problem with patch #1671 applied?
(In reply to comment #5) > Have you been able to reproduce the problem with patch #1671 applied? We've not had any further problems with ssh-agent since applying #1671 - looks like it's fixed. Thanks!
patch applied. This will be in openssh-5.4.
Mass move of RESOLVED bugs to CLOSED now that 5.3 is out.
*** Bug 1135 has been marked as a duplicate of this bug. ***
With the release of 5.4p1, this bug is now considered closed.