Hey. I found a "special" situation in which ssh connections crash every few tries and sometimes (but not always) one get's any of these along: [527879.021049] traps: sshd[14583] general protection ip:7fbc7f04a664 sp:7fff3939fe58 error:0 in libc-2.19.so[7fbc7efce000+19f000Hey. I found a special situation in which ssh connections crash every few tries and sometimes (but not always) one get's any of these along: [527879.021049] traps: sshd[14583] general protection ip:7fbc7f04a664 sp:7fff3939fe58 error:0 in libc-2.19.so[7fbc7efce000+19f000] [527945.727953] traps: sshd[14660] general protection ip:7f069558d664 sp:7fffc4223c88 error:0 in libc-2.19.so[7f0695511000+19f000] [528046.264330] traps: sshd[14826] general protection ip:7f1b26eed664 sp:7fff521d7178 error:0 in libc-2.19.so[7f1b26e71000+19f000] [536582.887955] traps: sshd[26078] general protection ip:7f96158b4664 sp:7fff2fef4a08 error:0 in libc-2.19.so[7f9615838000+19f000] [536628.489940] traps: sshd[26206] general protection ip:7f9cc14a9664 sp:7fffdacfb478 error:0 in libc-2.19.so[7f9cc142d000+19f000] [536734.550558] traps: sshd[26320] general protection ip:7f260fc18664 sp:7ffffb25be88 error:0 in libc-2.19.so[7f260fb9c000+19f000] [536841.887230] traps: sshd[26513] general protection ip:7f168b350664 sp:7fff8a85a2c8 error:0 in libc-2.19.so[7f168b2d4000+19f000] [536860.256030] traps: sshd[26572] general protection ip:7fba93937664 sp:7ffffcf18928 error:0 in libc-2.19.so[7fba938bb000+19f000] [536949.787928] sshd[27137]: segfault at 8100000038 ip 00007f84523e666 sp 00007fff2cc1d908 error 4 in libc-2.19.so[7f845236a000+19f000] [537088.405962] traps: sshd[27582] general protection ip:7f349cde6664 sp:7fffaf183ee8 error:0 in libc-2.19.so[7f349cd6a000+19f000] What I do is basically the following: Having sshd running (my sshd_config is attached), and gitolite3 (from sid) installed. Gitolite (which I use with the "git" username) in turn has entries like these: command="/usr/share/gitolite3/gitolite-shell admin",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 ... in its authorized_key files Then I repeatedly do: $ ssh git@myserver info Sometimes this works and I get: > hello someName, this is git@myserver running gitolite3 3.6.1-3 (Debian) on git 2.1.4 But more than every 2nd time it fails and I get > Write failed: Broken pipe Sometimes (not always) with a general protection or segfault. From my sshd_config, which uses a Match block for the git user (for reasons of hardening), I found that the > PermitOpen none line is the cause of the problem When I comment it, then the connections *always* succeed (well at least from about ~20 successive tries). ] [527945.727953] traps: sshd[14660] general protection ip:7f069558d664 sp:7fffc4223c88 error:0 in libc-2.19.so[7f0695511000+19f000] [528046.264330] traps: sshd[14826] general protection ip:7f1b26eed664 sp:7fff521d7178 error:0 in libc-2.19.so[7f1b26e71000+19f000] [536582.887955] traps: sshd[26078] general protection ip:7f96158b4664 sp:7fff2fef4a08 error:0 in libc-2.19.so[7f9615838000+19f000] [536628.489940] traps: sshd[26206] general protection ip:7f9cc14a9664 sp:7fffdacfb478 error:0 in libc-2.19.so[7f9cc142d000+19f000] [536734.550558] traps: sshd[26320] general protection ip:7f260fc18664 sp:7ffffb25be88 error:0 in libc-2.19.so[7f260fb9c000+19f000] [536841.887230] traps: sshd[26513] general protection ip:7f168b350664 sp:7fff8a85a2c8 error:0 in libc-2.19.so[7f168b2d4000+19f000] [536860.256030] traps: sshd[26572] general protection ip:7fba93937664 sp:7ffffcf18928 error:0 in libc-2.19.so[7fba938bb000+19f000] [536949.787928] sshd[27137]: segfault at 8100000038 ip 00007f84523e666 sp 00007fff2cc1d908 error 4 in libc-2.19.so[7f845236a000+19f000] [537088.405962] traps: sshd[27582] general protection ip:7f349cde6664 sp:7fffaf183ee8 error:0 in libc-2.19.so[7f349cd6a000+19f000] What I do is basically the following: Having sshd running (my sshd_config is attached), and gitolite3 (from sid) installed. Gitolite (which I use with the "git" username) in turn has entries like these: command="/usr/share/gitolite3/gitolite-shell admin",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 ... in its authorized_key files Then I repeatedly do: $ ssh git@myserver info Sometimes this works and I get: > hello someName, this is git@myserver running gitolite3 3.6.1-3 (Debian) on git 2.1.4 But more than every 2nd time it fails and I get > Write failed: Broken pipe Sometimes (not always) with a general protection or segfault. From my sshd_config, which uses a Match block for the git user (for reasons of hardening), I found that the > PermitOpen none line is the cause of the problem When I comment it, then the connections *always* succeed (well at least from about ~20 successive tries). I should probably further notice: systemd/logind/PAM is used (not sure if this could somehow interfere). Also, I'm a bit unsure whether the "main" sshd is crashing or whethr it's just the processes of the sessions. I didn't manually restart sshd, but it might be that systemd does that automatically? How would I find out? So some bug is hidden there... Cheers, Chris
Created attachment 2550 [details] sshd_config
Could you try running a sshd in debugging mode on a different port (i.e. "sshd -dddp 2222") and catching it in the act of crashing? Seeing where it fails would be a great help.
BTW, I can't replicate this with HEAD
Created attachment 2551 [details] ssh.log
Created attachment 2552 [details] sshd.log
(In reply to Damien Miller from comment #2) > Could you try running a sshd in debugging mode on a different port > (i.e. "sshd -dddp 2222") and catching it in the act of crashing? > Seeing where it fails would be a great help. Sure, see attached files: sshd and ssh output, from the later you see which tries failed (with which error) and which worked. Interestingly, the sshd quite *every time* after the end of the connection... is this because of -D? (In reply to Damien Miller from comment #3) > BTW, I can't replicate this with HEAD Mhh and have you tried with an older tag as well (i.e. 6.7p1?) and could replicate it there?
(In reply to Christoph Anton Mitterer from comment #6) > (In reply to Damien Miller from comment #2) > > Could you try running a sshd in debugging mode on a different port > > (i.e. "sshd -dddp 2222") and catching it in the act of crashing? > > Seeing where it fails would be a great help. > Sure, see attached files: sshd and ssh output, from the later you > see which tries failed (with which error) and which worked. > > Interestingly, the sshd quite *every time* after the end of the > connection... is this because of -D? -D and -d mean different things, but yes -d means "run in debug mode once then exit". If you want to leave it up you can use this instead of -d: /path/to/sshd -De -o LogLevel=debug3 > (In reply to Damien Miller from comment #3) > > BTW, I can't replicate this with HEAD > Mhh and have you tried with an older tag as well (i.e. 6.7p1?) and > could replicate it there? Note that you are running a vendor-modified version of OpenSSH: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-3 It's not all that easy for us to know exactly what each vendor has changed. Even things like which compiler flags they use can make a difference. Have you reported this to Debian? They're in a much better position to reproduce a problem.
Damien and I spent a couple of hours with a VM trying to figure this out and we now think we know what the cause is. I'll update this bug again once we're sure we are on the right track.
Hey. Sorry for not having answered earlier... this got somehow hidden under a huge pile of mails the last days =) Yes I did report it in Debian, see the URL set here in the bug report's "See Also section" (http://bugs.debian.org/778807) But nothing has happened there so far. When I reported this in the beginning, I had a short glance whether any of Debian's patches obviously touches something in this area,... nothing I'd have seen (OTOH I'm not an OpenSSH code expert). It's great to hear that you possibly found the issue :-) Thanks, Chris.
Created attachment 2617 [details] calloc permitted_adm_opens instead of malloc to ensure it's zeroed Here's the fix. I'll be commit this shortly.
The patch has been applied and will be in the 6.9 release. Thanks for the report!
Set all RESOLVED bugs to CLOSED with release of OpenSSH 7.1