linux mint 18.2 $ uname -mvr 4.15.0-47-generic #50~16.04.1-Ubuntu SMP Fri Mar 15 16:06:21 UTC 2019 x86_64 (but all the same on the other versions of the kernel, i'd used for simplicity the latest stable one) $ openssl version LibreSSL 2.9.1 i'd tried three variants of starting sshd: with systemd (just copied proper (and edited in accordance with my system) files from openssh-server-7.2p2.deb /etc & /lib), from rc.local ("/usr/local/sbin/sshd -4 -f /etc/ssh/sshd_config &") and from command line with sudo - results are the same (and it does not depend was sshd killed after start manually or not). reboot goes almost to the end, as i may see, all partitions are unmounted before the last screen. this "screen of the death" is not always the same, sometimes before register's values there are the last lines with the list of the network modules of the kernel. if sshd is not started, reboot proceed seamlessly. occasional observation: if sshd starts as service via systemd, it could not be managed neither from systemd-manager-pkexec, nor from command line (i.e kill & pkill does not work in this case). to be totally sure, i'd restored tonight a backup from two months ago, installed libressl 2.9.1, ldns 1.7, opensshd 8.0p1 and got the same sad result. donno how to save core dump to file.
i'd extracted binary sshd 7.2p2 from the deb, put it in the /opt and during the start of it got an error message about the absence of /var/run/sshd. after required mkdir sshd 7.2p2 started, worked and reboot fine, moreover, 8.0p1 began permit reboot without core dump. ;-) i'd installed 8.0p1 via checkinstall, maybe it would make sense to add the creation of /var/run/sshd to installation's script? ;-)
Created attachment 3263 [details] photo of kernel panic
i'd closed bug too early: 8.0p1 goes to the same kernel panic, if it was started from rc.local and was not killed manually before reboot. clear reboot was only once - when i'd started and killed it from the command line (check of successful start|kill is making only with netstat -tunaple) ...guess that the same mistake will occur with the start via systemd... any ideas/suggestions?
As a general rule, if a userspace program like sshd can cause a kernel panic that indicates a problem with the kernel. Anyway, I do have a proposed experiment for you: run the stock sshd *and* your new sshd on a different port (eg /usr/local/sbin/sshd -d -p 222) then reboot and see if it panics. I'm wondering if systemd is somehow getting out of sync and waiting for sshd to stop even though it already has and the kernel eventually shoots pid 1. BTW: > ("/usr/local/sbin/sshd -4 -f /etc/ssh/sshd_config &") The & is not needed, sshd will daemonize itself after startup.
as to the kernel, 4.15.0-47 is the latest stock's one (i.e. from linux mint repository) now, but the problem does not depend from the kernel version, it occurs on the old 4.4 and on experimental 5.0.5 kernels as well. excuse me, i didn't catch your idea - you suggest to run both sshd simultaneously or separately, one by one? and i didn't mention that i'm using 443 port in all my experiments: $ cat /etc/ssh/sshd_config|grep Port Port 443 #GatewayPorts no i'd totally cleared systemd related staff, and right now both sshd reboots clear if they were started from command line (after "mkdir /var/run/sshd" in case of 7.2p2). but 8.0p1 causes panic if it was started from rc.local. may i ask for a proper/clear examples of systemd related files for 8.0p1?
(In reply to admzzz from comment #5) [...] > excuse me, i didn't catch your idea - you suggest to run both sshd > simultaneously or separately, one by one? 1) Install the stock sshd without running the new sshd and make sure it reboots OK. 2) Start new locally built sshd in addition to the stock sshd and reboot. 3) Replace the stock sshd with the locally built one and reboot. Basically we're trying to tell whether the difference is the absence of the stock sshd rather than the presence of the locally built one. > and i didn't mention that i'm using 443 port in all my experiments: that shouldn't matter as long as the 2nd sshd uses a different port. > may i ask for a proper/clear examples of systemd related files for > 8.0p1? Sorry, I don't know a great deal about systemd configs.
your points 1) and 2) works fine, 3) caused panic: 8.0p1 does not reboot gracefully, if it was started from the systemd's 7.2p2 environment. though if both sshd was started simultaneously from command line, reboot went clearly. so, the problem is in the systemd's 7.2p2 files only. i'd deleted all systemd's files except ssh.service and ssh.socket in /lib/systemd/system, and reboot works fine at last. thank you for your help and answers.
close bugs that were resolved in OpenSSH 8.5 release cycle