After upgraded to openssh 5.9p1, we cannot ssh to this node from other, and find that port22 didn't bind ipv6 interface, after restart sshd it works well. We found this issue is resolved in official redhat version4.3p2, and in the comments#18, it reused the source code of OpenSSH. https://bugzilla.redhat.com/show_bug.cgi?id=640857 Here's the secure log: grep error secure* secure:May 7 15:07:25 wc3il01pdanic1v1 sshd[27451]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use. It's our ssh version: ]# rpm -qa|grep openssh openssh-5.9p1-1 openssh-clients-5.9p1-1 openssh-server-5.9p1-1 In /var/log/secure we found the following statement: May 7 18:02:00 wc3il01pddnic2v1 sshd[6124]: Received signal 15; terminating. May 7 18:02:00 wc3il01pddnic2v1 sshd[3919]: error: Bind to port 22 on :: failed: Address already in use. May 7 18:02:01 wc3il01pddnic2v1 sshd[3919]: Server listening on 0.0.0.0 port 22.
What do you get when you run sshd from openssh-5.9p1 in debug mode (ie "/path/to/your/sshd -ddd -p 2022")? You won't even need to connect to it. On fedora, I get: debug2: load_server_config: filename /usr/local/etc/sshd_config debug2: load_server_config: done config len = 435 debug2: parse_server_config: config /usr/local/etc/sshd_config len 435 debug3: /usr/local/etc/sshd_config:13 setting Port 22 debug3: /usr/local/etc/sshd_config:58 setting ChallengeResponseAuthentication yes debug3: /usr/local/etc/sshd_config:60 setting Ciphers aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour debug3: cipher ok: aes128-ctr [aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour] debug3: cipher ok: aes256-ctr [aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour] debug3: cipher ok: arcfour256 [aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour] debug3: cipher ok: arcfour128 [aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour] debug3: cipher ok: arcfour [aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour] debug3: ciphers ok: [aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour] debug3: /usr/local/etc/sshd_config:76 setting AllowTcpForwarding yes debug3: /usr/local/etc/sshd_config:77 setting GatewayPorts clientspecified debug3: /usr/local/etc/sshd_config:78 setting X11Forwarding yes debug3: /usr/local/etc/sshd_config:79 setting XAuthLocation /usr/bin/xauth debug3: /usr/local/etc/sshd_config:81 setting X11UseLocalhost yes debug3: /usr/local/etc/sshd_config:82 setting PrintMotd no debug3: /usr/local/etc/sshd_config:89 setting ClientAliveInterval 300 debug3: /usr/local/etc/sshd_config:90 setting ClientAliveCountMax 3 debug3: /usr/local/etc/sshd_config:100 setting Subsystem sftp /usr/local/libexec/sftp-server -f LOCAL7 -l INFO debug1: sshd version OpenSSH_5.9p1 debug3: Incorrect RSA1 identifier debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug3: Incorrect RSA1 identifier debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: rexec_argv[0]='/home/dtucker/openssh/portable/openssh-5.9p1/sshd' debug1: rexec_argv[1]='-ddd' debug1: rexec_argv[2]='-p' debug1: rexec_argv[3]='2022' debug3: oom_adjust_setup Set /proc/self/oom_score_adj from 0 to -1000 debug2: fd 3 setting O_NONBLOCK debug1: Bind to port 2022 on 0.0.0.0. Server listening on 0.0.0.0 port 2022. debug2: fd 4 setting O_NONBLOCK debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY debug1: Bind to port 2022 on ::. Server listening on :: port 2022. which indicates it's working as expected here. Are you sure you don't still have the old sshd or something else listening on port 22? immediately before starting the sshd, do you get any "LISTEN" states from "lsof -i :22" ?
Hi, Here's what we got, while currently the node's ssh function works well, is that log helpful? Or do I need to reproduce it? debug2: load_server_config: filename /etc/ssh/sshd_config debug2: load_server_config: done config len = 808 debug2: parse_server_config: config /etc/ssh/sshd_config len 808 debug3: /etc/ssh/sshd_config:14 setting Protocol 2 debug3: /etc/ssh/sshd_config:32 setting SyslogFacility AUTHPRIV debug3: /etc/ssh/sshd_config:33 setting LogLevel INFO debug3: /etc/ssh/sshd_config:38 setting PermitRootLogin no debug3: /etc/ssh/sshd_config:39 setting StrictModes yes debug3: /etc/ssh/sshd_config:40 setting MaxAuthTries 6 debug3: /etc/ssh/sshd_config:43 setting PubkeyAuthentication yes debug3: /etc/ssh/sshd_config:47 setting RhostsRSAAuthentication no debug3: /etc/ssh/sshd_config:49 setting HostbasedAuthentication no debug3: /etc/ssh/sshd_config:54 setting IgnoreRhosts yes debug3: /etc/ssh/sshd_config:58 setting PermitEmptyPasswords no debug3: /etc/ssh/sshd_config:59 setting PasswordAuthentication yes debug3: /etc/ssh/sshd_config:63 setting ChallengeResponseAuthentication no debug3: /etc/ssh/sshd_config:73 setting GSSAPIAuthentication yes debug3: /etc/ssh/sshd_config:75 setting GSSAPICleanupCredentials yes debug3: /etc/ssh/sshd_config:86 setting UsePAM yes debug3: /etc/ssh/sshd_config:89 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES debug3: /etc/ssh/sshd_config:90 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT debug3: /etc/ssh/sshd_config:91 setting AcceptEnv LC_IDENTIFICATION LC_ALL debug3: /etc/ssh/sshd_config:93 setting GatewayPorts no debug3: /etc/ssh/sshd_config:94 setting X11Forwarding no debug3: /etc/ssh/sshd_config:97 setting PrintMotd yes debug3: /etc/ssh/sshd_config:98 setting PrintLastLog no debug3: /etc/ssh/sshd_config:104 setting ClientAliveInterval 900 debug3: /etc/ssh/sshd_config:105 setting ClientAliveCountMax 0 debug3: /etc/ssh/sshd_config:114 setting Banner /etc/ssh/sshd_banner_pcm debug3: /etc/ssh/sshd_config:117 setting Subsystem sftp /usr/libexec/openssh/sftp-server debug1: sshd version OpenSSH_5.9p1 debug3: Incorrect RSA1 identifier debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug3: Incorrect RSA1 identifier debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: rexec_argv[0]='/usr/sbin/sshd' debug1: rexec_argv[1]='-ddd' debug1: rexec_argv[2]='-p' debug1: rexec_argv[3]='2022' debug3: oom_adjust_setup Set /proc/self/oom_adj from 0 to -17 debug2: fd 3 setting O_NONBLOCK debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY debug1: Bind to port 2022 on ::. Server listening on :: port 2022. debug2: fd 4 setting O_NONBLOCK debug1: Bind to port 2022 on 0.0.0.0. Server listening on 0.0.0.0 port 2022. Regards, Carol
The log does not show any problems binding to IPv6 or IPv4, so the problem is either a) you have something else listening on port 22 other than the newly-build sshd, or the log messages in your syslog are not from the newly-built sshd (possibly the original vendor-supplied one?)
As I asked before: "Are you sure you don't still have the old sshd or something else listening on port 22? immediately before starting the sshd, do you get any "LISTEN" states from "lsof -i :22" ?"
Hi, 1. How do I know whether there're other sshd or something else listten ning on port 22? 2. About the suggestion "starting the sshd, do you get any "LISTEN" states from "lsof -i :22" ?"", would you please clarify how shall I do to check the states? If you provide detailed steps would be appreciated. Regards, Carol
Hi, Any update? Regards, Carol
did you try running the lsof command I asked for earlier (twice)?
(In reply to comment #7) > did you try running the lsof command I asked for earlier (twice)? Sorry, I didn't quit understand your earlier request. Let me clarify it: Shall I do like this: 1. stop sshd 2. run 'lsof -i :22' and send you the result. Please let me know whether this is the correct procedure you need. Regards, Carol
lsof is "list open files". The "-i" means "show me what's listening on Internet sockets" (see the lsof man page for mor info). In this case, we're interested in what's listening on port 22 other than the sshd you just installed. So, yes, stop the sshd you just install, then run "lsof -i :22" (you'll probably need to run this as root). Depending on who or what is connected to ssh on the machine you may get a bunch of output. but the only lines we're interested in will have "LISTEN" in them and look something like this: $ sudo lsof -i :22 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 2044 root 3u IPv4 21174 0t0 TCP *:ssh (LISTEN) sshd 2044 root 4u IPv6 21176 0t0 TCP *:ssh (LISTEN) In this example, we have one sshd (pid 2044) listening on IPv4 and IPv6 sockets. I suspect what you'll see is one line with sshd listening even after you stop the new sshd you just installed. In your case, you can just run "sudo lsof -i :22 | grep LISTEN" once you've stopped sshd and that will show you whether or not there's something else listening on port 22, and if so what the name of the binary is.
Just found a server which had bind error before. And stop sshd then run the lsof command, there's no listen in them, but there're some "ESTABLISHED" in them. Actually I cannot reproduce this issue, is it helpful? Or would it be helpful to run lsof when it's reproduced? Regards, Carol
Hi, If it's hard to analyze this issue, I will keep an eye on it and when it's reproduced, I'll try to run the "lsof -i :22" command before start the sshd service. Is there anything I can do about it? Regards, Carol
Actually, I don't think there is any issue. The debug output from your system shows that it's happily binding correctly, which means that the log messages you observed are either due to another process binding to port 22, or that the log messages are in fact from the *old* version of sshd. If you do see this again, check that the message is really from the new sshd and that you don't have another process listening on port 22 (using the lsof command I gave earlier).
Thanks for your help. Then I'll keep an eye on it. When it's reproduced, I will use this command "lsof -i :22" to check whether there's another process bind on it. Thanks, Carol
Set all RESOLVED bugs to CLOSED with release of OpenSSH 7.1