Created attachment 2243 [details] Detailed Debug info. Problem Description - ====================== SSH/SSHD hang when SSHD is configured to allow only sftp connection requests from a particular user ( Match User setting in sshd_config ). Steps to reproduce - ===================== NOTE- These steps are for recreating the scenario only to a localhost( which acts as both Client & Server) and similar steps can be used to recreate it for a Client & Server ( different machines ). 1. Create a local user say "test" .Also,define a password for user "test" . 2. Set-up the password less( public key ) authentication for user "test" , i.e Copy /home/test/.ssh/id_rsa.pub to /home/test/.ssh/authorized_keys ( So, I don't need to type the test's password every time). 3. Edit the sshd_config file to include the Match User setting, so that user "test" can do only sftp to this server ( i.e localhost )- # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # ForceCommand cvs server Match User test ForceCommand internal-sftp 4. Restart SSHD . 5. Try to login using sftp (should be OK as expected): $ sftp test@localhost Connecting to localhost... test@localhost's password: sftp> 6. Try to ssh to the localhost (Here is the issue . It will hang.): SSH hangs after random number of repeated login requests ( see the infinite while loop used to send repeated login requests . Also ,Since Public key authentication has been configured for user "test" , it will not ask for its password ). (0) root @ ic15: 7.1.0.0: / # su test (0) test @ ic15: 7.1.0.0: / # while > true > do > ssh localhost > done This service allows sftp connections only. Connection to localhost closed. This service allows sftp connections only. Connection to localhost closed. This service allows sftp connections only. Connection to localhost closed. This service allows sftp connections only. Connection to localhost closed. This service allows sftp connections only. Connection to localhost closed. This service allows sftp connections only. Connection to localhost closed. <== It hangs here after a random number of login requests. The below SSHD processes are active at the same time - (0) root @ ic15: 7.1.0.0: / # ps -ef | grep sshd root 5111844 9502940 0 15:24:49 - 0:00 sshd: test [priv] test 9175114 5111844 0 15:24:50 - 0:00 sshd: test@pts/7 root 9502940 3604646 0 15:17:24 - 0:00 /usr/sbin/sshd I believe, there is some Communication problem between the monitor sshd process (5111844) and its child 9175114. (Some signal is being missed from either side ). Looks like, both Client (ssh) & Server (sshd) are waiting for each other and a deadlock kind of scenario is being created. Please check the attached .txt file for detailed debugging information.
the debug logs say 6.0p1 not 6.1p1. other than that, they're not all that useful. They just show the monitor sitting there waiting to be asked to do something (which is normal) and and on the slave side just shows that it's waiting for input. Which version (and ML) of AIX is this? Can you get the debug output from ssh and sshd and attach it to this bug? For the client it should be simple: $ while true; do ssh -vvv localhost; done for the server, you can try this assuming you have sudo available: $ while true; do ssh -o 'proxycommand sudo /usr/sbin/sshd -e -i -o loglevel=debug3' localhost; done it's possible that won't trigger the problem, in that case you'll need to set loglevel debug3 in your sshd_config and grep the debug logs out of syslog or something.
Actually I have tested with OpenSSH 5.8p1,6.0p1,6.1p1 & 6.2p1 and this problem is present in all these versions. The problem can be reproduced in AIX version 6.1 & 7.1 both with any of the above mentioned OpenSSH versions. I am attaching here the debug logs from both Client & Server. NOTE- One Debug log (Debug-Output-Without-Hang) is for successful scenario , i.e No SSH/SSHD doesn't hang with the configuration as mentioned in the recreation steps. Other Debug log(Debug-Output-With-Hang) is for the Hang scenario , i.e SSH/SSHD hangs with the configuration as mentioned in the recreation steps ( you need to try ssh login manually for random number of times to recreate it). One thing I noticed that for SSH/SSHD hang to happen - ssh login requests should be very frequent from Client side. If there is considerable amount of time gap between two consecutive ssh login requests , then chances for Hang is very rare.
Created attachment 2246 [details] Debug-Output-Without-Hang
Created attachment 2247 [details] Debug-Output-With-Hang
is this a stock version of openssh compiled from the source at openssh.com? I see entries in the logs which don't seem to belong ("debug3: ACCESS KEY", ". If so, can you reproduce the problem with a stock version? If so, please attach those logs.
Ok.. I will upload the logs from stock version very soon.
Created attachment 2343 [details] Debug-Output-with-Hang-NEW Attaching "Debug-Output-with-Hang-NEW.txt" which is the debug log from stock OpenSSH version 6.0p1.
Comment on attachment 2343 [details] Debug-Output-with-Hang-NEW >debug1: init_func_ptrs success that is not from the stock openssh available from openssh.com >debug1: Value of krb5 is 1 or that >debug1: Value of gssapi is 1 or that >From Server side - [...] >debug3: Value for authType is STD_AUTH or that >debug1: init_func_ptrs passed or that >debug1: value of krb5 is : 1 or that >debug1: value of gssapi is : 1 or that >debug1: Failed to collect Cookie from Keystore or that >debug1: Keystore Opening wil be failed after login or that >debug1: Cookie received : or that >debug1: Failed to collect Cookie from Keystore or that >debug1: Keystore Opening wil be failed after login or that. and it's not even spelt correctly. You are using a (heavily!) modified version of openssh. We have no idea what's in the software you're running. You need to seek assistance from whoever supplied this modified version of openssh. If you can download the tarball from openssh.com, compile, install and reproduce the problem then we may be able to help you. If you can do that, please reopen this bug and attach the output of ./configure, the config.h file and the debug output from the server.
closing resolved bugs as of 8.6p1 release