Bug 2090

Summary: SSH/SSHD hang with a Match User setting in sshd_config .
Product: Portable OpenSSH Reporter: Vikas K Vicky <vikas.uit10>
Component: sshdAssignee: Assigned to nobody <unassigned-bugs>
Status: CLOSED INVALID    
Severity: normal CC: djm, dtucker, rafaelantonioli
Priority: P5    
Version: 6.0p1   
Hardware: Other   
OS: AIX   
Attachments:
Description Flags
Detailed Debug info.
none
Debug-Output-Without-Hang
none
Debug-Output-With-Hang
none
Debug-Output-with-Hang-NEW none

Description Vikas K Vicky 2013-04-15 21:08:46 AEST 
Created attachment 2243 [details]
Detailed Debug info.

Problem Description - 
======================

SSH/SSHD hang when SSHD is configured to allow only sftp connection requests from a particular user 
( Match User setting in sshd_config ).


Steps to reproduce - 
=====================
NOTE- These steps are for recreating the scenario only to a localhost( which acts as both Client & Server) and similar steps can be used 
to recreate it for a Client & Server ( different machines ).

1. Create a local user say "test" .Also,define a password for user "test" .

2. Set-up the password less( public key ) authentication for user "test" ,
   i.e Copy /home/test/.ssh/id_rsa.pub to  /home/test/.ssh/authorized_keys ( So, I don't need to type the test's password every time).

3. Edit the sshd_config file to include the Match User setting, so that user "test"  can do only sftp to this server ( i.e localhost )- 

   # Example of overriding settings on a per-user basis
   #Match User anoncvs
   #       X11Forwarding no
   #       AllowTcpForwarding no
   #       ForceCommand cvs server
   Match User test
        ForceCommand internal-sftp

4. Restart SSHD .

5. Try to login using sftp (should be OK as
   expected):                                

         
   $ sftp test@localhost                                                    

   Connecting to localhost...                                               

   test@localhost's password:                                               

   sftp>  

6. Try to ssh to the localhost (Here is the issue . It will hang.):  
 
   SSH hangs after random number of repeated login requests ( see the infinite while loop used to send repeated login
   requests . Also ,Since Public key authentication has been configured for user "test" , it will not ask for its password ).
 
(0) root @ ic15: 7.1.0.0: /
# su test

(0) test @ ic15: 7.1.0.0: /
# while
> true
> do
> ssh localhost
> done
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.               <== It hangs here after a random number of login requests.


The below SSHD processes are active at the same time - 

(0) root @ ic15: 7.1.0.0: /
# ps -ef | grep sshd
    root 5111844 9502940   0 15:24:49      -  0:00 sshd: test [priv]
    test 9175114 5111844   0 15:24:50      -  0:00 sshd: test@pts/7
    root 9502940 3604646   0 15:17:24      -  0:00 /usr/sbin/sshd

I believe, there is some Communication problem between the monitor sshd process (5111844) and its child 9175114.
(Some signal is being missed from either side ).

Looks like, both Client (ssh) & Server (sshd) are waiting for each other and a deadlock kind of scenario is being created.
Please check the attached .txt file for detailed debugging information.
Comment 1 Darren Tucker 2013-04-16 11:47:26 AEST 
the debug logs say 6.0p1 not 6.1p1.  other than that, they're not all that useful.  They just show the monitor sitting there waiting to be asked to do something (which is normal) and and on the slave side just shows that it's waiting for input.

Which version (and ML) of AIX is this?

Can you get the debug output from ssh and sshd and attach it to this bug?  For the client it should be simple:

$ while true; do ssh -vvv localhost; done

for the server, you can try this assuming you have sudo available:

$ while true; do ssh -o 'proxycommand sudo /usr/sbin/sshd -e -i -o loglevel=debug3' localhost; done

it's possible that won't trigger the problem, in that case you'll need to set loglevel debug3 in your sshd_config and grep the debug logs out of syslog or something.
Comment 2 Vikas K Vicky 2013-04-16 21:13:54 AEST 
Actually I have tested with OpenSSH 5.8p1,6.0p1,6.1p1 & 6.2p1 and this problem is present in all these versions.

The problem can be reproduced in AIX version 6.1 & 7.1 both with any of the above mentioned OpenSSH versions.

I am attaching here the debug logs from both Client & Server.

NOTE- One Debug log (Debug-Output-Without-Hang) is for successful scenario , i.e No SSH/SSHD doesn't hang with the configuration as mentioned in the recreation steps.

Other Debug log(Debug-Output-With-Hang) is for the Hang scenario , i.e SSH/SSHD hangs with the configuration as mentioned in the recreation steps ( you need to try ssh login manually for random number of times to recreate it).

One thing I noticed that for SSH/SSHD hang to happen - ssh login requests should be very frequent from Client side. If there is considerable amount of time gap between two consecutive ssh login requests , then chances for Hang is very rare.
Comment 3 Vikas K Vicky 2013-04-16 21:20:16 AEST 
Created attachment 2246 [details]
Debug-Output-Without-Hang
Comment 4 Vikas K Vicky 2013-04-16 21:22:20 AEST 
Created attachment 2247 [details]
Debug-Output-With-Hang
Comment 5 Darren Tucker 2013-04-16 21:39:08 AEST 
is this a stock version of openssh compiled from the source at openssh.com?  I see entries in the logs which don't seem to belong ("debug3: ACCESS KEY", ".  If so, can you reproduce the problem with a stock version?  If so, please attach those logs.
Comment 6 Vikas K Vicky 2013-04-23 15:48:42 AEST 
Ok.. I will upload the logs from stock version very soon.
Comment 7 Vikas K Vicky 2013-10-03 23:41:24 AEST 
Created attachment 2343 [details]
Debug-Output-with-Hang-NEW

Attaching "Debug-Output-with-Hang-NEW.txt" which is the debug log from stock OpenSSH version 6.0p1.
Comment 8 Darren Tucker 2013-10-04 00:40:30 AEST 
Comment on attachment 2343 [details]
Debug-Output-with-Hang-NEW

>debug1: init_func_ptrs success

that is not from the stock openssh available from openssh.com

>debug1: Value of krb5 is 1

or that

>debug1: Value of gssapi is 1

or that

>From Server side - 
[...]
>debug3: Value for authType is STD_AUTH

or that

>debug1: init_func_ptrs passed

or that

>debug1: value of krb5 is : 1

or that

>debug1: value of gssapi is : 1

or that

>debug1: Failed to collect Cookie from Keystore

or that

>debug1: Keystore Opening wil be failed after login

or that

>debug1: Cookie received :

or that

>debug1: Failed to collect Cookie from Keystore

or that

>debug1: Keystore Opening wil be failed after login

or that.  and it's not even spelt correctly.

You are using a (heavily!) modified version of openssh.  We have no idea what's in the software you're running.  You need to seek assistance from whoever supplied this modified version of openssh.

If you can download the tarball from openssh.com, compile, install and reproduce the problem then we may be able to help you. If you can do that, please reopen this bug and attach the output of ./configure, the config.h file and the debug output from the server.
Comment 9 Damien Miller 2021-04-23 15:02:00 AEST 
closing resolved bugs as of 8.6p1 release