Bug 2091 - scp hangs while copying a large file and being executed as a background process ( with nohup )
Summary: scp hangs while copying a large file and being executed as a background proce...
Status: REOPENED
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: scp (show other bugs)
Version: 6.1p1
Hardware: Other AIX
: P5 normal
Assignee: Assigned to nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-04-15 22:40 AEST by Vikas K Vicky
Modified: 2020-02-12 04:23 AEDT (History)
3 users (show)

See Also:


Attachments
Detailed Debug info. (7.10 KB, text/plain)
2013-04-15 22:40 AEST, Vikas K Vicky
no flags Details
failing example using ksh (87.78 KB, text/plain)
2020-01-28 23:22 AEDT, karl.mumford
no flags Details
working example using csh (330.42 KB, text/plain)
2020-01-28 23:24 AEDT, karl.mumford
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vikas K Vicky 2013-04-15 22:40:08 AEST
Created attachment 2244 [details]
Detailed Debug info.

Problem Description - 
=====================

scp hangs while copying a large file and being executed as a background process ( with nohup ).

The following command is used to start an scp copy in the background and then exit the ksh. 
This should allow the user to leave a large file secure copy running unattended but instead it 
results in 2 hung processes as shown at the end of this bug report. This occurs only with ksh.

nohup scp /tmp/testfile test3@target_machine:/tmp/testfile &  


============================================================================================================

Steps tp Reproduce - 
=====================                      

                                                                         
On the source_machine:
=======================                      

1. login as user test1                                                   

   make sure there is a user test1,                                      

                                                                         

2. scp /tmp/testfile test3@target_machine:/tmp/testfile                  

   make sure ssh server is running on the target machine                 

   make sure the testfile is in folder /tmp/testfile, and is big enough, 

   about 780MB. 
                                                            

   ls -l /tmp/testfile                                                   

   -rwxrwxrwx    1 root     system    827621072 Aug 31 11:06   /tmp/testfile                                                            

                                                                         

3. set up scp, so that scp can be used without prompting for password         

   cd /home/test1                                                        

   ssh-keyget -t rsa                                                     

   This will prompt for a passphrase. Just press the enter key. It'll then generate an identification 
   (private key) and a public key.          

   Your public key has been saved in <your_home_dir>/.ssh/id_rsa.pub, /home/test1/.ssh/id_rsa.pub                                              

   ftp the /home/test1/.ssh/id_rsa.pub file to the target_machine /tmp  folder                                                                   

                                                                         

On the target_machine:
=======================                      

1. login as user test3                                                   

   make sure there is a user test3,                                      

                                                                         

2. set up scp, so scp can be used on the source_machine without          

   prompting for password  
                                                 

   cd /home/test3/.ssh                                                   

   cat /tmp/id_rsa.pub >> authorized_keys                                

   chmod 700 authorized_keys                                             

                                                                         

3. remove /tmp/testfile if it's existing                                 

   rm -i /tmp/testfile                                                   

   make sure there is no /tmp/testfile existing on the target_machine before running the scp command   

================================================================================================================                                        

                                                                         
Go to the source_machine: ( still login as test1 ) 
------------------------------------------------------                      

1. make sure /tmp/testfile is there and big enough                       

   ls -l /tmp/testfile                                                   

   -rwxrwxrwx    1 root     system    827621072 Aug 31 11:06             

/tmp/testfile                                                            

                                                                         

2. run the scp command                                                   

   nohup scp /tmp/testfile test3@target_machine:/tmp/testfile &          

                                                                         

Go to the target_machine: ( still login as test3 ) 
-----------------------------------------------------                     

   make sure the /tmp/testfile is being copied                           

   ls -l /tmp/testfile                                                   

   -rwxr-xr-x    1 test3    staff      86278144 Sep 01 12:25             

/tmp/testfile                                                            

                                                                         

   ls -l /tmp/testfile                                                   

   -rwxr-xr-x    1 test3    staff     169050112 Sep 01 12:25             

/tmp/testfile                                                            

                                                                         

   you can see the size is increasing.                                   

                                                                         

Go to the source_machine: ( still login as test1 ) 
-----------------------------------------------------                      

   completely logout from the source_machine, you should get the message 

of "You have running jobs".                                              

                                                                         

Go to the target_machine: ( still login as test3 ) 
-----------------------------------------------------                      

   check the size of /tmp/testfile again                                 

   ls -l /tmp/testfile                                                   

   -rwxr-xr-x    1 test3    staff     277184512 Sep 01 12:25             

/tmp/testfile                                                            

                                                                         

   ls -l /tmp/testfile                                                   

   -rwxr-xr-x    1 test3    staff     277184512 Sep 01 12:25             

/tmp/testfile                                                            

                                                                         

   you can see the size is not increasing any more. The scp has not      

   finished.                                                                

   The problem has been reproduced.

====================================================================================================================


Go to the source_machine and login as test1, there were 2 scp related    

processes hung after exiting from the ksh -                                

                                                                         

   ps -ef|grep scp                                                       

   test1 1245328       1   0 16:32:39      -  0:00 scp /tmp/testfile     

test3@target_machine:/tmp/testfile                                       

   test1 1638838 1245328   0 16:32:39      -  0:18 /usr//bin/ssh -x      

-oForwardAgent no -oPermitLocalCommand no -oClearAllForwardings yes -l   

test3 -- target_machine scp -t -- /tmp/testfile                          

   test1 4850090 3998138   0 16:42:59  pts/1  0:00 grep scp
Comment 1 Damien Miller 2013-10-24 10:46:14 AEDT
Can you repeat this using "scp -q"?
Comment 2 Damien Miller 2014-07-03 15:41:47 AEST
8 months with no followup = no bug
Comment 3 Damien Miller 2016-08-02 10:41:09 AEST
Close all resolved bugs after 7.3p1 release
Comment 4 karl.mumford 2018-05-03 22:58:12 AEST
This bug is still there in ssh v7.1p1 on AIX and can be reproduced in exactly the same way.
It should be noted that the scp hang only occurs in Kron shell i.e. the same procedure done in Bourne or C shells does not cause the hang.
A workaround is to use the 'screen' command instead of nohup and disconnect.
Another workaround is to ssh from the source system back to itself, run the nohup scp & command and then background the looped ssh connection with ~&
Comment 5 Damien Miller 2020-01-26 00:48:23 AEDT
Please attach a debug log ("scp -vvv ...") if you're still experiencing this. It's not possible to debug this with the information available.
Comment 6 karl.mumford 2020-01-28 23:22:37 AEDT
Created attachment 3353 [details]
failing example using ksh

output for hanging command when using Korn shell on AIX 6.1
nohup scp -vvv /tmp/testfile host:/tmp &
then logout
Comment 7 karl.mumford 2020-01-28 23:24:06 AEDT
Created attachment 3354 [details]
working example using csh

output for workingcommand when using C shell on AIX 6.1
nohup scp -vvv /tmp/testfile host:/tmp &
then logout
Comment 8 Yuuki Harano 2020-02-12 04:23:00 AEDT
I don't know about Korn Shell. However, if it sends SIGHUP to jobs when termination, then the bug is reproducible with zsh on Linux.
So, I debugged.

1. nohup executes scp with SIGHUP is SIG_IGN.
2. scp forks and executes ssh with SIGHUP is SIG_IGN.
3. scp sets up SIGHUP as its own handler, while ssh doesn't because it is SIG_IGN.
4. transfer starts.
5. logout.
6. shell sends SIGHUP to scp and ssh. (*1)
7. in scp, the specified signal handler is called. in ssh, SIGHUP is ignored.
8. in the signal handler in scp, it sends SIGHUP to ssh.
9. ssh ignores the SIGHUP too.
10. scp waits for ssh to exit in the signal handler, so it doesn't send data to ssh.

(*1) please configure your shell to send SIGHUP to jobs when shell's termination,
otherwide that doesn't happen.

8.1p1.