Bug 2726 - Uploading of large files (1GB+) fails when using SFTP in chrooted configuration
Summary: Uploading of large files (1GB+) fails when using SFTP in chrooted configuration
Status: CLOSED WORKSFORME
Alias: None
Product: Portable OpenSSH
Classification: Unclassified
Component: sftp (show other bugs)
Version: 7.3p1
Hardware: amd64 Linux
: P5 normal
Assignee: Assigned to nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-06-01 23:24 AEST by Artur
Modified: 2018-04-06 12:26 AEST (History)
2 users (show)

See Also:


Attachments
SSHD config used in a test chrooted environment (1.42 KB, text/plain)
2017-06-01 23:24 AEST, Artur
no flags Details
Debug log file (46.62 KB, application/octet-stream)
2017-06-12 20:48 AEST, Artur
no flags Details
WinSCP debug log (10.22 KB, application/octet-stream)
2017-06-12 21:09 AEST, Artur
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Artur 2017-06-01 23:24:35 AEST
Created attachment 2986 [details]
SSHD config used in a test chrooted environment

I am unable unable to upload large files (1GB or greater) via SFTP when using SFTP in a chrooted configuration (using ChrootDirectory and "ForceCommand internal-sftp" in SSHD config). The symptomps are that WinSCP (or other SFTP clients) interrupts the connection with SFTP server once number of bytes transferred exceeds 1GB. Further diagnosis showed that this behaviour is related to the value of Rekeylimit - when reaching the Rekeylimit (as set explicitly in config file) the connection is interrupted immediately. Unfortunately, setting it's value to "none" does not solve the problem - probably SSHD takes the default value (1GB?) and the connection is interrupted when number of transferred bytes exceeds this value. SSHD ignores values larger than 1G so the one cannot set it over 1GB. 

I did not experience this bug when using SFTP without chrooting it - in a "regular" SSHD/SFTP configuration session key is renegotiated properly and large files can be transferred without problems.

Verified on the latest Gentoo and Ubuntu 16.04 LTS operating systems.

Attached you can find reference SSHD config that can be used to replicate the issue. Rekeylimit can be set, for instance, to 10M to experience the issue with smaller files.
Comment 1 Damien Miller 2017-06-09 13:42:18 AEST
To diagnose this, we really need a debug log from the server. You can get one by running "ssh -ddd -p 2222" and connecting to port 2222 with your sftp client. Please attach the resulting debug trace.

Are you using sshd that you compiled from source or your OS vendors' packages?
Comment 2 Darren Tucker 2017-06-09 14:24:24 AEST
(In reply to Artur from comment #0)
> SSHD ignores values larger than 1G so the one cannot set
> it over 1GB. 

Note that as per RFC4344 that depends on the cipher.

"""
   Let L be the block length (in bits) of an SSH encryption method's
   block cipher (e.g., 128 for AES).  If L is at least 128, then, after
   rekeying, an SSH implementation SHOULD NOT encrypt more than 2**(L/4)
   blocks before rekeying again.
[...]
   If L is less than 128 (which is the case
   for older ciphers such as 3DES, Blowfish, CAST-128, and IDEA), then,
   [...] rekey at least once for every gigabyte
"""

sshd follows this, so you'll see more frequent rekeying for ciphers like 3des.  For AES ciphers it should be 4GB, so if you're using 3des you might be able to work around the problem by changing ciphers.
Comment 3 Artur 2017-06-12 20:48:10 AEST
Created attachment 2992 [details]
Debug log file
Comment 4 Artur 2017-06-12 21:06:32 AEST
Here you go, you can find the debug log from sample test server as attachment.

The SSHD was started, then I tried to upload 43.5M file via WinSCP, while RekeyLimit was set to "5M none" (just to speed up occurance of rekey issue, without this it would happen at 1GB). As expected, transfer broke at 5M with WinSCP reporting an issue "Server unexpectedly closed network connection".

The example comes from the Gentoo OS, so it is compiled from the source (this is a default installation approach on Gentoo) using the below parameters:

net-misc/openssh-7.3_p1-r7::gentoo  USE="bindist hpn pam (pie) (selinux) ssl -X -X509 -debug -kerberos -ldap -ldns -libedit (-libressl) -livecd -sctp -skey -ssh1 -static {-test}" 0 KiB

Regards,
Artur
Comment 5 Artur 2017-06-12 21:09:58 AEST
Created attachment 2993 [details]
WinSCP debug log
Comment 6 Damien Miller 2017-06-14 11:54:00 AEST
(In reply to Artur from comment #4)

> net-misc/openssh-7.3_p1-r7::gentoo  USE="bindist hpn pam (pie)
> (selinux) ssl -X -X509 -debug -kerberos -ldap -ldns -libedit
> (-libressl) -livecd -sctp -skey -ssh1 -static {-test}" 0 KiB

That is OpenSSH plus a number of third-party patches that we neither support nor maintain. Please try to reproduce against OpenSSH that you've built from source yourself, or take this bug up with your OS vendor.
Comment 7 Artur 2017-06-14 18:37:20 AEST
The same behavior I experienced on default Ubuntu 16.04 LTS distro (installed from the package) - from my point of view it seems the bug does not depend on a particular Linux distro.

As suggested ealier, you can easily replicate it on other Linux variants by using the attached sshd config.

Please let me know if the logs attached are sufficient or you still need logs from eg. Ubuntu, to re-assure this bug is OS independent.
Comment 8 Darren Tucker 2017-06-14 20:56:32 AEST
(In reply to Artur from comment #7)
> The same behavior I experienced on default Ubuntu 16.04 LTS distro

Ubuntu 16.04 ships an older version of OpenSSH than this bug report is against.

> (installed from the package) - from my point of view it seems the
> bug does not depend on a particular Linux distro.

Please consider it from our point of view: you are asking us to debug a system we have never seen, cannot interact with and, in the case of Gentoo, may well be unique.  There are any number of differences between that and source code we can debug, and we do not know what they are.  If we cannot replicate locally (and I've tried your config on -current on OpenBSD and 7.3p1 on Fedora and failed, despite those sessions rekeying multiple times) then we do not know out what the issue is.

So, there are a number of differences between the system that you're experiencing the problem on and the ones we can observe.

If you wish to make progress with this bug then your choices are:
a) help us reduce the variables by testing the *most recent version of* code we actually ship
b) report you problem to the people who ship the thing you are having the problem with (and they can report it to us if it turns out to be in the code we ship).

> Please let me know if the logs attached are sufficient or you still
> need logs from eg. Ubuntu, to re-assure this bug is OS independent.

No, we need logs from the code we ship, not from some other random binary.
Comment 9 Darren Tucker 2017-07-27 13:52:00 AEST
If you can't provide logs from the code we ship then there's not much more we can do to help you.  Please feel free to reopen if you are able to provide them.
Comment 10 Artur 2017-08-04 00:32:58 AEST
I verified OpenSSH behaviour once again, based on the most recent stable version (7.5p1) compiled from the sources. This version turn to work properly.

To summary, the reported issue affects default Ubuntu 16.04LTS and Gentoo distributions. As a workaround users of these systems may need to download, compile and use OpenSSH from the sources.

Therefore the bug can be closed.
Comment 11 Damien Miller 2018-04-06 12:26:46 AEST
Close all resolved bugs after release of OpenSSH 7.7.