Hello, OpenSSH guys. As you can see here... https://unix.stackexchange.com/questions/434825/ssh-speed-greatly-improved-via-proxycommand-but-why ...I've spent a good part of the last two days tracing a weird performance issue with SSH. The executive summary is that I was able to significantly increase the performance of SSH (as used by rsync and borgbackup) by delegating the actual connection to netcat: $ dd if=/dev/urandom bs=1M count=50 of=sample.data 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.24135 s, 217 MB/s $ rsync -av --progress ./sample.data user@192.168.1.150:/dev/shm/ sending incremental file list sample.data 52,428,800 100% 575.23kB/s 0:01:28 (xfr#1, to-chk=0/1) sent 52,458,757 bytes received 35 bytes 543,614.42 bytes/sec total size is 52,428,800 speedup is 1.00 $ ssh user@192.168.1.150 'rm /dev/shm/sample.data' $ rsync -av --progress -e 'ssh -o "ProxyCommand nc %h %p"' ./sample.data user@192.168.1.150:/dev/shm/ sending incremental file list sample.data 52,428,800 100% 3.40MB/s 0:00:14 (xfr#1, to-chk=0/1) sent 52,458,757 bytes received 35 bytes 2,690,194.46 bytes/sec total size is 52,428,800 speedup is 1.00 As you can see, this is a 5x speedup - basically, the use of nc as a ProxyCommand boosted the speed to the same levels reported by iperf3 for the link. The reason behind this, ended up being the call to enable TCP_NODELAY - which if I understood correctly, disables Nagle's algorithm to increase the responsiveness of SSH. Which is good; SSH is supposed to be used for interactive shells, so I understand that decision. But since rsync and borgbackup and many others are using SSH as a universal transport system, I believe it makes sense to provide a generic option to enable/disable Nagle - e.g. --nodelay (default) --delay ...to control Nagle - so that tools using SSH for bulk transfers can benefit from "full" packets sent on the wire, and avoid wasting bandwidth. Especially over slow links - wifi in my case, but it can be any kind of slow link - this will have significant benefits - as you can see in my aforementioned (extensive) report in Unix Stack Exchange. I hope you will consider adding this option to OpenSSH... My understanding from the feedback I got talking about this in other forums is that other SSH implementations did offer such an option - and it shouldn't be too difficult to add... (we just want to control the invocation of the related setsockopt via a run-time controlled boolean). Thanks for you time - I hope you will consider this. Kind regards, Thanassis Tsiodras, Dr.-Ing.
I think it is worth mentioning this very old patch [1], it contains some good input regards a "NoDelay" option. [1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2002-January/010245.html
@Kristian: Unless I am mistaken, that post argues from the other side - that of a fast link, where they *want* to disable Nagle. As you can see in my case (slow wifi link), enabling Nagle dramatically increases the transfer bandwidth... ...which basically reinforces why we need to have this option as a user-configurable one: as is right now, the call to setsockopt that enables TCP_NODELAY is unconditional. IMHO it shouldn't be - the end-user should be able to control this, so that when SSH is used for transport, users with fast links can choose to disable Nagle - and users like me (with slow links) can enable it and avoid wasting bandwidth by sending half-empty packets.
I'm not sure that I understand: Nagle's algorithm is only disabled for interactive connections. Are you asking it be disabled for non-interactive connections too?
@Damien Miller: What I am saying (please see the linked post in UNIX Stack Exchange for details) is that when used as a transport, SSH cannot predict what is best; in my case, stracing showed that TCP_NODELAY was enabled, greatly impacting the speed over my congested WiFi link. # from strace output - for details see the linked StackExchange post 1522665534.007899 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000021> A simple addition of two options that control the setting... --nodelay (default) --delay ...would allow tools like borgbackup and rsync (that use SSH as a transport) to try out and adapt to whatever works best. Note that in the link from Kristian posted above, you can see arguments from the other side of the Nagle spectrum - basically confirming that direct control of Nagle's setup is a desired trait! And an easy one to add, I believe... (it's just a global boolean flag).