I did some test and when enabling compression my CPU usage goes at 100% at about 70-150Mb/s, depending on file compressibility. So it is not very useful with a Gigabit Ethernet link. To improve this I would propose to add other compression schemes, such as lz4 ( https://github.com/lz4/lz4 ) or zstd ( https://github.com/facebook/zstd ). They are order of magnitudes faster than current zlib-6 compression and easily able to exceed Gigabit Ethernet speed.
See also bug #2887.
I came here to request alternate compression methods, and found someone had beat me to it. Consider this my +1 Currently OpenSSH only supports GZIP compression which has speed constraints on modern CPUs. Enabling compression on file transfers over a gigabit link actually increases the amount of time taken to transfer large files due to the limitations of GZIP. Please consider adding support for alternate compression methods. LZ4 and ZSTD would be excellent choices. Even better would be support for external compression methods like `tar` does with `-I`. That way in the future if new/better compression algorithms are found SSH will automatically be able to utilize them.
Transferring a 1.8GB VM image on a gigabit network: ``` root@green(~) :time scp test.bin yellow:/dev/null test.bin 100% 1821MB 108.0MB/s 00:16 # Enabling compression is significantly slower root@green(~) :time scp -C test.bin yellow:/dev/null test.bin 100% 1821MB 20.0MB/s 01:30 ``` Alternately using this "hack" with `cat`, `ssh`, and ZSTD I was able to decrease transfer time by ~40%. ``` root@green(~) :time cat test.bin | zstd | ssh yellow 'zstd - -d > /dev/null' real 0m9.691s user 0m12.178s sys 0m4.458s ``` For transferring large files, and/or groups of large files via scp/rsync having a modern compression method would be a **huge** win.
please add support for this.
Created attachment 3373 [details] Support for zstd against V_8_2_P1 This archive contains four patches against openssh v8.2.p1. Would love some feedback if the expectations regarding scp performance/CPU load are met.
@Sebastian Thanks for your patches. I'd suggest sending them on the OpenSSH Developers list: https://www.openssh.com/list.html and eventually as a Pull request on github: https://github.com/openssh/openssh-portable/pulls