On Fri, 27 Aug 2010, Mr. MailingLists wrote: > Also, in my experience, compression destroys file transfer rates over > OpenSSH (on my LAN). > >> $ scp chickenclucker:~/175MBfile . >> 175MBfile 100% 175MB 29.1MB/s 00:06 >> >> $ scp -C chickenclucker:~/175MBfile . >> 175MBfile 100% 175MB 4.6MB/s 00:38 The first issue is that your network is super fast. You are sending about 230 Mbps uncompressed. You have to be able to compress and uncompress at a faster rate than that. You don't want to use compression when the network is unbelievably fast, but you may want to use it for slower networks, though only for certain file types. The next issue is the file type. What kind of file were you transferring? The thing is, many common files are already compressed. This is true of most image, audio and video formats. If you try to compress a file that is already compressed, it will get larger, not smaller. This is because there is some overhead in compression, and random binary streams (essentially what you get in an already-compressed file) cannot be compressed. Try making a big text file with a lot of redundancy in it, something that will compress a lot. For example: mkdir junk ; cd junk for i in $(seq 1000); do echo "0000000000000000000000000000000000000000000" >> foo.txt ; done for i in $(seq 1000); do cat foo.txt >> bar.txt ; done for i in $(seq 10); do cat bar.txt >> baz.txt ; done rm foo.txt bar.txt $ wc -c baz.txt 440000000 baz.txt $ gzip -c baz.txt | wc -c 1279822 In other words, baz.txt will compresses from 440 MB down to 1.3 MB. Try making that baz.txt file and see what its transfer speed looks like with and without compression. On my computer, to compress the file takes about 3.2 seconds. If your CPUs on both ends are at least that fast, then compression probably will be faster for that file than uncompressed transfer. If compression doesn't help you there, it will never help on your LAN, but it still will help for compressable files across a slower network, especially when both machines have fast CPUs and are available (not overloaded with other jobs). A few days ago I was transferring some files from the U to Scripps in San Diego. The files were highly compressable, but transfer was very slow. I think it was because the CPU on the other end was dealing with a lot of jobs, so it wasn't readily available for uncompressing the incoming data stream, and that slowed it way down. Mike