Comments inline:

On 8/27/2010 10:56 AM, Mike Miller wrote:
> On Fri, 27 Aug 2010, Mr. MailingLists wrote:
> 
>> Also, in my experience, compression destroys file transfer rates over 
>> OpenSSH (on my LAN).
>>
>>> $ scp chickenclucker:~/175MBfile .
>>> 175MBfile 100% 175MB 29.1MB/s 00:06
>>>
>>> $ scp -C chickenclucker:~/175MBfile .
>>> 175MBfile 100% 175MB 4.6MB/s 00:38
> 
> 
> The first issue is that your network is super fast.  You are sending about 
> 230 Mbps uncompressed.  You have to be able to compress and uncompress at 
> a faster rate than that.  You don't want to use compression when the 
> network is unbelievably fast, but you may want to use it for slower 
> networks, though only for certain file types.

Trying to compress something that cannot be compressed also uses CPU (a lot) time even
though you get nothing in return (but possibly a larger file).

> 
> The next issue is the file type.  What kind of file were you transferring? 

dd if=/dev/random of=175MBFile bs=1024k count=175

> The thing is, many common files are already compressed.  This is true of 
> most image, audio and video formats.  If you try to compress a file that 
> is already compressed, it will get larger, not smaller.  This is because 
> there is some overhead in compression, and random binary streams 
> (essentially what you get in an already-compressed file) cannot be 
> compressed.

Exactly, it really depends on how compressible the data is.

> 
> Try making a big text file with a lot of redundancy in it, something that 
> will compress a lot.  For example:
> 
> mkdir junk ; cd junk
> for i in $(seq 1000); do echo "0000000000000000000000000000000000000000000" >> foo.txt ; done
> for i in $(seq 1000); do cat foo.txt >> bar.txt ; done
> for i in $(seq 10); do cat bar.txt >> baz.txt ; done
> 
> rm foo.txt bar.txt
> 
> $ wc -c baz.txt
> 440000000 baz.txt
> 
> $ gzip -c baz.txt | wc -c
> 1279822
> 
> 
> In other words, baz.txt will compresses from 440 MB down to 1.3 MB.  Try 
> making that baz.txt file and see what its transfer speed looks like with 
> and without compression.  On my computer, to compress the file takes about 
> 3.2 seconds.  If your CPUs on both ends are at least that fast, then 
> compression probably will be faster for that file than uncompressed 
> transfer.  If compression doesn't help you there, it will never help on 
> your LAN, but it still will help for compressable files across a slower 
> network, especially when both machines have fast CPUs and are available 
> (not overloaded with other jobs).

My test case is being performed on the same machine transmitting to a jail. Here are my
findings:

dd if=/dev/random of=175MBFile bs=1024k count=175
scp 175MBFile chickenclucker:~/
175MBFile 100% 175MB 35.0MB/s 00:05
scp -C 175MBFile chickenclucker:~/
175MBFile 100% 175MB 9.7MB/s 00:18

time gzip -c 175MBFile > 175MBFile.gz
12.513s
183500800 Aug 27 11:58 175MBFile
183556808 Aug 27 11:58 175MBFile.gz

dd if=/dev/zero of=175MBFile bs=1024k count=175
scp 175MBFile chickenclucker:~/
175MBFile 100% 175MB 35.0MB/s 00:05
scp -C 175MBFile chickenclucker:~/
175MBFile 100% 175MB 35.0MB/s 00:05

time gzip -c 175MBFile > 175MBFile.gz
0m2.552s
183500800 Aug 27 12:03 175MBFile
178393 Aug 27 12:03 175MBFile.gz

Interesting results and I learned something new today (I friggan love when that happens!).

> 
> A few days ago I was transferring some files from the U to Scripps in San 
> Diego.  The files were highly compressable, but transfer was very slow.  I 
> think it was because the CPU on the other end was dealing with a lot of 
> jobs, so it wasn't readily available for uncompressing the incoming data 
> stream, and that slowed it way down.
> 
> Mike
>