On Fri, Jul 27, 2012 at 10:36:18PM -0500, Robert Nesius wrote:
> > > I think there is more post-processing on the server side
> >
> > The server load is 5% in both cases.
> 
> Sorry Florin - I didn't explain my thoughts very clearly...
> 
> I meant more post processing in the form of micro reads/writes to the
> underlying filesystem in CIFS.

> > >                                                           and that's
> > going
> > > to slow down server responses and that's going to push back upstream and
> > > cause writes to wait.
> >
> > On the contrary, with CIFS, more data gets sent through the network,
> > because the client keeps track of cluster allocation, reads and
> > updates the filesystem metadata.  With CIFS is "here's the file,
> > kthxby".
> >
> 
> I'm going to assume that first CIFS was meant to be an iSCSI, and if so

Correct.

> then what you said is exactly what I was thinking.  And I don't think the
> issue is the amount of data being sent over the network in either case.
>  It's the number of interactions! required between the client and server,
> and the server and the underlying disk.  Yes, the iSCSI system is tracking
> more than the CIFS system, but when it comes time to throw a lot of bits at
> the storage medium, iSCSI is optimized to get it done in a more streamlined
> fashion.  Wasn't the whole point of iSCSI to sidestep the inherent
> slowdowns/inefficiencies with network filesystems like CIFS?  You're

Let's assume XFS/EXT4 and NTFS perform a similar amount of work to
store a large file on disk.  They all support extents, the file system
is empty, there is only one client, appending to a newly created file.
You cannot get a more blue sky scenario for a file system.

Now, that work is comprised of some amount of disk I/O and some amount
of computation.  We can ignore the computation part since both
machines are mostly idle and their raw CPU capability dwarf the
problem at hand.  We are left with the I/O:  in the case of iSCSI that
I/O has to cross the network, reducing the link bandwidth and adding to
latency, while in case of Samba that I/O is between the server's disk
and its memory.

That's why I contend that Samba should be faster than iSCSI for my
test.

> http://jpaul.me/?p=787 is showing exactly what you're reporting, btw.

Funny, as my use case is the same - I have a large archive of RAW
images (~21MB each).

> > Of course they are waiting for network IO.  The question is why is
> > CIFS waiting for network IO and what can I do to reduce that wait.
> >
> > > Are you writing to a RAM DISK?
> >
> > No, I'm writing to a SATA disk that can write sequentially more than
> > 100MBytes/second.
> 
> Just wondering why you mentioned your server has 16GB of RAMsince that's
> irrelevant then, unless you're pointing out there's no way it's swapping.

I'm pointing that Samba is not waiting for the disk since it can throw
the file over to the Linux cache.

>  100MBytes/second throughput on your hard drive is also irrevelant if the
> disk-access patterns are sending the heads around to update underlying
> filesystem structures.
> 
> Try allocating a 12 GB RAM DISK on your cifs server and serving it up via
> CIFS and then watch CIFS close the gap to iSCSI when you drop your 11gb
> file on it.

I did, and the results (attache) are not pretty... The tmpfs behaves the same as
the disk.  Storage is not the bottleneck, Windows <-> Network <-> Samba is.

> > > > The CIFS results were so bad, I was concerned there was a problem with
> > > > the hardware.  It just occurred me two days ago that I could flip
> > things
> > > > around, use iSCSI and test the system performance in a different way.
> 
> But as you pointed out later, that's Apples and Oranges.

Umm... they might be Apples and Oranges in the general case, but in
the case of one client, one file they are the same ;)  And I only set
up this comparison for the purpose of eliminating the hardware as a
failure point.  Which it did.  So now the question moves someplace
else.

> > > Maybe watch each of your tests with wireshark and watch latencies?
> >
> > How would that help me?  What would that tell me that I don't know
> > already?  CIFS and iSCSI use different ports, different protocols,
> > different services...
> 
> Anyway, just trying to help.

I fully understand, and my questions were batched to deal with the
limitations of e-mail communication, not intended to overwhelm the
recipient ;)

>                              Good luck with it, if you're still spending
> time on it.

At this point I pretty much gave up on getting more out of Samba. I hope
Synology is a good answer, if not, I'll come back to it.  If I have
some time I'll try installing Illumian just to check out their CIFS
implementation.

Best,
florin

-- 
Beware of software written by optimists!
-------------- next part --------------
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0  99   1   0   0| 722k  430k|   0     0 |   0     0 | 285   364 
  0   0 100   0   0   0|   0  1638B| 366B  415B|   0     0 | 102   169 
  0   0 100   0   0   0|   0  2458B|  12B   34B|   0     0 |  64   117 
  0   0 100   0   0   0|   0   127k|  24B   42B|   0     0 |  75   134 
  0   0 100   0   0   0|4915B 1638B|  85B   71B|   0     0 |  62   118 
  0   0 100   0   0   0| 819B    0 | 326B  176B|   0     0 |  74   118 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  61   115 
  0   0 100   0   0   0|   0   819B| 155B  204B|   0     0 |  62   119 
  0   1  99   0   0   0|   0     0 |  42M  215k|   0     0 |3041  5555 
  1   1  98   0   0   0|   0    11k|  43M  281k|   0     0 |3590  6330 
  1   1  99   0   0   0|   0     0 |  34M  263k|   0     0 |3169  5675 
  1   1  99   0   0   0|   0     0 |  45M  283k|   0     0 |3738  6643 
  1   1  98   0   0   0|   0     0 |  36M  271k|   0     0 |3293  5858 
  1   1  99   0   0   0|   0   819B|  31M  260k|   0     0 |3018  5466 
  1   1  98   0   0   0|   0     0 |  40M  276k|   0     0 |3458  6246 
  1   1  98   0   0   0|   0  1946B|  40M  283k|   0     0 |3504  6267 
  1   1  98   0   0   0|   0     0 |  12M  232k|   0     0 |2191  4066 
  1   1  99   0   0   0|   0     0 |  56M  303k|   0     0 |4188  7567 
  1   1  99   0   0   0|   0     0 |  45M  284k|   0     0 |3650  6642 
  1   1  99   0   0   0|   0   819B|  47M  284k|   0     0 |3712  6717 
  1   1  99   0   0   0|   0     0 |  40M  265k|   0     0 |3483  6061 
  1   1  99   0   0   0|   0  1946B|  44M  277k|   0     0 |3763  6393 
  1   1  99   0   0   0|   0     0 |  36M  275k|   0     0 |3318  6060 
  1   1  99   0   0   0|   0     0 |  43M  279k|   0     0 |3630  6511 
  1   1  99   0   0   0|   0     0 |  39M  269k|   0     0 |3345  6041 
  1   1  98   0   0   0|   0   819B|  26M  254k|   0     0 |2823  5155 
  1   1  98   0   0   0|   0     0 |  46M  284k|   0     0 |3691  6670 
  1   1  98   0   0   0|   0  1946B|  46M  283k|   0     0 |3703  6668 
  1   1  99   0   0   0|   0     0 |  37M  263k|   0     0 |3226  5828 
  1   1  98   0   0   0|   0     0 |  27M  246k|   0     0 |2776  4962 
  1   1  98   0   0   0|   0     0 |5455k  173k|   0     0 |1503  2846 
  1   1  98   0   0   0|   0   819B|  24M  251k|   0     0 |2746  5008 
  1   1  98   0   0   0|   0     0 |  30M  251k|   0     0 |2968  5288 
  1   1  98   0   0   0| 819B 1946B|  36M  259k|   0     0 |3299  5707 
  1   1  98   0   0   0|   0  5734B|  24M  241k|   0     0 |2687  4672 
  2   1  98   0   0   0|   0     0 |6245k  250k|   0     0 |2157  4082 
  2   1  98   0   0   0|   0     0 |6040k  236k|   0     0 |2082  3825 
  1   1  98   0   0   0|   0   819B|  15M  254k|   0     0 |2484  4591 
  1   1  98   0   0   0|   0     0 |  27M  248k|   0     0 |2866  5178 
  1   1  98   0   0   0|   0    19k|  31M  257k|   0     0 |3063  5605 
  1   1  99   0   0   0|   0  4096B|  45M  283k|   0     0 |3681  6797 
  1   1  99   0   0   0|   0     0 |  43M  276k|   0     0 |3603  6458 
  1   1  98   0   0   0|   0     0 |8296k  214k|   0     0 |1914  3566 
  1   1  98   0   0   0|   0   819B|  36M  256k|   0     0 |3216  5726 
  1   1  99   0   0   0|   0     0 |  39M  274k|   0     0 |3431  6225 
  1   1  98   0   0   0|   0    11k|3412k  223k|   0     0 |1784  3424 
  1   1  98   0   0   0|   0     0 |7912k  215k|   0     0 |1887  3518 
  1   1  98   0   0   0|   0     0 |  39M  271k|   0     0 |3437  6115 
  1   1  98   0   0   0|   0     0 |  10M  213k|   0     0 |1926  3530 
  1   1  99   0   0   0|   0   819B|  35M  258k|   0     0 |3151  5768 
  1   1  98   0   0   0|   0     0 |  30M  250k|   0     0 |3006  5278 
  1   1  98   0   0   0|   0  1946B|  56M  301k|   0     0 |4241  7623 
  1   1  99   0   0   0|   0     0 |  28M  245k|   0     0 |2821  5003 
  1   1  98   0   0   0|   0     0 |  22M  223k|   0     0 |2528  4329 
  1   1  98   0   0   0|   0     0 |  22M  250k|   0     0 |2718  4915 
  1   1  98   0   0   0|   0  1638B|4845k  235k|   0     0 |1966  3746 
  1   1  98   0   0   0|8192B    0 |  22M  259k|   0     0 |2841  5086 
  1   1  98   0   0   0|   0  1946B|2441k  236k|   0     0 |1928  3664 
  1   1  98   0   0   0|   0     0 |  16M  247k|   0     0 |2462  4519 
  1   1  98   0   0   0|   0     0 |  36M  276k|   0     0 |3280  5964 
  1   1  98   0   0   0|   0     0 |  17M  245k|   0     0 |2452  4591 
  1   1  99   0   0   0|   0   819B|  51M  298k|   0     0 |3985  7424 
  0   1  99   0   0   0|   0  2458B|  63M  322k|   0     0 |4498  8354 
  1   1  98   0   0   0|   0  5325B|  23M  231k|   0     0 |2552  4660 
  1   1  99   0   0   0|   0     0 |  43M  274k|   0     0 |3542  6531 
  1   1  98   0   0   0|   0     0 |  40M  277k|   0     0 |3482  6428 
  2   1  98   0   0   0|   0     0 |1950k  228k|   0     0 |1804  3489 
  2   1  98   0   0   0|   0   819B|2370k  235k|   0     0 |1872  3598 
  1   1  98   0   0   0|   0     0 |  20M  255k|   0     0 |2648  4935 
  1   1  98   0   0   0|   0  1946B|  33M  259k|   0     0 |3125  5678 
  1   1  98   0   0   0|   0     0 |  39M  271k|   0     0 |3463  6113 
  1   1  98   0   0   0|   0     0 |  42M  281k|   0     0 |3623  6437 
  1   1  99   0   0   0|   0     0 |  46M  282k|   0     0 |3760  6592 
  1   1  98   0   0   0|   0   819B|  30M  243k|   0     0 |2959  5127 
  1   1  98   0   0   0|   0     0 |  27M  245k|   0     0 |2767  5108 
  1   1  98   0   0   0|   0  1946B|  17M  239k|   0     0 |2389  4496 
  1   1  98   0   0   0|   0     0 |  22M  248k|   0     0 |2612  4867 
  1   1  99   0   0   0|   0     0 |  45M  287k|   0     0 |3722  6898 
  1   1  99   0   0   0|   0     0 |  46M  284k|   0     0 |3720  6813 
  1   0  99   0   0   0|   0   819B|  14M  184k|   0     0 |1896  3418 
  0   0 100   0   0   0|   0     0 | 886B  893B|   0     0 |  66   131 
  0   0 100   0   0   0|   0  1946B|  12B   34B|   0     0 |  66   118 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  60   112 
  0   0 100   0   0   0|   0     0 |  49B   59B|   0     0 |  60   114 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  57   109 
  0   1  99   0   0   0|   0   819B| 414B  390B|   0     0 | 136   166 
-------------- next part --------------
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  1   0  99   0   0   0| 319k  191k|   0     0 |   0     0 |1454  2567 
  0   0 100   0   0   0|   0   819B| 366B  936B|   0     0 |  85   152 
  0   0 100   0   0   0|   0    17k|  66B  131B|   0     0 |  65   133 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  59   113 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  59   113 
  0   0 100   0   0   0|   0     0 | 314B  165B|   0     0 |  59   116 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  58   112 
  0   0  99   0   0   0|   0   819B|  42M  217k|   0     0 |3078  5614 
  1   1  99   0   0   0|   0  4403B|  41M  270k|   0     0 |3506  6182 
  1   1  99   0   0   0|   0     0 |  35M  266k|   0     0 |3228  5796 
  1   1  99   0   0   0|   0     0 |  44M  282k|   0     0 |3656  6537 
  1   1  98   0   0   0|   0     0 |  35M  261k|   0     0 |3270  5726 
  1   1  98   0   0   0|   0     0 |  30M  257k|   0     0 |3009  5436 
  1   1  99   0   0   0|   0   819B|  38M  271k|   0     0 |3368  6095 
  1   1  99   0   0   0|   0  1946B|  43M  299k|   0     0 |3680  6687 
  1   1  98   0   0   0|   0     0 |  10M  231k|   0     0 |2144  3992 
  0   1  99   0   0   0|   0     0 |  56M  303k|   0     0 |4166  7515 
  1   1  99   0   0   0|   0     0 |  45M  281k|   0     0 |3668  6583 
  1   1  98   0   0   0|   0     0 |  48M  292k|   0     0 |3836  6841 
  1   1  99   0   0   0|   0   819B|  39M  263k|   0     0 |3456  5936 
  0   1  99   0   0   0|   0  1946B|  44M  278k|   0     0 |3718  6505 
  1   1  98   0   0   0|   0     0 |  37M  279k|   0     0 |3368  6074 
  1   1  98   0   0   0|   0     0 |  45M  293k|   0     0 |3739  6727 
  1   1  99   0   0   0|   0     0 |  37M  265k|   0     0 |3255  5888 
  1   1  98   0   0   0|   0     0 |  28M  260k|   0     0 |2933  5360 
  1   1  99   0   0   0|   0   819B|  44M  276k|   0     0 |3632  6520 
  1   1  98   0   0   0|   0  1946B|  46M  279k|   0     0 |3681  6644 
  1   1  99   0   0   0|   0     0 |  36M  260k|   0     0 |3187  5737 
  1   1  98   0   0   0|   0     0 |  25M  233k|   0     0 |2617  4608 
  1   1  98   0   0   0|   0     0 |8335k  227k|   0     0 |2006  3765 
  1   0  98   0   0   0|   0     0 |  27M  249k|   0     0 |2830  5083 
  1   1  98   0   0   0|   0   819B|  36M  271k|   0     0 |3276  5835 
  1   1  98   0   0   0|   0  1946B|  28M  246k|   0     0 |2894  5101 
  1   1  99   0   0   0|   0     0 |  26M  239k|   0     0 |2752  4806 
  1   1  98   0   0   0|   0     0 |2381k  235k|   0     0 |1907  3666 
  1   1  98   0   0   0|   0     0 |6048k  239k|   0     0 |2091  3869 
  1   1  98   0   0   0|   0     0 |  19M  256k|   0     0 |2606  4822 
  1   1  98   0   0   0|   0   819B|  24M  253k|   0     0 |2747  5020 
  1   1  98   0   0   0|   0  1946B|  39M  275k|   0     0 |3399  6223 
  1   1  99   0   0   0|   0     0 |  45M  286k|   0     0 |3641  6737 
  1   1  99   0   0   0|   0     0 |  35M  262k|   0     0 |3182  5711 
  1   1  98   0   0   0|   0     0 |9876k  222k|   0     0 |2022  3758 
  1   1  99   0   0   0|   0     0 |  44M  270k|   0     0 |3572  6310 
  1   0  99   0   0   0|   0   819B|  29M  272k|   0     0 |3013  5617 
  1   1  98   0   0   0|   0  1946B|5720k  228k|   0     0 |1890  3599 
  1   1  98   0   0   0|   0     0 |  17M  231k|   0     0 |2308  4228 
  1   1  98   0   0   0|   0     0 |  33M  266k|   0     0 |3111  5597 
  1   1  98   0   0   0|   0     0 |  10M  214k|   0     0 |1927  3562 
  1   1  98   0   0   0|   0     0 |  43M  281k|   0     0 |3564  6469 
  1   1  98   0   0   0|   0   819B|  34M  261k|   0     0 |3188  5628 
  0   1  99   0   0   0|   0  1946B|  55M  302k|   0     0 |4117  7595 
  1   1  98   0   0   0|   0     0 |  17M  211k|   0     0 |2238  3832 
  1   1  99   0   0   0|   0     0 |  30M  241k|   0     0 |2891  5049 
  1   1  98   0   0   0|   0     0 |9655k  242k|   0     0 |2186  4078 
  1   1  98   0   0   0|   0     0 |  20M  255k|   0     0 |2633  4852 
  1   1  98   0   0   0|   0  1638B|6976k  240k|   0     0 |2090  3940 
  2   1  98   0   0   0|   0  1946B|3190k  252k|   0     0 |2051  3932 
  1   1  98   0   0   0| 819B    0 |  33M  276k|   0     0 |3139  5701 
  1   1  98   0   0   0|   0     0 |  32M  269k|   0     0 |3093  5636 
  1   1  98   0   0   0|   0     0 |  20M  247k|   0     0 |2508  4732 
  0   1  99   0   0   0|   0     0 |  63M  317k|   0     0 |4439  8176 
  0   1  99   0   0   0|   0   819B|  51M  295k|   0     0 |3952  7233 
  1   1  98   0   0   0|   0  7680B|  19M  230k|   0     0 |2378  4340 
  0   1  99   0   0   0|   0     0 |  61M  315k|   0     0 |4386  8055 
  1   1  98   0   0   0|   0     0 |  13M  244k|   0     0 |2294  4334 
  2   1  98   0   0   0|   0     0 |2279k  229k|   0     0 |1824  3522 
  1   1  98   0   0   0|   0     0 |7135k  237k|   0     0 |2043  3912 
  1   1  98   0   0   0|   0  4096B|  31M  267k|   0     0 |3068  5634 
  1   1  98   0   0   0|   0    11k|  36M  266k|   0     0 |3259  5811 
  1   1  99   0   0   0|   0     0 |  42M  291k|   0     0 |3644  6495 
  1   1  99   0   0   0|   0     0 |  43M  275k|   0     0 |3622  6304 
  1   1  98   0   0   0|   0     0 |  41M  264k|   0     0 |3427  5981 
  1   1  98   0   0   0|   0     0 |  16M  222k|   0     0 |2229  4047 
  1   1  98   0   0   0|   0   819B|  24M  253k|   0     0 |2719  5062 
  1   1  98   0   0   0|   0  1946B|  18M  243k|   0     0 |2412  4521 
  1   1  99   0   0   0|   0     0 |  53M  304k|   0     0 |4045  7455 
  1   1  99   0   0   0|   0     0 |  48M  296k|   0     0 |3810  7043 
  1   1  98   0   0   0|   0     0 |  20M  233k|   0     0 |2407  4366 
  0   0 100   0   0   0|   0     0 |5224k   55k|   0     0 | 649  1147 
  0   0 100   0   0   0|   0   819B| 368B  367B|   0     0 |  63   121 
  0   0 100   0   0   0|   0  1946B|  12B   34B|   0     0 |  60   114 
  0   0 100   0   0   0|   0     0 |  49B   59B|   0     0 |  58   111 
  0   0 100   0   0   0|   0     0 |  12B   34B|   0     0 |  59   112 
-------------- next part --------------
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  1   0  99   0   0   0| 209k  125k|   0     0 |   0     0 |1814  3244 
  0   0 100   0   0   0|   0     0 |  12B    0 |   0     0 |  74   114 
  0   0 100   0   0   0|   0     0 | 256B  105B|   0     0 |  60   117 
  0   0 100   0   0   0|   0  2458B|   0     0 |   0     0 |  57   110 
  0   0 100   0   0   0|7373B   13M|  13M   56k|   0     0 |1491  2707 
  0   0  99   0   0   1|4096B   47M|  47M  177k|   0     0 |5002  9450 
  0   1  99   0   0   1|4915B   72M|  73M  271k|   0     0 |7663    14k
  0   1  99   0   0   1|3277B   80M|  81M  296k|   0     0 |8488    16k
  0   1  99   0   0   1|2458B   76M|  76M  279k|   0     0 |8245    16k
  0   1  99   0   0   1|3277B   76M|  77M  283k|   0     0 |8301    16k
  0   1  99   0   0   1|2458B   76M|  77M  283k|   0     0 |8291    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8299    16k
  0   1  99   0   0   1|2458B   77M|  78M  287k|   0     0 |8370    16k
  0   1  99   0   0   1|2458B   77M|  78M  283k|   0     0 |8358    16k
  0   1  99   0   0   1|2458B   77M|  78M  287k|   0     0 |8369    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8272    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8225    16k
  0   1  99   0   0   1|2458B   75M|  76M  276k|   0     0 |8110    15k
  0   1  99   0   0   1|2458B   75M|  76M  276k|   0     0 |8033    15k
  0   1  99   0   0   1|2458B   75M|  76M  278k|   0     0 |8113    16k
  0   1  99   0   0   1|1638B   75M|  76M  275k|   0     0 |8142    16k
  0   1  99   0   0   1|3277B   77M|  78M  286k|   0     0 |8285    16k
  0   1  99   0   0   1|2458B   77M|  78M  282k|   0     0 |8298    16k
  0   1  99   0   0   1|2458B   76M|  77M  281k|   0     0 |8213    16k
  0   1  99   0   0   1|1638B   75M|  76M  275k|   0     0 |8158    16k
  0   1  99   0   0   1|2458B   77M|  78M  286k|   0     0 |8344    16k
  0   0  99   0   0   1|2458B   77M|  78M  283k|   0     0 |8266    16k
  0   0  99   0   0   1|2458B   76M|  77M  281k|   0     0 |8249    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8267    16k
  0   1  99   0   0   1|1638B   74M|  75M  270k|   0     0 |8149    16k
  0   1  99   0   0   1|   0    71M|  72M  250k|   0     0 |7894    15k
  0   1  99   0   0   1|   0    69M|  70M  242k|   0     0 |7654    15k
  0   0  99   0   0   0|   0    45M|  46M  160k|   0     0 |5135  9739 
  0   0 100   0   0   0|   0    24M|  24M   86k|   0     0 |2756  5251 
  0   0 100   0   0   0|   0    22M|  23M   80k|   0     0 |2555  4842 
-------------- next part --------------
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  1   0  99   0   0   0| 183k 7671k|   0     0 |   0     0 |2394  4376 
  0   0 100   0   0   0|   0    51k|  52k  372B|   0     0 |  91   145 
  0   0 100   0   0   0|   0  3277B|2531B   61B|   0     0 |  65   124 
  0   0 100   0   0   0|4096B   17M|  17M   66k|   0     0 |1847  3416 
  0   0  99   0   0   0|4096B   41M|  41M  155k|   0     0 |4329  8185 
  0   1  99   0   0   1|4915B   75M|  76M  284k|   0     0 |7957    15k
  0   1  99   0   0   1|3277B   78M|  79M  292k|   0     0 |8334    16k
  0   1  99   0   0   1|2458B   77M|  77M  283k|   0     0 |8305    16k
  0   1  99   0   0   1|2458B   76M|  77M  281k|   0     0 |8284    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8272    16k
  0   1  99   0   0   1|2458B   75M|  76M  279k|   0     0 |8210    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8266    16k
  0   1  99   0   0   1|2458B   76M|  76M  280k|   0     0 |8188    16k
  0   1  99   0   0   1|2458B   76M|  77M  283k|   0     0 |8269    16k
  0   1  99   0   0   1|2458B   77M|  78M  284k|   0     0 |8295    16k
  0   1  99   0   0   1|2458B   76M|  77M  282k|   0     0 |8230    16k
  0   1  99   0   0   1|2458B   77M|  78M  284k|   0     0 |8316    16k
  0   1  98   0   0   1|2458B   76M|  77M  280k|   0     0 |8148    16k
  0   0  99   0   0   1|2458B   74M|  75M  273k|   0     0 |7951    15k
  0   1  99   0   0   1|1638B   75M|  76M  275k|   0     0 |8100    16k
  0   1  99   0   0   0|3277B   76M|  77M  285k|   0     0 |8234    16k
  0   1  99   0   0   1|2458B   75M|  76M  276k|   0     0 |8109    15k
  0   0  99   0   0   1|2458B   75M|  76M  279k|   0     0 |8132    16k
  0   1  99   0   0   1|1638B   75M|  76M  274k|   0     0 |8137    16k
  0   1  99   0   0   1|3277B   76M|  77M  285k|   0     0 |8239    16k
  0   0  99   0   0   1|1638B   75M|  76M  278k|   0     0 |8131    16k
  0   1  99   0   0   1|2458B   76M|  76M  279k|   0     0 |8172    16k
  0   0  99   0   0   1|2458B   76M|  77M  281k|   0     0 |8208    16k
  0   1  99   0   0   1|2458B   74M|  75M  272k|   0     0 |8083    15k
  0   0  99   0   0   1|   0    71M|  72M  252k|   0     0 |7931    15k
  0   1  99   0   0   1|   0    68M|  69M  240k|   0     0 |7598    15k
  0   0  99   0   0   0|   0    43M|  43M  150k|   0     0 |4780  9221 
  0   0  99   0   0   0|   0    34M|  35M  122k|   0     0 |3867  7377 
  0   0 100   0   0   0|   0    17M|  17M   59k|   0     0 |1912  3617 
  0   0 100   0   0   0|   0    18M|  18M   63k|   0     0 |2037  3873 
  0   0 100   0   0   0|   0  8810k|8917k   31k|   0     0 |1064  1954 
  0   0 100   0   0   0|   0  9222k|9329k   32k|   0     0 |1109  2006 
  0   0 100   0   0   0|   0  4510k|4564k   16k|   0     0 | 561  1044 
  0   0 100   0   0   0|   0  4916k|4976k   18k|   0     0 | 630  1150 
  0   0 100   0   0   0|  98k 2255k|2284k 8986B|   0     0 | 351   626 
  0   0 100   0   0   0|   0  2254k|2282k   11k|   0     0 | 323   577 
  0   0 100   0   0   0|   0  1231k|1246k 4998B|   0     0 | 202   364 
  0   0 100   0   0   0|   0  1239k|1245k 4956B|   0     0 | 197   364 
  0   0 100   0   0   0|   0   617k| 623k 2204B|   0     0 | 130   240 
  0   0 100   0   0   0|   0   617k| 624k 2214B|   0     0 | 134   242 
  0   0 100   0   0   0|   0   412k| 416k 2046B|   0     0 | 106   195 
  0   0 100   0   0   0|   0    19k|  20k  136B|   0     0 |  67   126 
  0   0 100   0   0   0|   0  2458B|2531B   61B|   0     0 |  66   127 
  0   0 100   0   0   0|   0  8602B|   0     0 |   0     0 |  60   115 
  0   0 100   0   0   0|   0   819B|   0     0 |   0     0 |  57   109 
  0   0 100   0   0   0|   0     0 |   0   567B|   0     0 |  57   110 
  0   0 100   0   0   0|   0     0 |  12B    8B|   0     0 |  57   109 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  58   110 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  56   107 
  0   0 100   0   0   0|   0  1946B|   0     0 |   0     0 |  58   112 
  0   0 100   0   0   0|   0  1638B|   0     0 |   0     0 |  60   113 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  59   112 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  57   109 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 |  57   109 
  0   0 100   0   0   0|   0     0 |   0     0 |   0     0 | 111   119 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mailman.mn-linux.org/pipermail/tclug-list/attachments/20120728/2750d8c9/attachment.pgp>