[Click] Bidirectional Performance Issues

[Eddie Kohler]

Hi Dietmar,

I would add to Beyers' comments just that when we measured Click's 
performance, we did not play with the BURST parameters.  I would leave those 
parameters out.  Our Queues had length 1024.  Full vs. half-duplex is a very 
good suggestion; make sure you check that.  What results do you get with no 
BURST parameters?

Eddie


Beyers Cronje wrote:
> Hi Dietmar,
> 
> A couple of recommendations. First off in Eddie's Click thesis there's an
> excellent overview on overload behaviour and will provide you with enough
> information to do your own analyses to determine if the problem lies with
> CPU, PCI Bus, Memory. See section 8.3 on page 98 of
> http://pdos.csail.mit.edu/papers/click:kohler-phd/thesis.pdf
> 
> Increasing the burst size might have the opposite effect than what you
> intended. Click runs in one thread and it's click's scheduler schedules task
> elements one at a time. Increasing the burst size could mean that it would
> take longer for that element to finish processing when scheduled as well as
> take longer to get scheduled again. So in short, increasing the burst size
> might not necessarily increase performance.
> 
> I would also increase the queue size. I think the default queue size is
> 1000. Monitor the queue element's drops and highwater_length handlers. Drops
> at the queue could indicate ToDevice is not scheduled enough, which could
> mean the NICs Tx queue is overloaded.  Dropping packets at Click's queue
> element is an expensive drop as cpu, pci and memory operations have already
> been done on the packet. It's less expensive to drop a packet at the NIC as
> no resources are wasted then.
> 
> Changing the e1000 driver's IntDelay wont make any difference when using
> PollDevice as irqs are disabled when polling is used. I would definately
> play around with the Rx and Tx descriptor sizes though.
> 
> Are the NIC links full-duplex or halfduplex?
> 
> Beyers
> 
> 
> 
> 
> On 7/28/06, Dietmar Tölle <dietmar.toelle at esk.fraunhofer.de> wrote:
>> Hello altogether,
>>
>>
>>
>> We're trying to set up a switch/router using Click with high performance,
>> but we have trouble with bidirectional traffic and the polling drivers.
>>
>>
>>
>> First of all: Our system
>>
>> Pentium D830 + 2GB RAM
>>
>> Mainboard with Intel 975X Chipset and 2 PCI-X interfaces
>>
>> 2 Intel Pro/1000 MT Dual Port Server Cards for PCI-X (82546GB chipset)
>>
>>
>>
>> Click: Standard-version 1.5.0 from the Web
>>
>> E1000 driver 5.7, which is included in the package, compiled with
>> NAPI-option
>>
>> Patched linux kernel 2.6.16.13
>>
>>
>>
>> Click-script: Just a simple cross-connect across the 2 cards over the
>> PCI-X bus (one card in, the other card out, and vice versa)
>>
>> PollDevice(eth2, PROMISC true, BURST 32) -> Queue(100) -> ToDevice(eth5,
>> BURST 16);
>>
>> PollDevice(eth5, PROMISC true, BURST 32) -> Queue(100) -> ToDevice(eth2,
>> BURST 16);
>>
>>
>>
>> Using this script the performance is as follows:
>>
>>
>>
>> Unidirectional traffic:
>>
>> With max. packet size, we achieve about 1 gbps (75 kpps)
>>
>>
>>
>> Bidirectional traffic:
>>
>> With max. packet size, we achieve about 600 mbps (50 kpps) per direction
>>
>> With min. packet size, throughput drops to 15 mbps (30 kpps) per direction
>>
>>
>>
>> And the strangest fact of all: FromDevice performs EVEN BETTER than
>> PollDevice (about 5-10%)???
>>
>>
>>
>> We tried to tweak the parameter in the script (BURST) and the e1000
>> parameters (IntDelay, Descriptors, etc.), but it seems to have no effect at
>> all!
>>
>>
>>
>> Did anyone come across similar problems or can give us any
>> recommendations?
>>
>> Does this problem arise from the PCI-bus?
>>
>>
>>
>> Thanks in advance!
>>
>>
>>
>> Best regards,
>>
>> Dietmar
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> click mailing list
>> click at amsterdam.lcs.mit.edu
>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>