[Click] Bidirectional Performance Issues

[Beyers Cronje]

Hi Dietmar,

A couple of recommendations. First off in Eddie's Click thesis there's an
excellent overview on overload behaviour and will provide you with enough
information to do your own analyses to determine if the problem lies with
CPU, PCI Bus, Memory. See section 8.3 on page 98 of
http://pdos.csail.mit.edu/papers/click:kohler-phd/thesis.pdf

Increasing the burst size might have the opposite effect than what you
intended. Click runs in one thread and it's click's scheduler schedules task
elements one at a time. Increasing the burst size could mean that it would
take longer for that element to finish processing when scheduled as well as
take longer to get scheduled again. So in short, increasing the burst size
might not necessarily increase performance.

I would also increase the queue size. I think the default queue size is
1000. Monitor the queue element's drops and highwater_length handlers. Drops
at the queue could indicate ToDevice is not scheduled enough, which could
mean the NICs Tx queue is overloaded.  Dropping packets at Click's queue
element is an expensive drop as cpu, pci and memory operations have already
been done on the packet. It's less expensive to drop a packet at the NIC as
no resources are wasted then.

Changing the e1000 driver's IntDelay wont make any difference when using
PollDevice as irqs are disabled when polling is used. I would definately
play around with the Rx and Tx descriptor sizes though.

Are the NIC links full-duplex or halfduplex?

Beyers




On 7/28/06, Dietmar Tölle <dietmar.toelle at esk.fraunhofer.de> wrote:
>
> Hello altogether,
>
>
>
> We're trying to set up a switch/router using Click with high performance,
> but we have trouble with bidirectional traffic and the polling drivers.
>
>
>
> First of all: Our system
>
> Pentium D830 + 2GB RAM
>
> Mainboard with Intel 975X Chipset and 2 PCI-X interfaces
>
> 2 Intel Pro/1000 MT Dual Port Server Cards for PCI-X (82546GB chipset)
>
>
>
> Click: Standard-version 1.5.0 from the Web
>
> E1000 driver 5.7, which is included in the package, compiled with
> NAPI-option
>
> Patched linux kernel 2.6.16.13
>
>
>
> Click-script: Just a simple cross-connect across the 2 cards over the
> PCI-X bus (one card in, the other card out, and vice versa)
>
> PollDevice(eth2, PROMISC true, BURST 32) -> Queue(100) -> ToDevice(eth5,
> BURST 16);
>
> PollDevice(eth5, PROMISC true, BURST 32) -> Queue(100) -> ToDevice(eth2,
> BURST 16);
>
>
>
> Using this script the performance is as follows:
>
>
>
> Unidirectional traffic:
>
> With max. packet size, we achieve about 1 gbps (75 kpps)
>
>
>
> Bidirectional traffic:
>
> With max. packet size, we achieve about 600 mbps (50 kpps) per direction
>
> With min. packet size, throughput drops to 15 mbps (30 kpps) per direction
>
>
>
> And the strangest fact of all: FromDevice performs EVEN BETTER than
> PollDevice (about 5-10%)???
>
>
>
> We tried to tweak the parameter in the script (BURST) and the e1000
> parameters (IntDelay, Descriptors, etc.), but it seems to have no effect at
> all!
>
>
>
> Did anyone come across similar problems or can give us any
> recommendations?
>
> Does this problem arise from the PCI-bus?
>
>
>
> Thanks in advance!
>
>
>
> Best regards,
>
> Dietmar
>
>
>
>
>
>
>
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>