Fw: [Click] performance experiment problems - low forwarding rates

Thu May 27 17:51:30 EDT 2004

With respect to the low forwarding rate problem we are experiencing,
we are now in the following situation: the source and sink are in good
shape; the source sends everything we ask for (up to 400Kpps), and the
sink receives everything the router sent.

The problem lies now exclusively with the router. We have been looking
at the router NICs' statistics, and it seems that there are no FIFO
drops, but there are a lot of Queue drops as well as ToDevice drops
(reminder: we are running the Simple configuration in which the router
just forwards the packet with no processing whatsoever). Queue drops
indicate CPU limitations if I am not mistaken: packets are pushed onto
the Queue faster than ToDevice can pull them. ToDevice drops occur
when the NIC transmit queue is full.

Also, during the operation of the router, the message "polling!
icr=0xfoo" shows up. It is printed from within the NIC driver's
interrupt handler with foo always being 1, apart from the last time it
is printed, when it is 3. Aren't interrupts supposed to be disabled?!
(we are using Click's polling extensions with compatible NICs)

Looking through the driver code, we figured out that 1 means "Transmit
desc written back" and 3 means the former plus "Transmit queue empty".
I get the feeling an interrupt is delivered when we are out of DMA
descriptors and also when the transmit NIC queue is finally empty so
that we can transmit some more; this is a very sloppy explanation
though; had anyone experienced this before? Maybe the person who
inserted that diagnostic message can help us out! :-)

One last thing: we increased the available DMA descriptors from 256 to
1024, but there was no improvement. We are really getting down low in
the driver now, so it would be great if someone could suggest out of
experience some concrete things to try...

Thanks for any help :-)

nikitas

----- Original Message ----- 
From: "Giorgio Calarco" <gcalarco at deis.unibo.it>
To: "Nikitas Liogkas" <nikitas at CS.UCLA.EDU>
Cc: <click at amsterdam.lcs.mit.edu>
Sent: Tuesday, May 18, 2004 12:07 am
Subject: Re: [Click] performance experiment problems - low forwarding
rates

> Nikitas,
>
> it should be quite difficult to experience PCI bus performance
limitations
> if you use 64-byte packets, this should happen only when
> you increase the packet lenght.
> This is particularly true with your sources PCs, keep in mind that
> you have around 4 Gb/s of bus bandwidth
> (which is lower than the the nic bandwidth...1Gb/s).
>
> However, some of your values seem quite resonable:
> with a 800MHz pentium + PCI1.0 bus as a source we could
> generate around 170-180kpps...thus, if you can
> reach 280kpps with your Athlon you should consider that
> quite normal (if you want to speed up, actually we are reaching
> more than 700kpps with a 3GHz CPU + PCIX bus ... +
> FastUDPSource ).
>
> The proc filesystem also exports some interesting handlers
> for monitoring the PRO1000 activity (take a look inside the
> /proc/net/PRO_LAN_Adapters directory), like FIFO errors, this could
help
> you understanding if you are losing packets inside the NIC
> FIFO. If so, the NIC is not receiving DMA descriptors
> fast enough (this means that you have CPU-memory
> limitations)
>
> The router:
> with similar hardware (1.6GHz pentium IV+ PCI 1.0 bus) we could
> forward around 350-370Kpps (with a RFC1812 router config,
> thus having more packet processing than you... but we
> were using a bit faster CPU).
> I would also try using the PRO1000XT or MT Server
> Adapters instead of the dual port to see if you can go beyond
> (even if they have a PCI-X interface, you can also plug them
> on a standard PCI bus).
>
>
> ciao
> giorgio
>
>
> PS: if you have full text access to the LNCS on-line library,
> our past testbed + results was better described here:
>
> Volume 2720 / 2003
> Title:  High-Speed Networks and Multimedia Communications:
> 6th IEEE International Conference, HSNMC 2003, Estoril,
> Portugal, July 23-25, 2003. Proceedings
>
>
> ----- Original Message ----- 
> From: "Nikitas Liogkas" <nikitas at CS.UCLA.EDU>
> To: <click at amsterdam.lcs.mit.edu>
> Sent: Tuesday, May 18, 2004 6:48 AM
> Subject: [Click] performance experiment problems - low forwarding
rates
>
>
> > Hi to all!
> >
> > We've been trying to set up a testbed here at UCLA, in order to
run
> > Click and measure some optimizations we have implemented. So,
> > following the example set forth by the TOCS and ASPLOS paper, we
> > have two machines serving as sources, two serving as sinks, and
one
> > router that is receiving packets from the sources and is supposed
> > to deliver them to the sinks (machine configurations are at the
> > end of this email). The sources and sinks are connected with
gigabit
> > ethernet point-to-point links to the router, and we are running
the
> > latest CVS version of Click. Here are the machine configurations:
> >
> > - sources and sinks: AMD Athlon 1,666 MHz with Intel PRO/1000
> > Desktop Adapter gigabit ethernet cards (with Click's polling
> > extensions) on a PCI 64bit/66MHz bus running Linux 2.4.21
> > (patched with Click's patch)
> >
> > - router: Pentium III 1,266 MHz with two Intel PRO/1000 Dual
> > Port Server Adapter gigabit ethernet cards (with Click's
> > polling extensions) on a single PCI 32bit/33MHz bus
> > running Linux 2.4.21 (patched with Click's patch)
> >
> > The problem is that we are experiencing very low forwarding rates
> > (when I say packets below, I mean 64-byte packets).
> > For example, running the Simple configuration, which just forwards
> > packets without any processing, we can get up to 240,000 pkts/sec,
> > while the TOCS paper reports 452,000 pkts/sec for a similar
machine
> > to ours. The polling extensions for the gigabit cards seem to be
working
> (we
> > get a e1000_poll_on chatter when the router starts up).
> >
> > Another (maybe related) problem is that the sources sending rate
> > saturates around 280,000 pkts/sec. If we go beyond that, we start
> > experiencing drops on the ToDevice which is sending out the
packets
> > (PCI bus limitation?) Surprisingly, in the ASPLOS paper
> > it is mentioned that the sources in their experiments, which were
arguably
> > weaker than our sources, can generate 1 million pkts/sec each!
> >
> > Has anyone else experienced something similar?
> > Any ideas/suggestions will be greatly appreciated :-)
> >
> > Nikitas Liogkas
> > University of California - Los Angeles
> > _______________________________________________
> > click mailing list
> > click at amsterdam.lcs.mit.edu
> > https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>