[Click] performance experiment problems - low forwarding rates

[Giorgio Calarco]

Nikitas,

it should be quite difficult to experience PCI bus performance limitations
if you use 64-byte packets, this should happen only when
you increase the packet lenght.
This is particularly true with your sources PCs, keep in mind that
you have around 4 Gb/s of bus bandwidth
(which is lower than the the nic bandwidth...1Gb/s).

However, some of your values seem quite resonable:
with a 800MHz pentium + PCI1.0 bus as a source we could
generate around 170-180kpps...thus, if you can
reach 280kpps with your Athlon you should consider that
quite normal (if you want to speed up, actually we are reaching
more than 700kpps with a 3GHz CPU + PCIX bus ... +
FastUDPSource ).

The proc filesystem also exports some interesting handlers
for monitoring the PRO1000 activity (take a look inside the
/proc/net/PRO_LAN_Adapters directory), like FIFO errors, this could help
you understanding if you are losing packets inside the NIC
FIFO. If so, the NIC is not receiving DMA descriptors
fast enough (this means that you have CPU-memory
limitations)

The router:
with similar hardware (1.6GHz pentium IV+ PCI 1.0 bus) we could
forward around 350-370Kpps (with a RFC1812 router config,
thus having more packet processing than you... but we
were using a bit faster CPU).
I would also try using the PRO1000XT or MT Server
Adapters instead of the dual port to see if you can go beyond
(even if they have a PCI-X interface, you can also plug them
on a standard PCI bus).


ciao
giorgio


PS: if you have full text access to the LNCS on-line library,
our past testbed + results was better described here:

Volume 2720 / 2003
Title:  High-Speed Networks and Multimedia Communications:
6th IEEE International Conference, HSNMC 2003, Estoril,
Portugal, July 23-25, 2003. Proceedings


----- Original Message ----- 
From: "Nikitas Liogkas" <nikitas at CS.UCLA.EDU>
To: <click at amsterdam.lcs.mit.edu>
Sent: Tuesday, May 18, 2004 6:48 AM
Subject: [Click] performance experiment problems - low forwarding rates


> Hi to all!
>
> We've been trying to set up a testbed here at UCLA, in order to run
> Click and measure some optimizations we have implemented. So,
> following the example set forth by the TOCS and ASPLOS paper, we
> have two machines serving as sources, two serving as sinks, and one
> router that is receiving packets from the sources and is supposed
> to deliver them to the sinks (machine configurations are at the
> end of this email). The sources and sinks are connected with gigabit
> ethernet point-to-point links to the router, and we are running the
> latest CVS version of Click. Here are the machine configurations:
>
> - sources and sinks: AMD Athlon 1,666 MHz with Intel PRO/1000
> Desktop Adapter gigabit ethernet cards (with Click's polling
> extensions) on a PCI 64bit/66MHz bus running Linux 2.4.21
> (patched with Click's patch)
>
> - router: Pentium III 1,266 MHz with two Intel PRO/1000 Dual
> Port Server Adapter gigabit ethernet cards (with Click's
> polling extensions) on a single PCI 32bit/33MHz bus
> running Linux 2.4.21 (patched with Click's patch)
>
> The problem is that we are experiencing very low forwarding rates
> (when I say packets below, I mean 64-byte packets).
> For example, running the Simple configuration, which just forwards
> packets without any processing, we can get up to 240,000 pkts/sec,
> while the TOCS paper reports 452,000 pkts/sec for a similar machine
> to ours. The polling extensions for the gigabit cards seem to be working
(we
> get a e1000_poll_on chatter when the router starts up).
>
> Another (maybe related) problem is that the sources sending rate
> saturates around 280,000 pkts/sec. If we go beyond that, we start
> experiencing drops on the ToDevice which is sending out the packets
> (PCI bus limitation?) Surprisingly, in the ASPLOS paper
> it is mentioned that the sources in their experiments, which were arguably
> weaker than our sources, can generate 1 million pkts/sec each!
>
> Has anyone else experienced something similar?
> Any ideas/suggestions will be greatly appreciated :-)
>
> Nikitas Liogkas
> University of California - Los Angeles
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click