Fw: [Click] performance experiment problems - low forwarding
rates
Andy Van Maele
andy.vanmaele at intec.ugent.be
Fri May 28 18:53:23 EDT 2004
Hello Nikitas,
I had a similar problem with the polling of the Intel PRO/1000 Dual Port
Server Adapter. As far as I understood the problem, it comes down to the
fact that the Dual Port Server shares one of his interupts - which he
gets from the BIOS - with another device - e.g. another network card.
Let me illustrate the problem with. On my PC, I have 2 Intel Pro/1000
Dual NICs, and 2 other NICs that don't support polling. The 2x2 polling
capable ethernet devices are eth1 to eth4, the two other are eth5 and
eth6 (no polling possible, only interrupt driven).
Without interfering in the BIOS, the following interrupts are assigned
to the ethernet devices (info from /proc/interrupts ):
5: 17557 XT-PIC eth3
10: 387 XT-PIC eth2, eth6
11: 44 XT-PIC eth5
12: 18238 XT-PIC eth1, eth4
As you can see, eth2 shares the interrupt with eth6. When all ethernet
devices are working in interrupt mode, both drivers are using this
interrupt to get their packets out.
If, on the other hand, eth2 is used in polling mode, it doesn't use
interrupts anymore to process his packets. So you would expect that the
following lines of code from
e1000_main.c (starting line1909)
static void
e1000_intr(int irq, void *data, struct pt_regs *regs)
{
struct net_device *netdev = data;
struct e1000_adapter *adapter = netdev->priv;
uint32_t icr;
int i = E1000_MAX_INTR;
while(i && (icr = E1000_READ_REG(&adapter->hw, ICR))) {
if(icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) {
adapter->hw.get_link_status = 1;
mod_timer(&adapter->watchdog_timer, jiffies);
}
if(netdev->polling){
printk("%s: polling! icr=0x%x\n", netdev->name, icr);
} else {
e1000_clean_rx_irq(adapter);
e1000_clean_tx_irq(adapter);
}
would not be executed as you would expect that there are no interrupts.
But if eth6 is recieving an interrupt - when it needs to process data -
then the lines of code are activated by interrupt 10 in my case. So the
interrupt-driver of eth6 is causing the execution of the line of code.
Basically, you can see that the only thing that happens is that the
e1000 driver is writing to the stdout that he recieved an interrupt
while he is in polling mode, and that he cannot do anything else with
this interrupt than telling you he has an interrupt.
As eth2 was the only interface that had a lot of packet loss, I decided
to try and outcomment the line of code that report the "polling!
icr=...". So if the e1000 driver did recieve an interrupt signal while
being in polling mode, he does nothing anymore. This has brought the
performance of the NIC back to normal values.
Apparently telling you recieve an intterrupt while working in polling
mode, isn't a nice thing to do...
I hope my experiences can help you to solve your problem
yours,
Andy Van Maele
Nikitas Liogkas wrote:
>With respect to the low forwarding rate problem we are experiencing,
>we are now in the following situation: the source and sink are in good
>shape; the source sends everything we ask for (up to 400Kpps), and the
>sink receives everything the router sent.
>
>The problem lies now exclusively with the router. We have been looking
>at the router NICs' statistics, and it seems that there are no FIFO
>drops, but there are a lot of Queue drops as well as ToDevice drops
>(reminder: we are running the Simple configuration in which the router
>just forwards the packet with no processing whatsoever). Queue drops
>indicate CPU limitations if I am not mistaken: packets are pushed onto
>the Queue faster than ToDevice can pull them. ToDevice drops occur
>when the NIC transmit queue is full.
>
>Also, during the operation of the router, the message "polling!
>icr=0xfoo" shows up. It is printed from within the NIC driver's
>interrupt handler with foo always being 1, apart from the last time it
>is printed, when it is 3. Aren't interrupts supposed to be disabled?!
>(we are using Click's polling extensions with compatible NICs)
>
>Looking through the driver code, we figured out that 1 means "Transmit
>desc written back" and 3 means the former plus "Transmit queue empty".
>I get the feeling an interrupt is delivered when we are out of DMA
>descriptors and also when the transmit NIC queue is finally empty so
>that we can transmit some more; this is a very sloppy explanation
>though; had anyone experienced this before? Maybe the person who
>inserted that diagnostic message can help us out! :-)
>
>One last thing: we increased the available DMA descriptors from 256 to
>1024, but there was no improvement. We are really getting down low in
>the driver now, so it would be great if someone could suggest out of
>experience some concrete things to try...
>
>Thanks for any help :-)
>
>nikitas
>
>
>----- Original Message -----
>From: "Giorgio Calarco" <gcalarco at deis.unibo.it>
>To: "Nikitas Liogkas" <nikitas at CS.UCLA.EDU>
>Cc: <click at amsterdam.lcs.mit.edu>
>Sent: Tuesday, May 18, 2004 12:07 am
>Subject: Re: [Click] performance experiment problems - low forwarding
>rates
>
>
>
>
>>Nikitas,
>>
>>it should be quite difficult to experience PCI bus performance
>>
>>
>limitations
>
>
>>if you use 64-byte packets, this should happen only when
>>you increase the packet lenght.
>>This is particularly true with your sources PCs, keep in mind that
>>you have around 4 Gb/s of bus bandwidth
>>(which is lower than the the nic bandwidth...1Gb/s).
>>
>>However, some of your values seem quite resonable:
>>with a 800MHz pentium + PCI1.0 bus as a source we could
>>generate around 170-180kpps...thus, if you can
>>reach 280kpps with your Athlon you should consider that
>>quite normal (if you want to speed up, actually we are reaching
>>more than 700kpps with a 3GHz CPU + PCIX bus ... +
>>FastUDPSource ).
>>
>>The proc filesystem also exports some interesting handlers
>>for monitoring the PRO1000 activity (take a look inside the
>>/proc/net/PRO_LAN_Adapters directory), like FIFO errors, this could
>>
>>
>help
>
>
>>you understanding if you are losing packets inside the NIC
>>FIFO. If so, the NIC is not receiving DMA descriptors
>>fast enough (this means that you have CPU-memory
>>limitations)
>>
>>The router:
>>with similar hardware (1.6GHz pentium IV+ PCI 1.0 bus) we could
>>forward around 350-370Kpps (with a RFC1812 router config,
>>thus having more packet processing than you... but we
>>were using a bit faster CPU).
>>I would also try using the PRO1000XT or MT Server
>>Adapters instead of the dual port to see if you can go beyond
>>(even if they have a PCI-X interface, you can also plug them
>>on a standard PCI bus).
>>
>>
>>ciao
>>giorgio
>>
>>
>>PS: if you have full text access to the LNCS on-line library,
>>our past testbed + results was better described here:
>>
>>Volume 2720 / 2003
>>Title: High-Speed Networks and Multimedia Communications:
>>6th IEEE International Conference, HSNMC 2003, Estoril,
>>Portugal, July 23-25, 2003. Proceedings
>>
>>
>>----- Original Message -----
>>From: "Nikitas Liogkas" <nikitas at CS.UCLA.EDU>
>>To: <click at amsterdam.lcs.mit.edu>
>>Sent: Tuesday, May 18, 2004 6:48 AM
>>Subject: [Click] performance experiment problems - low forwarding
>>
>>
>rates
>
>
>>
>>
>>>Hi to all!
>>>
>>>We've been trying to set up a testbed here at UCLA, in order to
>>>
>>>
>run
>
>
>>>Click and measure some optimizations we have implemented. So,
>>>following the example set forth by the TOCS and ASPLOS paper, we
>>>have two machines serving as sources, two serving as sinks, and
>>>
>>>
>one
>
>
>>>router that is receiving packets from the sources and is supposed
>>>to deliver them to the sinks (machine configurations are at the
>>>end of this email). The sources and sinks are connected with
>>>
>>>
>gigabit
>
>
>>>ethernet point-to-point links to the router, and we are running
>>>
>>>
>the
>
>
>>>latest CVS version of Click. Here are the machine configurations:
>>>
>>>- sources and sinks: AMD Athlon 1,666 MHz with Intel PRO/1000
>>>Desktop Adapter gigabit ethernet cards (with Click's polling
>>>extensions) on a PCI 64bit/66MHz bus running Linux 2.4.21
>>>(patched with Click's patch)
>>>
>>>- router: Pentium III 1,266 MHz with two Intel PRO/1000 Dual
>>>Port Server Adapter gigabit ethernet cards (with Click's
>>>polling extensions) on a single PCI 32bit/33MHz bus
>>>running Linux 2.4.21 (patched with Click's patch)
>>>
>>>The problem is that we are experiencing very low forwarding rates
>>>(when I say packets below, I mean 64-byte packets).
>>>For example, running the Simple configuration, which just forwards
>>>packets without any processing, we can get up to 240,000 pkts/sec,
>>>while the TOCS paper reports 452,000 pkts/sec for a similar
>>>
>>>
>machine
>
>
>>>to ours. The polling extensions for the gigabit cards seem to be
>>>
>>>
>working
>
>
>>(we
>>
>>
>>>get a e1000_poll_on chatter when the router starts up).
>>>
>>>Another (maybe related) problem is that the sources sending rate
>>>saturates around 280,000 pkts/sec. If we go beyond that, we start
>>>experiencing drops on the ToDevice which is sending out the
>>>
>>>
>packets
>
>
>>>(PCI bus limitation?) Surprisingly, in the ASPLOS paper
>>>it is mentioned that the sources in their experiments, which were
>>>
>>>
>arguably
>
>
>>>weaker than our sources, can generate 1 million pkts/sec each!
>>>
>>>Has anyone else experienced something similar?
>>>Any ideas/suggestions will be greatly appreciated :-)
>>>
>>>Nikitas Liogkas
>>>University of California - Los Angeles
>>>_______________________________________________
>>>click mailing list
>>>click at amsterdam.lcs.mit.edu
>>>https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>>
>>>
>>_______________________________________________
>>click mailing list
>>click at amsterdam.lcs.mit.edu
>>https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>
>>
>>
>
>_______________________________________________
>click mailing list
>click at amsterdam.lcs.mit.edu
>https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>
>
--
****************************************************
Andy Van Maele
Department of Information Technology (INTEC),
Ghent University - IMEC,
Sint-Pietersnieuwstraat 41, B-9000 Gent, Belgium
tel.: +32-9264 9958
fax.: +32-9264 9960
e-mail : andy.vanmaele at intec.UGent.be
****************************************************
More information about the click
mailing list