[Click] Click on Multi-core problem

[ahmed A.]

Hi,

I am examining the forwarding performance of CLICK in our four core CPU
machine. I am assigning two different simple forwarding path to
two different core each time and watch the forwarding rate using a CLICK
counter. My CLICK configuration file is as follow :

d1::PollDevice(eth24,PROMISC true,BURST 16) ->  queue1::Queue(10000) ->
c1::Counter -> td1::ToDevice(eth22);
pd2::PollDevice(eth25,PROMISC true,BURST 16) -> queue2::Queue(10000) ->
c2::Counter -> td2::ToDevice(eth23);

        Idle -> ToDevice(eth24);
        Idle -> ToDevice(eth25);


        StaticThreadSched(pd1 0,td1 0,pd2 1,td2 1);

        CpuPin(0 0,1 1);

I was expecting to have almost the same forwarding rate (counter rate) for
both paths whatever was the assigned cores, but actually I got different
results
depending on the cores that I use, for example when I use core 0 and 1, I
got *1.0 Million Packets Per Second MPPS* for core 0 and *1.42 MPPS* for
core 1. for core 1 and 2
I got *1.39 MPPS* for both cores. and for core 2 and 3 I got *1.42 MPPS* for
core 2 and core *1 MPPS* for core 2. In summary, there  alway are two cores
with bad performance comparing to the
other cores.

By checking the *monitored_empty_polls_rate* of the cores, I found out that
the cores with bad performance has 0.781 monitored_empty_polls_rate whereas
the good
cores has 207111 *monitored_empty_polls_rate*. The number of dropped packets
in the NIC port assigned to the bad cores are much bigger than number of
dropped packets assigned to the good cores. My explanation is the Polldevice
is not
getting enough CPU cycles (i.e not scheduled enough) to poll packets and
refill the DMA ring with skb buffers, but I have no idea why ????

Does the Linux scheduler interfere with  click ? I checked the load at each
core using *top* but I could not see any other processes running on the bad
cores, they are idle all the time.
I would appreciate any tips or help.

thank you in advance

Ahmed

PS: I have 2.6.18 kernel running in fedora filesystem with CLICK 1.6 and
e1000 batched driver