[Click] Click on Multi-core problem

[Giorgio Calarco]

Dear Ahmed,

after reading your performance test I asked myself these two questions,
which I hope can be helpful for explaining this odd behaviour:

- what is your CPU internal design ? Are the 4 cores using
a separate, unique L2 cache each (each core has got its own L2 cache)
or are they coupled ? I.e., core (0,1) use a shared L2 cache
(and the same for (2,3)) ?

- the second question I propose to you is : have you disabled Hyperthreading ?
That is, are the 4 cores virtual or physical ?

Ciao, Giorgio

On Tue, Jul 12, 2011 at 4:53 PM, ahmed A. <amego83 at gmail.com> wrote:
> Hi,
>
> I am examining the forwarding performance of CLICK in our four core CPU
> machine. I am assigning two different simple forwarding path to
> two different core each time and watch the forwarding rate using a CLICK
> counter. My CLICK configuration file is as follow :
>
> d1::PollDevice(eth24,PROMISC true,BURST 16) ->  queue1::Queue(10000) ->
> c1::Counter -> td1::ToDevice(eth22);
> pd2::PollDevice(eth25,PROMISC true,BURST 16) -> queue2::Queue(10000) ->
> c2::Counter -> td2::ToDevice(eth23);
>
>        Idle -> ToDevice(eth24);
>        Idle -> ToDevice(eth25);
>
>
>        StaticThreadSched(pd1 0,td1 0,pd2 1,td2 1);
>
>        CpuPin(0 0,1 1);
>
> I was expecting to have almost the same forwarding rate (counter rate) for
> both paths whatever was the assigned cores, but actually I got different
> results
> depending on the cores that I use, for example when I use core 0 and 1, I
> got *1.0 Million Packets Per Second MPPS* for core 0 and *1.42 MPPS* for
> core 1. for core 1 and 2
> I got *1.39 MPPS* for both cores. and for core 2 and 3 I got *1.42 MPPS* for
> core 2 and core *1 MPPS* for core 2. In summary, there  alway are two cores
> with bad performance comparing to the
> other cores.
>
> By checking the *monitored_empty_polls_rate* of the cores, I found out that
> the cores with bad performance has 0.781 monitored_empty_polls_rate whereas
> the good
> cores has 207111 *monitored_empty_polls_rate*. The number of dropped packets
> in the NIC port assigned to the bad cores are much bigger than number of
> dropped packets assigned to the good cores. My explanation is the Polldevice
> is not
> getting enough CPU cycles (i.e not scheduled enough) to poll packets and
> refill the DMA ring with skb buffers, but I have no idea why ????
>
> Does the Linux scheduler interfere with  click ? I checked the load at each
> core using *top* but I could not see any other processes running on the bad
> cores, they are idle all the time.
> I would appreciate any tips or help.
>
> thank you in advance
>
> Ahmed
>
> PS: I have 2.6.18 kernel running in fedora filesystem with CLICK 1.6 and
> e1000 batched driver
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>