[Click] 10GE Click performance

Wed Jan 25 17:35:13 EST 2012

This is very interesting and it would be great to update Click to use 
skb_linearize correctly.

Click elements assume that packets are linear -- i.e. that they have 
contiguous header and data; that all p->length() bytes of data are 
accessible starting from p->data(). In early versions if Linux ALL 
packets were linear. For some time, though, Linux has supported 
"nonlinear" skbs, where the packet data is fragmented into different 
memory areas.

It looks like your device is generating non-linear SKBs. Which kind of 
surprises me, I didn't know that many devices did so.

Click will fail badly, causing an out-of-bounds memory access, if an 
element tries to access data that's in some fragment other than the 
first. That is why we added the linearize.

Here are two possibilities.

1. The Myricom inherently generates non-linear skbs. We should add some 
configuration option to Click that let it handle non-linear skbs as well.

2. Myricom generates linear skbs, but some other part of the Linux data 
path is making the skb nonlinear, before the packet reaches Click. We 
should figure out whether this is happening and maybe adjust Click to 
handle the packets a different way.

Based on my reading of the code, the case is (1). Myricom generates 
fragmented skbs.

I wonder if you could print out what nskb->data_len is. It would also be 
great if you had some more time to investigate. Do you know what 
elements you plan to include in the data path?

Eddie

On 1/19/12 4:11 PM, rchertov wrote:
> If the following change is applied to Click pulled from today
> (c95065dc173af754774254e1356cc83acd20c94d), then the 1.8 performance
> levels return.
> My machine can now happily receive 400K pps, while before the change it
> could only receive at 266K pps.
>
>
> diff --git a/include/click/packet.hh b/include/click/packet.hh
> index bd3f461..005ab14 100644
> --- a/include/click/packet.hh
> +++ b/include/click/packet.hh
> @@ -1358,10 +1358,10 @@ Packet::make(struct sk_buff *skb)
>           atomic_dec(&skb->users);
>        }
>    # if HAVE_SKB_LINEARIZE
> -    if (nskb&&  skb_linearize(nskb) != 0) {
> +    /*    if (nskb&&  skb_linearize(nskb) != 0) {
>           kfree_skb(nskb);
>           nskb = 0;
> -    }
> +       }*/
>    # endif
>        return reinterpret_cast<Packet *>(nskb);
>    }
>
> Roman
>
> On 01.19.2012 12:54, rchertov wrote:
>> Eddie,
>>
>> The degradation happens after
>> ee66e4894316dd99579e3a42696ea41c4eadd646
>> commit.  I am trying to determine which actual change in the commit
>> causes an issue.
>>
>> Roman
>>
>> On 01.18.2012 18:05, Eddie Kohler wrote:
>>> Hi Roman,
>>>
>>> There weren't many changes in FromDevice in the 1.8-2.0 time frame.
>>> git log will show you what they were. The major change, if I
>>> remember
>>> right, was using a new method of stealing packets from Linux for
>>> patchless purposes.
>>>
>>> How easy is it for you to run these tests? Have you ever used `git
>>> bisect`?
>>>
>>> Eddie
>>>
>>>
>>> On 01/18/2012 08:34 PM, rchertov wrote:
>>>> Some more interesting tidbits of data.  If I run release 1.8, I get
>>>> the
>>>> speed I need.  However, after release 2.0, the problem starts to
>>>> manifest itself.
>>>>
>>>> Roman
>>>>
>>>> On 01.18.2012 15:05, rchertov wrote:
>>>>> Ok, I got some numbers finally.
>>>>> I have two dual socket quad core Intel(R) Xeon(R) CPU E5450  @
>>>>> 3.00GHz
>>>>> boxes with Myricom 10GE cards
>>>>> using SR transceivers running patched linux 2.6.24.7.  The myricom
>>>>> driver is the one included in the 2.6.24.7 kernel.
>>>>>
>>>>> Topology:
>>>>> Src ->   Dst;
>>>>>
>>>>> Source config:
>>>>> src1 :: RatedSource(LENGTH 2112, RATE 400000, LIMIT 24000000)
>>>>>        ->   UDPIPEncap(SRC 10.1.1.4, SPORT 6667, DST 10.1.1.2, DPORT
>>>>> 6667)
>>>>>        ->   EtherEncap(0x0800, 00:60:DD:47:94:2A, 00:15:17:7C:E4:A1)
>>>>>        ->   ctr :: AverageCounter
>>>>>        ->   ToDevice(eth2, BURST 16);
>>>>>
>>>>> Destination config:
>>>>>    FromDevice(eth2, PROMISC true, BURST 512) ->   ctr ::
>>>>> AverageCounter
>>>>> ->
>>>>> Discard;
>>>>>
>>>>>
>>>>> Source 2.0.1 Destination 2.0.1 (its actually from yesterday's git
>>>>> pull)
>>>>> Src rate: 204364 count 24000000
>>>>> Dst rate: 204364 count 23977282
>>>>>
>>>>> Source 1.7.0rc1 Destination 2.0.1
>>>>> Src rate: 400000 count 24000000
>>>>> Dst rate: 264502 count 15869088
>>>>>
>>>>> Source 1.7.0rc1 Destination 1.7.0rc1
>>>>> Src rate: 400000 count 24000000
>>>>> Dst rate: 400000 count 23999893
>>>>>
>>>>> Source 2.0.1 Destination 1.7.0rc1
>>>>> Src rate: 400000 count 24000000
>>>>> Dst rate: 400000 count 23999750
>>>>>
>>>>> So based on the runs, it appears the issue is with packet
>>>>> reception
>>>>> in
>>>>> the current version of Click compared to 1.7.0rc1.  It is also not
>>>>> clear
>>>>> to me as to why the sending rate is reduced when both machines run
>>>>> 2.0.1
>>>>> click.
>>>>>
>>>>> Roman
>>>>>
>>>>> On 01.18.2012 08:02, Eddie Kohler wrote:
>>>>>> Well, shoot. This performance degradation is unfortunate.
>>>>>>
>>>>>> There are two questions:
>>>>>> 1. Why is performance low (~300Kpps or less)?
>>>>>> 2. Why has performance gotten worse (300->200)?
>>>>>>
>>>>>> Let's attack #2 first.
>>>>>>
>>>>>> Has a non-Device configuration, such as InfiniteSource->Idle,
>>>>>> slowed
>>>>>> down?
>>>>>>
>>>>>> Eddie
>>>>>>
>>>>> _______________________________________________
>>>>> click mailing list
>>>>> click at amsterdam.lcs.mit.edu
>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>>>
>>>> _______________________________________________
>>>> click mailing list
>>>> click at amsterdam.lcs.mit.edu
>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>
>> _______________________________________________
>> click mailing list
>> click at amsterdam.lcs.mit.edu
>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click