[Click] Large latency with RouteBricks setup

[Mihai Dobrescu]

On Thu, Aug 25, 2011 at 10:43 PM, George Porter <gmporter at cs.ucsd.edu> wrote:
> Hi Mihai,
>
> On Thu, Aug 25, 2011 at 10:16 AM, Mihai Dobrescu
> <mihai.dobrescu at gmail.com> wrote:
>> For the moment I don't have a proper setup to test the described
>> behavior, but the problem seems to be related to the batch settings in
>> the NIC driver.
>>
>> b1) An easy fix would be to try to change the the batch value from 16
>> to 8 or 4 to see if the latency improves. Check the driver's source
>> folder (ixgbe_main.c) and look for something like rxdctl |= (16<<16);
>> txdctl |= (16<<16).
>
> I tried adjusting this parameter to a few different settings.  It
> worked with batching levels of 16 and 8.  For some reason when I tried
> 4 and 2 packets we're delivered out of the driver regardless of how
> many I sent.
>
> Looking at the latency, it seems that for small bursts of packets
> (e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is
> very low--about 14us or so.

That's good news :)
>
> When I try large packet bursts (e.g., 16,000 packets back to back)
> there is an interesting queueing-type behavior in which the latency
> for the earlier packets is 14us, and then it linearly climbs up to
> ~560us or so, and then jumps back down to 14 and repeats in a sawtooth
> pattern.

There are 2 forms of batching: NIC batching and Click batching. The
described behavior might be related to Click batching. You could check
the "BURST" parameters in PollDevice and ToDevice to adjust the Click
batching.

> This may be due to an RSS hash issue, given that the receive
> queue stats show that most packets end up in a single queue (vs being
> spread over the 4 queues that are instantiated in the nic).  I'm going
> to have my packet generator randomize the source IP address to
> hopefully spread those packets over those queues and retry.
>
The RSS delivery to a particular queue is done based on the
(SrcIP,DstIP,SrcPort,DstPort) tuple. If you send packets using the
same headers, they will all end up in the same queue.

Mihai