[Click] Large latency with RouteBricks setup

[George Porter]

Hi Mihai,

On Thu, Aug 25, 2011 at 10:16 AM, Mihai Dobrescu
<mihai.dobrescu at gmail.com> wrote:
> For the moment I don't have a proper setup to test the described
> behavior, but the problem seems to be related to the batch settings in
> the NIC driver.
>
> b1) An easy fix would be to try to change the the batch value from 16
> to 8 or 4 to see if the latency improves. Check the driver's source
> folder (ixgbe_main.c) and look for something like rxdctl |= (16<<16);
> txdctl |= (16<<16).

I tried adjusting this parameter to a few different settings.  It
worked with batching levels of 16 and 8.  For some reason when I tried
4 and 2 packets we're delivered out of the driver regardless of how
many I sent.

Looking at the latency, it seems that for small bursts of packets
(e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is
very low--about 14us or so.

When I try large packet bursts (e.g., 16,000 packets back to back)
there is an interesting queueing-type behavior in which the latency
for the earlier packets is 14us, and then it linearly climbs up to
~560us or so, and then jumps back down to 14 and repeats in a sawtooth
pattern.  This may be due to an RSS hash issue, given that the receive
queue stats show that most packets end up in a single queue (vs being
spread over the 4 queues that are instantiated in the nic).  I'm going
to have my packet generator randomize the source IP address to
hopefully spread those packets over those queues and retry.

Thanks again for your help,
George