[Click] Large latency with RouteBricks setup

[George Porter]

Hi Mihai,

I've attached some more latency results from RouteBricks.  I was able
to vary the 5-tuple on the generated packets, and have latency
measurements for RouteBricks under a few different circumstances
(figure is attached):
 - ixgbe configured with a batch size of 16
 - ixgbe configured with a batch size of 4
 - Click/minfwdtest.click with a BURST size of 16
 - Click/minfwdtest.click with a BURST size of 4 (I also tried BURST
size of 1 but the latency was really high so I didn't include here)

The latency ranges from about 14us up to 100-120us in the 99th
percentile, for an ixgbe batch factor of 4.  Setting the batch setting
lower than 4 for some reason caused packets to no longer be delivered
to Click.  I'm still looking into that.

I also noticed that when I didn't randomize the 5-tuple (indicated by
the 'uniform' lines) the latency was much lower.  Does anyone on this
list know why the latency would be higher with a randomized 5-tupe?  I
would assume that it has to do with packets being spread across four
NIC queues instead of one queue, leading to those packets being
delivered by ixgbe up to Click 1/4th as often, leading to 4x the
latency.

This represents the lowest latency I've been able to achieve so far.
If you have any other suggestions let me know.  Thanks again for all
your help--it has been great getting RouteBricks up and running.

-George

On Fri, Aug 26, 2011 at 8:21 AM, George Porter <gmporter at cs.ucsd.edu> wrote:
> Thanks Mihai,
>
>>> Looking at the latency, it seems that for small bursts of packets
>>> (e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is
>>> very low--about 14us or so.
>>
>> That's good news :)
>
> Yep.  I believe that this single packet or small groups of packet
> latency results show the benefits of a polling 10G driver (since none
> of the other RouteBricks-specific functionality is being engaged, such
> as the multi-queue support and/or the distributed overlay forwarding
> network).  Adding NAPI support to the ixgbe driver would be really
> great to prevent packets from getting "stuck" in the ixgbe driver
> (which prevents things like ssh from working since the SYN packet
> isn't forwarded to the destination.
>
>>> When I try large packet bursts (e.g., 16,000 packets back to back)
>>> there is an interesting queueing-type behavior in which the latency
>>> for the earlier packets is 14us, and then it linearly climbs up to
>>> ~560us or so, and then jumps back down to 14 and repeats in a sawtooth
>>> pattern.
>>
>> There are 2 forms of batching: NIC batching and Click batching. The
>> described behavior might be related to Click batching. You could check
>> the "BURST" parameters in PollDevice and ToDevice to adjust the Click
>> batching.
>
> I will try that today.  However the sawtooth is over 10,000+ packets,
> and so I don't think it is related to a small constant amount of
> batching.  Otherwise I would expect latency of packets to go up for
> 8-16 packets then go back down.  What I see now is that the first
> 10-20 packets have latencies in the 12-14us range, then it linearly
> goes up to ~500-1000us over the next 10,000 packets, then there is a
> discontinuous jump back to 14us and the process repeats itself.  I
> think maybe the issue is that only one kernel thread is handling the
> packets at this time.
>
>> The RSS delivery to a particular queue is done based on the
>> (SrcIP,DstIP,SrcPort,DstPort) tuple. If you send packets using the
>> same headers, they will all end up in the same queue.
>
> I'm going to vary the 5-tuple and try again to see if I can spread the
> packets across the nic queues, which  should uniformly spread the
> packets across those nic queues.
>
> Thanks,
> George
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: latencyComparisonRouteBricks.png
Type: image/png
Size: 7907 bytes
Desc: not available
Url : http://amsterdam.lcs.mit.edu/pipermail/click/attachments/20110829/ff200e7d/attachment.png