[Click] Large latency with RouteBricks setup

[George Porter]

Thanks Mihai,

>> Looking at the latency, it seems that for small bursts of packets
>> (e.g., 8, 16, 32 and thereabouts) the latency through RouteBricks is
>> very low--about 14us or so.
>
> That's good news :)

Yep.  I believe that this single packet or small groups of packet
latency results show the benefits of a polling 10G driver (since none
of the other RouteBricks-specific functionality is being engaged, such
as the multi-queue support and/or the distributed overlay forwarding
network).  Adding NAPI support to the ixgbe driver would be really
great to prevent packets from getting "stuck" in the ixgbe driver
(which prevents things like ssh from working since the SYN packet
isn't forwarded to the destination.

>> When I try large packet bursts (e.g., 16,000 packets back to back)
>> there is an interesting queueing-type behavior in which the latency
>> for the earlier packets is 14us, and then it linearly climbs up to
>> ~560us or so, and then jumps back down to 14 and repeats in a sawtooth
>> pattern.
>
> There are 2 forms of batching: NIC batching and Click batching. The
> described behavior might be related to Click batching. You could check
> the "BURST" parameters in PollDevice and ToDevice to adjust the Click
> batching.

I will try that today.  However the sawtooth is over 10,000+ packets,
and so I don't think it is related to a small constant amount of
batching.  Otherwise I would expect latency of packets to go up for
8-16 packets then go back down.  What I see now is that the first
10-20 packets have latencies in the 12-14us range, then it linearly
goes up to ~500-1000us over the next 10,000 packets, then there is a
discontinuous jump back to 14us and the process repeats itself.  I
think maybe the issue is that only one kernel thread is handling the
packets at this time.

> The RSS delivery to a particular queue is done based on the
> (SrcIP,DstIP,SrcPort,DstPort) tuple. If you send packets using the
> same headers, they will all end up in the same queue.

I'm going to vary the 5-tuple and try again to see if I can spread the
packets across the nic queues, which  should uniformly spread the
packets across those nic queues.

Thanks,
George