[Click] Large latency with RouteBricks setup

[Mihai Dobrescu]

Hi,

For the moment I don't have a proper setup to test the described
behavior, but the problem seems to be related to the batch settings in
the NIC driver.

You can see more details on the NIC driver batching in the Routebricks
paper (http://www.sigops.org/sosp/sosp09/papers/dobrescu-sosp09.pdf)
"NIC-driven batching means that a packet may wait for 16 packets
before transmission". This won't help you if you generate packets at
low rates, but it's very useful for high packet rates.

You can either a) generate packets at high rates or b) change the
batch settings of the driver.

For the option b) see the datasheet for the Intel 82598 card
(http://download.intel.com/design/network/datashts/319282.pdf). You
can have a look at section 4.4.3.5.6 Receive Descriptor Control –
RXDCTL and section 4.4.3.7.6 Transmit Descriptor Control – TXDCTL.
"WTHRESH controls the write-back of processed receive descriptors.
This threshold refers to the number of receive descriptors in the
on-chip buffer which are ready to be written back to host memory. In
the absence of external events (explicit flushes), the write-back
occurs only after at least WTHRESH descriptors are available for
write-back."

b1) An easy fix would be to try to change the the batch value from 16
to 8 or 4 to see if the latency improves. Check the driver's source
folder (ixgbe_main.c) and look for something like rxdctl |= (16<<16);
txdctl |= (16<<16).
b2) Alternatively you can set some timeout in the driver: i.e. if the
driver doesn't get X packets within Y milliseconds then call an
explicit flush.

Cheers,
Mihai

On Mon, Aug 8, 2011 at 7:47 PM, George Porter <gmporter at cs.ucsd.edu> wrote:
> Hello everyone,
>
> I've recently setup a RouteBricks installation, and am really close to
> having it work, but ran into a problem I was hoping someone might be
> able to help me with.
>
> I've got an HP DL380G6 server with two intel E5520 4-core CPUs, along
> with a dual-port intel 82598EB NIC (with two SFP+ interfaces).  I
> followed the instructions on the RouteBricks wiki for recompiling the
> kernel, rebuilding the ixgbe driver with the RB patches, and
> installing everything.  I've setup three machines in series (A -> RB
> -> B), and am able to ping between A and RB, and RB and B.  If I set
> /proc/sys/net/ipv4/ip_forward on the RB server I can ping from A to B
> through RB (I tested this by then setting that proc field to 0 and saw
> the pings stop).  I loaded the 'minfwtest.click' file with
> click-install and it seemed to load OK (dmesg didn't show any errors,
> nor did /click/errors).  However, when I try to ping from A to B
> through RouteBricks, I see huge ping times:
>
> root at dcswitch73:/home/gmporter# ping 192.168.5.74
> PING 192.168.5.74 (192.168.5.74) 56(84) bytes of data.
> 64 bytes from 192.168.5.74: icmp_req=1 ttl=64 time=13104 ms
> 64 bytes from 192.168.5.74: icmp_req=2 ttl=64 time=12096 ms
> 64 bytes from 192.168.5.74: icmp_req=3 ttl=64 time=11088 ms
> 64 bytes from 192.168.5.74: icmp_req=4 ttl=64 time=10080 ms
> 64 bytes from 192.168.5.74: icmp_req=5 ttl=64 time=9072 ms
> 64 bytes from 192.168.5.74: icmp_req=6 ttl=64 time=8064 ms
> 64 bytes from 192.168.5.74: icmp_req=7 ttl=64 time=7056 ms
> [snip]
> 64 bytes from 192.168.5.74: icmp_req=21 ttl=64 time=17136 ms
> 64 bytes from 192.168.5.74: icmp_req=22 ttl=64 time=16128 ms
> 64 bytes from 192.168.5.74: icmp_req=23 ttl=64 time=15120 ms
> ^C
> --- 192.168.5.74 ping statistics ---
> 43 packets transmitted, 23 received, 46% packet loss, time 42335ms
> rtt min/avg/max/mdev = 7056.587/18845.638/30240.430/7061.770 ms, pipe 31
>
> Any thoughts--has anyone seen anything like this before?  Also, is
> there a good way to debug/diagnose what might be going on inside of
> click on the RouteBricks server?
>
> Thanks,
> George
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>