[Click] Large latency with RouteBricks setup

[George Porter]

Hello everyone,

I've recently setup a RouteBricks installation, and am really close to
having it work, but ran into a problem I was hoping someone might be
able to help me with.

I've got an HP DL380G6 server with two intel E5520 4-core CPUs, along
with a dual-port intel 82598EB NIC (with two SFP+ interfaces).  I
followed the instructions on the RouteBricks wiki for recompiling the
kernel, rebuilding the ixgbe driver with the RB patches, and
installing everything.  I've setup three machines in series (A -> RB
-> B), and am able to ping between A and RB, and RB and B.  If I set
/proc/sys/net/ipv4/ip_forward on the RB server I can ping from A to B
through RB (I tested this by then setting that proc field to 0 and saw
the pings stop).  I loaded the 'minfwtest.click' file with
click-install and it seemed to load OK (dmesg didn't show any errors,
nor did /click/errors).  However, when I try to ping from A to B
through RouteBricks, I see huge ping times:

root at dcswitch73:/home/gmporter# ping 192.168.5.74
PING 192.168.5.74 (192.168.5.74) 56(84) bytes of data.
64 bytes from 192.168.5.74: icmp_req=1 ttl=64 time=13104 ms
64 bytes from 192.168.5.74: icmp_req=2 ttl=64 time=12096 ms
64 bytes from 192.168.5.74: icmp_req=3 ttl=64 time=11088 ms
64 bytes from 192.168.5.74: icmp_req=4 ttl=64 time=10080 ms
64 bytes from 192.168.5.74: icmp_req=5 ttl=64 time=9072 ms
64 bytes from 192.168.5.74: icmp_req=6 ttl=64 time=8064 ms
64 bytes from 192.168.5.74: icmp_req=7 ttl=64 time=7056 ms
[snip]
64 bytes from 192.168.5.74: icmp_req=21 ttl=64 time=17136 ms
64 bytes from 192.168.5.74: icmp_req=22 ttl=64 time=16128 ms
64 bytes from 192.168.5.74: icmp_req=23 ttl=64 time=15120 ms
^C
--- 192.168.5.74 ping statistics ---
43 packets transmitted, 23 received, 46% packet loss, time 42335ms
rtt min/avg/max/mdev = 7056.587/18845.638/30240.430/7061.770 ms, pipe 31

Any thoughts--has anyone seen anything like this before?  Also, is
there a good way to debug/diagnose what might be going on inside of
click on the RouteBricks server?

Thanks,
George