Performance with EtherPro1000 when connecting multiple routers

[Brecht Vermeulen]

Hi all,

We've got some 3 routers with dual PIII 1,266 GHz Tyan motherboards (64
bit/66 MHz PCI bus) and with Intel EtherPro 1000/F cards and have some
performance issues:

- if we measure only 1 router under test with the Smartbits (with two
Gigabit cards), we have a throughput of 
approx. 1 Gbit/s (462000pps) for 250 byte packets, 1,9 Gbit/s
(500000pps, full duplex) for 450 byte packets and 500 Mbit/s (740000pps)
for 64 byte packets.
(the click configuration is basically the make-ip-conf.pl script, but
with queues exchanged by MSQueue and LookupiprouteMP and also 
print "StaticThreadSched(PollDevice0 1, todevice1 1, PollDevice1 0,
todevice0 0);\n";              

- if however we connect three such routers to each other (with 2 e1000
cards each) and measure the throughput, the 
throughput falls sometimes to only 14000pps while the middle router has
very high numbers for the cycles of the polldevices (threadmonitor) of
80000.
(sometimes syslog get filled with 
Jun  5 14:01:36 apollo140 kernel: e1000_intr: polling!
Jun  5 14:01:36 apollo140 kernel: e1000_intr: icr 8)
(anyone an idea how this come, I think it is some interrupt while in
polling mode, but why ?)
The 
Rx_FIFO_Errors            1187252689
Rx_Missed_Errors          1187252689
numbers in /proc/net/PRO_LAN_Adapters/eth5.info
are also increasing steadily (polldevice doesn't read the packets fast
enough from the card ?)
If I change the burst size of PollDevice and Todevice to 1 or 2, then
the performance is again around 300000 pps (but still lower than a
single router)

So, there seems to be some problems with the communication between 2
e1000 cards, has anyone a clue on this (flow control ?) as I can't see
it.

thanks for any information,
Brecht