24000 packets/second

[Robert Morris]

In a fairly simple configuration I can get our kernel-based software
to forward 24000 packets/second. Ordinary Linux forwards at 33000 p/s
in the same configuration.

I'm sending TCP from zeus through darkstar to the-seventh-bridge,
using 256-byte packets. zeus and tsb are OpenBSD. darkstar is a Linux
PPro running our software. All of them with 100 mbit ethernet cards
and full-duplex ethernet switch connections. darkstar is forwarding
the packets out the same interface they arrived on.

I've attached the configuration file I used (it's conf/kbm.ipb).  It
doesn't include an IP routing table lookup, but it does include the IP
checksum check and re-calculation. The other missing pieces are
fragmentation and options processing; since they're not actually used
much, they won't impact typical performance.

Here's the configuration file:

# Run this on darkstar.

# darkstar 18.26.4.60  00:80:C8:4B:25:00
# tsb      18.26.4.200 00:a0:c9:9c:fd:9c
# zeus     18.26.4.97  00:00:c0:ca:68:ef

# 0. Packets to tsb.
# 1. Packets to zeus.
# 2. ARP queries.
# 3. ARP responses.
# 4. Others.
fi :: Filter(12/0800 30/121a04C8,
             12/0800 30/121a0461,
             12/0806 20/0001,
             12/0806 20/0002,
             -);

KernelReader(eth0) -> [0]fi;
outq :: Queue(20) -> KernelWriter(eth0);
tol :: ToLinux;

# Common code to send IP packets, including ARP.
# IP dst annotation must already be set.
arpq :: ArpQuerier(18.26.4.60, 00:80:C8:4B:25:00);
fi[3] -> t1::Tee -> [1]arpq;
arpq[0] -> outq;
t1[1] -> tol;

# No ARP responder... Not proxying.
fi[2] -> tol;

# Route packets between tsb and zeus.
# They attach darkstar's ethernet address to packets,
# so the real work is in ARP, not IP routing.
fi[0] -> xx::Strip(14)
      -> CheckIPChecksum
      -> DecTTL
      -> SetIPChecksum
      -> GetIPAddress(16)
      -> [0]arpq;
fi[1] -> xx;

# Give everything else to Linux.
fi[4] -> tol;