[Click] Outgoing traffic uses one network interface (out of two) only

[Quentin Monnet]

Hello,

I have the following network configuration :
Host1 (H1) <----- 10.0.0.0/24 -----> Router1 (R1) <----- 192.168.1.0/24
-----> Router2 (R2) <----- 10.0.1.0/24 -----> Host2 (H2)

I would like to run (at userlevel) the canonical IP router from the SOSP
paper on R1 and R2, so as to ping H2 with H1.
I use the same configuration file for R1 and R2 (changing only IP and
MAC addresses).

Routing works well on R1, but I have something wrong with R2. When
debugging with tcpdump, it seems that the whole outgoing traffic goes
through the first network interface declared with "FromDevice".
I mean that if my configuration file contains:

"FromDevice(eth0) -> [0]c0;
FromDevice(eth1) -> [0]c1;"

then all the outgoing packets are sent to eth0, whatever network
interface I give to "ToDevice".
If my file contains:

"FromDevice(eth1) -> [0]c1;
FromDevice(eth0) -> [0]c0;"

then everything goes to eth1.

My network seems to work well without Click: I can ping H2 with H1. I
have tried several machines and Click versions, but I can't figure out
what is wrong.

R1 is Ubuntu 10.10 with kernel 2.6.35 and Click version 1.8. There is no
problem.
R2: I tried Ubuntu 11.04 with kernel version 2.6.38 and Click version
from git (it was 1.8 then).
As I don't know where the problem comes from, I've tried with another
router for R2: Ubuntu 9.04 with kernel 2.6.28 and Click version 2.0. I
get the same behaviour. I can't see what the difference from R1 is.
I am working at userlevel only.

I give the Click configuration file I use for R1 below. R2 is the same
with modified IP and MAC addresses.
Thanks to anyone who could help.

Regards,
Quentin

--------------------------------------------------------------------------------------

// Kernel configuration for cone as a router between
// 192.168.1 (eth0) and 10.0.0 (vboxnet1).
// Proxy ARPs for 10.0.0 on eth0.
 
// eth0,        00:00:C0:AE:67:EF, 192.168.1.222
// vboxnet1,    00:00:C0:4F:71:EF, 10.0.0.1
 
// 0. ARP queries
// 1. ARP replies
// 2. IP
// 3. Other
// We need separate classifiers for each interface because
// we only want proxy ARP on eth0.
c0 :: Classifier(12/0806 20/0001,
                 12/0806 20/0002,
                 12/0800,
                 -);
 
c1 :: Classifier(12/0806 20/0001,
                 12/0806 20/0002,
                 12/0800,
                 -);
 
 
FromDevice(eth0) -> [0]c0;
FromDevice(vboxnet1) -> [0]c1;
FromHost(virtual) -> [0]c0;

out0 :: Queue(200) -> ToDevice(eth0) -> Print(eth0) -> Discard;
out1 :: Queue(200) -> ToDevice(vboxnet1) -> Print(vboxnet1) -> Discard;
tol :: ToHost(virtual);
 
// An "ARP querier" for each interface.
arpq0 :: ARPQuerier(192.168.1.222, 00:00:c0:ae:67:ef);
arpq1 :: ARPQuerier(10.0.0.1, 00:00:c0:4f:71:ef);
 
// Deliver ARP responses to ARP queriers as well as Linux.
t :: Tee(3);
c0[1] -> t;
c1[1] -> t;
t[0] -> tol;
t[1] ->[1]arpq0
t[2] ->[1]arpq1
 
// Connect ARP outputs to the interface queues.
arpq0 -> out0;
arpq1 -> out1;
 
// Proxy ARP on eth0 for 18.26.7, as well as cone's IP address.
ar0 :: ARPResponder(192.168.1.222        00:00:c0:ae:67:ef,
                    10.0.0.1/24        00:00:c0:4f:71:ef);
c0[0] -> ar0 -> out0;
 
// Ordinary ARP on eth1.
ar1 :: ARPResponder(10.0.0.1        00:00:c0:4f:71:ef);
c1[0] -> ar1 -> out1;
 
// IP routing table. Outputs:
// 0: packets for this machine.
// 1: packets for 18.26.4.
// 2: packets for 18.26.7.
// All other packets are sent to output 1, with 18.26.4.1 as the gateway.
rt :: RadixIPLookup(    192.168.1.222/32        0,
                        10.0.1.1/24        1,
                        10.0.0.1/32        0,
                        192.168.1.222/24        1,
                        10.0.0.1/24        2);
 
// Hand incoming IP packets to the routing table.
// CheckIPHeader checks all the lengths and length fields
// for sanity.
ip ::   Strip(14)
     -> ciph :: CheckIPHeader(INTERFACES 192.168.1.222/24 10.0.0.1/24)
     -> [0]rt;
c0[2] -> Paint(1) -> ip;
c1[2] -> Paint(2) -> ip;
 
// IP packets for this machine.
// ToHost expects ethernet packets, so cook up a fake header.
rt[0] -> EtherEncap(0x0800, 1:1:1:1:1:1, 2:2:2:2:2:2) -> tol;
 
// These are the main output paths; we've committed to a
// particular output device.
// Check paint to see if a redirect is required.
// Process record route and timestamp IP options.
// Fill in missing ip_src fields.
// Discard packets that arrived over link-level broadcast or multicast.
// Decrement and check the TTL after deciding to forward.
// Fragment.
// Send outgoing packets through ARP to the interfaces.
rt[1] -> DropBroadcasts
      -> cp1 :: PaintTee(1)
      -> gio1 :: IPGWOptions(192.168.1.222)
      -> FixIPSrc(192.168.1.222)
      -> dt1 :: DecIPTTL
      -> fr1 :: IPFragmenter(1500)
      -> [0]arpq0;

rt[2] -> DropBroadcasts
      -> cp2 :: PaintTee(2)
      -> gio2 :: IPGWOptions(10.0.0.1)
      -> FixIPSrc(10.0.0.1)
      -> dt2 :: DecIPTTL
      -> fr2 :: IPFragmenter(1500)
      -> [0]arpq1;
 
// DecIPTTL[1] emits packets with expired TTLs.
// Reply with ICMPs. Rate-limit them?
dt1[1] -> ICMPError(192.168.1.222, timeexceeded) -> [0]rt;
dt2[1] -> ICMPError(192.168.1.222, timeexceeded) -> [0]rt;
 
// Send back ICMP UNREACH/NEEDFRAG messages on big packets with DF set.
// This makes path mtu discovery work.
fr1[1] -> ICMPError(10.0.0.1, unreachable, needfrag) -> [0]rt;
fr2[1] -> ICMPError(10.0.0.1, unreachable, needfrag) -> [0]rt;
 
// Send back ICMP Parameter Problem messages for badly formed
// IP options. Should set the code to point to the
// bad byte, but that's too hard.
gio1[1] -> ICMPError(192.168.1.222, parameterproblem) -> [0]rt;
gio2[1] -> ICMPError(192.168.1.222, parameterproblem) -> [0]rt;
 
// Send back an ICMP redirect if required.
cp1[1] -> ICMPError(192.168.1.222, redirect, host) -> [0]rt;
cp2[1] -> ICMPError(10.0.0.1, redirect, host) -> [0]rt;
 
// Unknown ethernet type numbers.
c0[3] -> Discard;
c1[3] -> Discard;