[Click] SMP Click - Question Task scheduling

Thu Feb 7 12:16:44 EST 2008

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

To add to what Beyers said, you have to remember that cycles used per
PollDevice/ToDevice heavily depend on the element connection chain.  For
instance,
   PollDevice -> A -> B -> ... -> W -> q::Queue;
   q -> ToDevice;

In this example when PollDevice gets scheduled it has to run through all
of the elements until it gets to the Queue.  This happens per packet.
Hence, for the same packet rate PollDevice will consume more of the CPU
and leave less CPU time for the ToDevice.

Also, you have only two CPUs and you allocate two threads for packet
processing.  However, you have to remember that elements that you did
not assign to any CPU have to run somewhere.  Ideally, you would use 3
threads then: one for the main thread, two for eth1->eth0, and three
eth0->eth1.

Roman

Beyers Cronje wrote:
> Hi Ashok,
> 
> As far as I can tell when you run PollDevice then ToDevice always gets
> scheduled. Once scheduled PollDevice will push at most BURST number of
> packets where BURST is 8 by default, and ToDevice will try and transmit
> BURST number of packets where BURST is 16 by default.
> 
> It could happen that ToDevice can not enqueue the packet to the adapter TX
> ring because it is full, which could lead to PollDevice pushing packets
> faster than what ToDevice can transmit them. Check ToDevice's 'drops'
> handler for this. In which case it might help increasing you adapter's TX
> ringsize, the default with e1000 is 256, I typically increase this to the
> maximum of 4096. You can also experiment with different settings for BURST
> in PollDevice and ToDevice. Also play with Queue's capacity value.
> 
> Click also provides the ScheduleInfo element for you to manually tune
> scheduling paramaters of each element. I am NOT sure if ScheduleInfo is
> compatible with SMP Click, I have never tried it before.
> See http://read.cs.ucla.edu/click/elements/scheduleinfo
> So you could configure ToDevice to get scheduled more often than PollDevice.
> This can obviously lead to the receiving adapater's RX ring getting full if
> PollDevice is not scheduled fast/often enough to enqueue the packets to
> Click. At least dropping at the RX adapter is a lot less expensive than
> dropping at Click's Queue or the TX adapter, so this might be a "good"
> thing.
> 
> Hope this helps a bit.
> 
> Beyers
> 
> On Feb 7, 2008 5:44 PM, Ashok Ambati <ashok.ambati at gmail.com> wrote:
> 
>> Hello,
>>
>> I am a new user. I am configuring the kernel module to route between 2
>> interfaces. On a 2-CPU server, I am creating 2 threads - one polling eth0
>> and transmitting to eth1 and the other polling eth1 and trasmitting to
>> eth0.
>> I do not understand how each thread switches between the 2 tasks so as to
>> maintain a balance between the incoming and outgoing packets. For the eth1
>> -> eth0 path where there is heavy traffic, the queue fronting the
>> ToDevice(eth0) is overflowing. Am I doing something wrong?
>>
>> Here is my config -
>>
>>
>> // Generated by make-ip-conf.pl
>>
>> // eth0 192.168.200.136 00:04:23:C5:5D:CA
>>
>> // eth1 9.47.83.136 00:04:23:C5:5D:CB
>>
>> // Shared IP input path and routing table
>>
>> ip :: Strip(14)
>>
>> -> CheckIPHeader(INTERFACES 192.168.200.136/255.255.255.0
>> 9.47.83.136/255.255.255.0)
>>
>> -> rt :: StaticIPLookup(
>>
>> 192.168.200.136/32 0,
>>
>> 192.168.200.255/32 0,
>>
>> 192.168.200.0/32 0,
>>
>> 9.47.83.136/32 0,
>>
>> 9.47.83.255/32 0,
>>
>> 9.47.83.0/32 0,
>>
>> 192.168.200.0/255.255.255.0 1,
>>
>> 9.47.83.0/255.255.255.0 2,
>>
>> 255.255.255.255/32 0.0.0.0 0,
>>
>> 0.0.0.0/32 0,
>>
>> 0.0.0.0/0.0.0.0 9.47.83.1 2);
>>
>> // ARP responses are copied to each ARPQuerier and the host.
>>
>> arpt :: Tee(3);
>>
>> // Input and output paths for eth0
>>
>> c0 :: Classifier(12/0806 20/0001, 12/0806 20/0002, 12/0800, -);
>>
>> pd0 :: PollDevice(eth0);
>>
>> todevice0 :: ToDevice(eth0, 8);
>>
>> pd0 -> c0;
>>
>> out0 :: Queue(2000) -> todevice0;
>>
>> c0[0] -> ar0 :: ARPResponder(192.168.200.136/24 00:04:23:C5:5D:CA) ->
>> out0;
>>
>> arpq0 :: ARPQuerier(192.168.200.136, 00:04:23:C5:5D:CA) -> out0;
>>
>> c0[1] -> arpt;
>>
>> arpt[0] -> [1]arpq0;
>>
>> c0[2] -> Paint(1) -> ip;
>>
>> c0[3] -> Print("eth0 non-IP") -> Discard;
>>
>> // Input and output paths for eth1
>>
>> c1 :: Classifier(12/0806 20/0001, 12/0806 20/0002, 12/0800, -);
>>
>> pd1 :: PollDevice(eth1);
>>
>> pd1 -> c1;
>>
>> todevice1 :: ToDevice(eth1, 8);
>>
>> out1 :: Queue(2000) -> todevice1;
>>
>> c1[0] -> ar1 :: ARPResponder(9.47.83.136/24 00:04:23:C5:5D:CB) -> out1;
>>
>> arpq1 :: ARPQuerier(9.47.83.136, 00:04:23:C5:5D:CB) -> out1;
>>
>> c1[1] -> arpt;
>>
>> arpt[1] -> [1]arpq1;
>>
>> c1[2] -> Paint(2) -> ip;
>>
>> c1[3] -> Print("eth1 non-IP") -> Discard;
>>
>> // Local delivery
>>
>> toh :: ToHost;
>>
>> arpt[2] -> toh;
>>
>> rt[0] -> EtherEncap(0x0800, 1:1:1:1:1:1, 2:2:2:2:2:2) -> toh;
>>
>> // Forwarding path for eth0
>>
>> rt[1] -> DropBroadcasts
>>
>> -> cp0 :: PaintTee(1)
>>
>> -> gio0 :: IPGWOptions(192.168.200.136)
>>
>> -> FixIPSrc(192.168.200.136)
>>
>> -> dt0 :: DecIPTTL
>>
>> -> fr0 :: IPFragmenter(1500)
>>
>> -> [0]arpq0;
>>
>> dt0[1] -> ICMPError(192.168.200.136, timeexceeded) -> rt;
>>
>> fr0[1] -> ICMPError(192.168.200.136, unreachable, needfrag) -> rt;
>>
>> gio0[1] -> ICMPError(192.168.200.136, parameterproblem) -> rt;
>>
>> cp0[1] -> ICMPError(192.168.200.136, redirect, host) -> rt;
>>
>> // Forwarding path for eth1
>>
>> rt[2] -> DropBroadcasts
>>
>> -> cp1 :: PaintTee(2)
>>
>> -> gio1 :: IPGWOptions(9.47.83.136)
>>
>> -> FixIPSrc(9.47.83.136)
>>
>> -> dt1 :: DecIPTTL
>>
>> -> fr1 :: IPFragmenter(1500)
>>
>> -> [0]arpq1;
>>
>> dt1[1] -> ICMPError(9.47.83.136, timeexceeded) -> rt;
>>
>> fr1[1] -> ICMPError(9.47.83.136, unreachable, needfrag) -> rt;
>>
>> gio1[1] -> ICMPError(9.47.83.136, parameterproblem) -> rt;
>>
>> cp1[1] -> ICMPError(9.47.83.136, redirect, host) -> rt;
>>
>> //CPU assignment for Polling and Target devices
>>
>> StaticThreadSched(pd0 0);
>>
>> StaticThreadSched(todevice0 1);
>>
>> StaticThreadSched(pd1 1);
>>
>> StaticThreadSched(todevice1 0);
>>
>> // ThreadMonitor();
>>
>> Thanks.
>>
>>
>>
>> Ashok.
>> _______________________________________________
>> click mailing list
>> click at amsterdam.lcs.mit.edu
>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHqzz7T8ksiSCF2AYRAvEmAJ9g4SMvDD/NqEhBX/uQ/QeXhatWKwCfXBJu
8bdyWfMj8HcWnJEun3cz4PM=
=NQlu
-----END PGP SIGNATURE-----