[Click] SMP Click - Question Task scheduling

Fri Feb 8 23:06:36 EST 2008

Hi Beyers,

Thanks very much for your suggestions. I increased TxDescriptors to 4096 but
still saw queue overflows. Then I changed the queue size to 2000. The queue
stopped overflowing but all the packets were not going through. A check on
the Ethernet driver showed a high count on rx_no_buffer_count and
rx_missed_errors. So I increased RxDescriptors to 4096. But the behavior is
the same.

I also played with the BURST settings on the poll device and to device.
Setting BURST to 32 caused queue overflows again.

Not sure what else I can try. One thought is that the packet processing by
the thread is consuming so much CPU it is unable to poll the adapter often
enough.

Ashok.

On 2/7/08, Beyers Cronje <bcronje at gmail.com> wrote:
>
> Hi Ashok,
>
> As far as I can tell when you run PollDevice then ToDevice always gets
> scheduled. Once scheduled PollDevice will push at most BURST number of
> packets where BURST is 8 by default, and ToDevice will try and transmit
> BURST number of packets where BURST is 16 by default.
>
> It could happen that ToDevice can not enqueue the packet to the adapter TX
> ring because it is full, which could lead to PollDevice pushing packets
> faster than what ToDevice can transmit them. Check ToDevice's 'drops'
> handler for this. In which case it might help increasing you adapter's TX
> ringsize, the default with e1000 is 256, I typically increase this to the
> maximum of 4096. You can also experiment with different settings for BURST
> in PollDevice and ToDevice. Also play with Queue's capacity value.
>
> Click also provides the ScheduleInfo element for you to manually tune
> scheduling paramaters of each element. I am NOT sure if ScheduleInfo is
> compatible with SMP Click, I have never tried it before.
> See http://read.cs.ucla.edu/click/elements/scheduleinfo
> So you could configure ToDevice to get scheduled more often than
> PollDevice. This can obviously lead to the receiving adapater's RX ring
> getting full if PollDevice is not scheduled fast/often enough to enqueue the
> packets to Click. At least dropping at the RX adapter is a lot less
> expensive than dropping at Click's Queue or the TX adapter, so this might be
> a "good" thing.
>
> Hope this helps a bit.
>
> Beyers
>
>  On Feb 7, 2008 5:44 PM, Ashok Ambati <ashok.ambati at gmail.com> wrote:
>
> > Hello,
> >
> > I am a new user. I am configuring the kernel module to route between 2
> > interfaces. On a 2-CPU server, I am creating 2 threads - one polling
> > eth0
> > and transmitting to eth1 and the other polling eth1 and trasmitting to
> > eth0.
> > I do not understand how each thread switches between the 2 tasks so as
> > to
> > maintain a balance between the incoming and outgoing packets. For the
> > eth1
> > -> eth0 path where there is heavy traffic, the queue fronting the
> > ToDevice(eth0) is overflowing. Am I doing something wrong?
> >
> > Here is my config -
> >
> >
> > // Generated by make-ip-conf.pl
> >
> > // eth0 192.168.200.136 00:04:23:C5:5D:CA
> >
> > // eth1 9.47.83.136 00:04:23:C5:5D:CB
> >
> > // Shared IP input path and routing table
> >
> > ip :: Strip(14)
> >
> > -> CheckIPHeader(INTERFACES 192.168.200.136/255.255.255.0
> > 9.47.83.136/255.255.255.0)
> >
> > -> rt :: StaticIPLookup(
> >
> > 192.168.200.136/32 0,
> >
> > 192.168.200.255/32 0,
> >
> > 192.168.200.0/32 0,
> >
> > 9.47.83.136/32 0,
> >
> > 9.47.83.255/32 0,
> >
> > 9.47.83.0/32 0,
> >
> > 192.168.200.0/255.255.255.0 1,
> >
> > 9.47.83.0/255.255.255.0 2,
> >
> > 255.255.255.255/32 0.0.0.0 0,
> >
> > 0.0.0.0/32 0,
> >
> > 0.0.0.0/0.0.0.0 9.47.83.1 2);
> >
> > // ARP responses are copied to each ARPQuerier and the host.
> >
> > arpt :: Tee(3);
> >
> > // Input and output paths for eth0
> >
> > c0 :: Classifier(12/0806 20/0001, 12/0806 20/0002, 12/0800, -);
> >
> > pd0 :: PollDevice(eth0);
> >
> > todevice0 :: ToDevice(eth0, 8);
> >
> > pd0 -> c0;
> >
> > out0 :: Queue(2000) -> todevice0;
> >
> > c0[0] -> ar0 :: ARPResponder(192.168.200.136/24 00:04:23:C5:5D:CA) ->
> > out0;
> >
> > arpq0 :: ARPQuerier(192.168.200.136, 00:04:23:C5:5D:CA) -> out0;
> >
> > c0[1] -> arpt;
> >
> > arpt[0] -> [1]arpq0;
> >
> > c0[2] -> Paint(1) -> ip;
> >
> > c0[3] -> Print("eth0 non-IP") -> Discard;
> >
> > // Input and output paths for eth1
> >
> > c1 :: Classifier(12/0806 20/0001, 12/0806 20/0002, 12/0800, -);
> >
> > pd1 :: PollDevice(eth1);
> >
> > pd1 -> c1;
> >
> > todevice1 :: ToDevice(eth1, 8);
> >
> > out1 :: Queue(2000) -> todevice1;
> >
> > c1[0] -> ar1 :: ARPResponder(9.47.83.136/24 00:04:23:C5:5D:CB) -> out1;
> >
> > arpq1 :: ARPQuerier(9.47.83.136, 00:04:23:C5:5D:CB) -> out1;
> >
> > c1[1] -> arpt;
> >
> > arpt[1] -> [1]arpq1;
> >
> > c1[2] -> Paint(2) -> ip;
> >
> > c1[3] -> Print("eth1 non-IP") -> Discard;
> >
> > // Local delivery
> >
> > toh :: ToHost;
> >
> > arpt[2] -> toh;
> >
> > rt[0] -> EtherEncap(0x0800, 1:1:1:1:1:1, 2:2:2:2:2:2) -> toh;
> >
> > // Forwarding path for eth0
> >
> > rt[1] -> DropBroadcasts
> >
> > -> cp0 :: PaintTee(1)
> >
> > -> gio0 :: IPGWOptions(192.168.200.136)
> >
> > -> FixIPSrc(192.168.200.136)
> >
> > -> dt0 :: DecIPTTL
> >
> > -> fr0 :: IPFragmenter(1500)
> >
> > -> [0]arpq0;
> >
> > dt0[1] -> ICMPError(192.168.200.136, timeexceeded) -> rt;
> >
> > fr0[1] -> ICMPError(192.168.200.136, unreachable, needfrag) -> rt;
> >
> > gio0[1] -> ICMPError(192.168.200.136, parameterproblem) -> rt;
> >
> > cp0[1] -> ICMPError(192.168.200.136, redirect, host) -> rt;
> >
> > // Forwarding path for eth1
> >
> > rt[2] -> DropBroadcasts
> >
> > -> cp1 :: PaintTee(2)
> >
> > -> gio1 :: IPGWOptions(9.47.83.136)
> >
> > -> FixIPSrc(9.47.83.136)
> >
> > -> dt1 :: DecIPTTL
> >
> > -> fr1 :: IPFragmenter(1500)
> >
> > -> [0]arpq1;
> >
> > dt1[1] -> ICMPError(9.47.83.136, timeexceeded) -> rt;
> >
> > fr1[1] -> ICMPError(9.47.83.136, unreachable, needfrag) -> rt;
> >
> > gio1[1] -> ICMPError(9.47.83.136, parameterproblem) -> rt;
> >
> > cp1[1] -> ICMPError(9.47.83.136, redirect, host) -> rt;
> >
> > //CPU assignment for Polling and Target devices
> >
> > StaticThreadSched(pd0 0);
> >
> > StaticThreadSched(todevice0 1);
> >
> > StaticThreadSched(pd1 1);
> >
> > StaticThreadSched(todevice1 0);
> >
> > // ThreadMonitor();
> >
> > Thanks.
> >
> >
> >
> > Ashok.
> > _______________________________________________
> > click mailing list
> > click at amsterdam.lcs.mit.edu
> > https://amsterdam.lcs.mit.edu/mailman/listinfo/click
> >
>
>