[Click] Queues vs NIC tx/rx rings

[Beyers Cronje]

Hi Roman,

ToDev will actually put the packet into the device TX ring. In your
configuration the Queue element provides the connection point between
the click push path (originating from Poll) and click pull path
(originating from ToDev). Click relies on both the Push and Pull paths
to operate.

Both Poll and ToDev are schedulable by the click scheduler, when Poll
is scheduled it will pull BURST amount of packets from the device's rx
queue and Push these packets to the Queue. When Poll is scheduled it
will initiate the Pull path and pull BURST amount packets from the
Queue element.

Theoretically I suppose you would be able to write your own ToDev
element which does not rely on Pulling packets from a Queue, but
instead receives packets directly from Poll element via Push. This
would probably only be usable in a simple configuration like Poll ->
ToDev and will not be scalable in a more complex configuration.

While the Click router is busy processing packets through either the
Push or Pull paths, packets received from the network on the device
are queued on the device's RX queue. This device RX queue will keep
the received packets queued until Click retrieves them through the
Poll element. If Click does not complete the packet processing in time
to call Poll this device rx queue can overflow, efectively dropping
packets. Depending on the traffic load and how expensive your Click
configuration is, increasing  your device's rx/tx queue sizes should
help prevent packet drops on the device's queues. (At the expense of
ram ??)

When your Queue element's size is too small, Poll element might queue
packets too fast for ToDev to pull them from the queue due a slow NIC
tx action or whatever, which will lead to drops at the click Queue
element. Also, I'm not sure how the click scheduler permorms it's
schedule selection, which might also influence click Queue drops as
well.

There're obviously more factors that can lead to performance/drops
issues, so the above desciptions provides one such explanation.

Beyers

On 7/8/05, rchertov at purdue.edu <rchertov at purdue.edu> wrote:
> Hello,
>    I have been trying to understand the effect of various queue (ring) sizes on
> the e1000 device driver and the click Queue element.  I use a click router to
> connect two nodes just by connecting the devices together so that the Click
> node acts as a layer 2 bridge and performs simple forwading.  As I understand
> from the Click paper, Click takes packets out of the RX ring of the driver and
> then passes it into the Queue element, assuming the config is
> Poll->Queue->ToDev.  Then the Queue puts the packet into the TX ring buffer of
> the device driver.

>    I have done some experiments on the 100 Mbit network where a UDP generator
> uses a raw socket to put the packets on the wire.   The generator reports how
> many times the send call was done successfully.  On the reciever I run
> Click/udpcount to count packets and the rate.  I have tried changing the tx/rx
> sizes via the ethtool as well as txqueuelen via ifconfig.  In all of the runs
> there does not seem to be much difference between the rates and packet counts.
> Only difference occurs is when the click Queue is 50 or larger.  Then the send
> count == recieve count with the 148K pps rate.  In other cases the rate is about
> 148K pps but there are packet drops.  So I am curious as to why if I have large
> tx/rx buffers and a small Queue I get drops.  It would seem that Click would
> just need to quickly copy from the RX ring buffer to the TX ring buffer and
> there should not be much need for intermediate queueing. (or does the bus speed
> dictate a need for a large intermediate queue?)  Additionally if I have small
> (96) tx/rx rings but a Queue >= 50, then all the pakcets maket it through.  So I
> am curious in what cases the device driver ring sizes actually play any role?
> 
> 
> Any insight would be much appreciated.
> 
> 
> Roman Chertov
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>