[Click] Queues vs NIC tx/rx rings

[rchertov at purdue.edu]

Hello,
   I have been trying to understand the effect of various queue (ring) sizes on
the e1000 device driver and the click Queue element.  I use a click router to
connect two nodes just by connecting the devices together so that the Click
node acts as a layer 2 bridge and performs simple forwading.  As I understand
from the Click paper, Click takes packets out of the RX ring of the driver and
then passes it into the Queue element, assuming the config is
Poll->Queue->ToDev.  Then the Queue puts the packet into the TX ring buffer of
the device driver.  

   I have done some experiments on the 100 Mbit network where a UDP generator
uses a raw socket to put the packets on the wire.   The generator reports how
many times the send call was done successfully.  On the reciever I run
Click/udpcount to count packets and the rate.  I have tried changing the tx/rx
sizes via the ethtool as well as txqueuelen via ifconfig.  In all of the runs
there does not seem to be much difference between the rates and packet counts. 
Only difference occurs is when the click Queue is 50 or larger.  Then the send
count == recieve count with the 148K pps rate.  In other cases the rate is about
148K pps but there are packet drops.  So I am curious as to why if I have large
tx/rx buffers and a small Queue I get drops.  It would seem that Click would
just need to quickly copy from the RX ring buffer to the TX ring buffer and
there should not be much need for intermediate queueing. (or does the bus speed
dictate a need for a large intermediate queue?)  Additionally if I have small
(96) tx/rx rings but a Queue >= 50, then all the pakcets maket it through.  So I
am curious in what cases the device driver ring sizes actually play any role?  


Any insight would be much appreciated.


Roman Chertov