[Click] User level Click Queues vs Kernel Queues

[Beyers Cronje]

Hi Michael,

My 2cents below.

Beyers

On 6/11/05, Michael Sirivianos <msirivia at uci.edu> wrote:
> Hi,
> 
> We are trying to setup a simple experiment with userlevel click to
> measure the performance degradation of a router performing some crypto
> computations.
> 
> We would prefer to use it at kernel space, but 2.4 kernels do not really
> work on our PC's (unless we spend all our time trying to choose the
> correct compiling options) and due to gcc/glib incompatibilities we
> cannot even compile them.

I would say it is well worth sorting your kernel compilation issues
out. I typically start from a config of the installed distribution's
configuration file, on my FC2 installation these are located at
/usr/src/linux-2.6.5-1.358/configs/
>From there you can disable all unnecesary options etc.
I'm currently using gcc 3.3.3 and everything compiles 100%. (I seem to
recall struggling with gcc 3.4)

> 
> our configuration is sthng like:
> 
> fromDevice->classifier-> Queue(200) -> Unqueue->StripIpheader-
> >...OurElement->ARPQuerier.

Why are you using Unqueue ? Anyone correct me if I'm wrong, but
typically Unqueue is used in MT Click to provide an additional
schedulable element to help with parallism ?
So the following config should probably work better:
fromDevice->classifier-> StripIpheader->
...OurElement->ARPQuerier -> Queue(200) -> ToDevice.

FromDevice will push packets to Queue, where ToDevice will pull from the queue.

> 
> However, we observe that no matter how expensive the computation in
> OurElement is and no matter how much the sender rate increases, the
> Queue never builds up over 1 packet. Instead we have packet losses, that
> we have not been able to pinpoint exactly where they happen. By the
> network monitor tools we infer, its not in the Ethernet interface, and
> most likely not in the IP input queue at the kernel.
> 
> We theorize that we may have drops at a kernel/userspace queue but we
> would like an opinion on that.
> 
> Am I correct to assume that  a frame is directly forwarded from the
> device to a kernel queue and then userlevel click which is a single
> thread  reads the frame from this in-kernel queue? Then click proceeds
> with processing the packets at the rest of the modules including our
> Queue element? Thus, the same thread puts the packet in the click queue
> and then processes it. This is the only explanation I could find for not
> being able to build up our Click queue over 1 packet.

Userlevel FromDevice receives packets from the kernel over a raw
packet socket.  Now from what I can make out netif_receive_skb  in
net/core/dev.c delivers the received sk_buff  to all protocol handlers
for the given packet. So for user-level Click the skb gets delivered
to both the standard protocol handler and the packet socket handler. I
other words the packet gets processed twice.
FromDevice reads the packets from this packet socket. 

Eddie, anyone, could the packet loss be due to socket receive queue overflows ? 
I presume user-level click does not get scheduled quickly/often enough
to empty the socket receive queues ?

My tests so far seem to indicate a maximum of around 50,000 pps loss
free (64 byte packets) throughput in user-level click with a simple
etherswitch configuration. (linux 2.4.26 with P4 PCI e1000)
 
> Is it impossible to conduct this experiment at the userlevel?
You can do the experiment at userlevel, keeping in mind the
performance related issues with running in user-level.

> At the kernel level would the NIC be the producer of packets that
> directly places frame in our Click Queue and the Elements Unqueue->...
> would be the consumers?

Search the list for polling questions, there are a number of posts
describing how the poll process works.


> Thanks,
> Michael
> 
> 
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>