[Click] More on high latency...

[José María González]

Hi,

I run into the same problem than Nick and Marko. I wrote a simple BPF 
based program that just reads packets from 2 BPF devices. I tried to 
listen to both of them at the same time by using select(), poll(), 
and kqueue's kevent(). The first 2 worked perfectly, but with kqueue I
have the same problem than Marko: kevent() returns only after 2 packets
have been received in a BPF descriptor. 

I googled for a while, and I found that kqueue seems to have problems
with BPF when using BIOCIMMEDIATE and/or a BIOCSRTIMEOUT (which Click 
does):

http://www.freebsd.org/cgi/query-pr.cgi?pr=64178

[Note: we're using 4.10-RELEASE, and the patched sys/net/bpf.c is not 
there]


Unless I'm misunderstanding something (something likely), I find
arguable whether click does benefit from the use of kqueue, instead of 
the traditional select()/poll() mechanisms. 

For starters, kqueue is a new (2002, FreeBSD >= 4.1 or FreeBSD >= 5.0)
event notification mechanism that substitutes select() and poll(). The
2 main differences between kqueue and the previous mechanisms are: 

- kqueue is designed to scale with the number of descriptors a process
is listening to. In the original Lemon's paper, the performance of 
using poll() versus using kevent() starts to differ significantly when
listening to hundrers of descriptors. This should not be a problem for
click, which captures its traffic at BPF, instead of opening a
per-connection socket. Unless the user has *lots* of FromDevice's
(hundreds), it's hard to see how she will benefit from kqueue. 

- kqueue is designed to be more efficient, by storing per-process info in
the kernel instead of passing the full descriptor info every time a 
process wants to select()/poll() on them. From the original Lemon's paper,
it seems the cost of a call to kevent() (the kqueue equivalent to 
select()/poll() ) is significantly lower when listening to 100 descriptors. 
It may be the case that the cost is also different when listening to a 
handful of descriptor, but I doubt it (if only because the author would 
have reported it ;)

Again, I don't think this will be a problem for click users (I use a
pretty complicated click setup in my project, and still only use 10 BPF
descriptors). 

[Note: The original Lemon's paper is available at

http://people.freebsd.org/~jlemon/papers/kqueue.ps ]


The solution should be pretty straightforward: disable kqueue use unless 
explicitly requested by the user. Unless the user explicitly requests a 
CLI/compilation flag, change line 46 at lib/master.cc as follows:

-    _kqueue = kqueue();
+    _kqueue = -1;


Regards. 
-Chema





Marko Zec wrote:
> On Monday 28 March 2005 22:03, Nicholas Weaver wrote:
> > The high latency is definatly occuring within pcap:
> >
> > Printing the time just after pcap_dispatch calls
> > FromDevice_get_packet shows that this is the source of the injected
> > latency.
> >
> > I'm going to try compiling and linking to a different version of pcap
> > and see if it makes a difference.
> 
> 
> Hi Nick,
> 
> following your first report on this issue a week ago, I did some 
> digging...  My initial finding was that the problem might actually be 
> with the kqueue + bpf not playing nicely together in the FreeBSD 
> kernel, so this might have nothing to do with neither Click nor 
> libpcap.
> 
> What I observed in the kernel was that on each received packet the bpf 
> code correctly tried to wake up the listening process using a kqueue 
> macro, but the kqueue code for an unknown reason coalesced two such 
> events before actually doing the wakeup call.  Having no clue how the 
> kqueue kernel infrastructure is supposed to work internally I stopped 
> at that point, and then went into a suspend mode for a week after 
> becoming a happy father last weekend...  Will look more into this one 
> of these days.
> 
> Cheers,
> 
> Marko
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click