[Click] click userland patches for large speed improvements

[Luigi Rizzo]

On Thu, Jul 07, 2011 at 10:43:25PM +0200, Beyers Cronje wrote:
> Hi Luigi,
> 
> Are there any plans to port netmap to Linux? I'm a big fan of pushing
> development in Click userlevel and believe netmap would definitely be an
> asset.

i am hoping to get some time to work on it in september/october.
Of course if someone has spare cycles and is interested to work on
it i am happy to help/review the code.

cheers
luigi

> Beyers
> 
> On Fri, Jul 1, 2011 at 7:47 PM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:
> 
> > If someone is interest in performance of userland click, i'd suggest
> > the following two patches and looking at netmap (i already discussed
> > what follows with Eddie, and i am hoping someone more fluent than
> > me in C++ can polish the code and add a support for thread-local lists).
> >
> > To get an idea of what you can get on a single core i7-870 CPU with
> > the stock version and with these patches:
> >
> >                                        1.8.0           With patches
> >    InfiniteSource -> Discard           515Kpps         18.56Mpps
> >    InfiniteSource -> Queue -> Discard  500Kpps         13.41Mpps
> >
> >                                        pcap            netmap
> >    FromDevice->Queue->ToDevice         420Kpps         3.97 Mpps
> >
> >
> > Click userland performance was never a priority given the high cost
> > (until now) of packet I/O. But once packet i/o has become quite fast,
> > it turns out that there are to other big offenders:
> > - the C++ memory allocator is quite expensive, and replacing it with
> >  thread-local freelists (Packet objects and data buffers can be made
> >  all with the same size) gives huge savings -- 100ns per packet or more
> >  even on a fast machine;
> >
> > - everytime an element wants a timestamp, it calls a syscall
> > (gettimeofday()
> >  or similar) which consumes another 400-800ns per call. There are many
> >  elements (e.g. InfiniteSource, Counter, etc.) which timestamp packets.
> >
> > Attached there are a couple of patches which address these problems:
> >
> > - patch-pcap    makes FromDevice and ToDevice use libpcap properly,
> >                supporting I/O in bursts to amortize the syscall overhead.
> >                This has been tested on FreeBSD.
> >
> > - patch-more
> >   + introduces a NOTS option for InfiniteSource to remove timestamps.
> >     This gives a 10x performance improvement in simple apps using
> > InfiniteSource
> >
> >   + replaces the allocator for Packet and data buffers with local
> > freelists;
> >     not thread safe, but this is easy to introduce. This gives another
> > 1.5-2x
> >     speed improvement after the 10x gained removing timestamps;
> >
> >   + enables BURST operation in Discard, giving another 2x speed improvement
> >
> > Using netmap instead of pcap is another big win, as you can see the
> > forwarding
> > performance of a simple FromDevice->Queue->ToDevice chain goes up by 10x
> > You can find netmap at http://info.iet.unipi.it/~luigi/netmap/
> >
> > cheers
> > luigi
> >
> > _______________________________________________
> > click mailing list
> > click at amsterdam.lcs.mit.edu
> > https://amsterdam.lcs.mit.edu/mailman/listinfo/click
> >
> >