[Click] 10GE Click performance

[Luigi Rizzo]

On Wed, Jan 25, 2012 at 05:18:21PM -0500, Eddie Kohler wrote:
> Hi Luigi,
> 
> So just to check: Does #3 include our allocator fixes?

yes it does (at least in the initial version i sent you -- your
code was marginally faster than mine but if i remember well
the difference only showed up in the infinitesource->sink case)

> It might be possible to get rid of at least one of the copies in #3.

yes but i believe it might require the use of the native netmap API
(unless you processg the packet to completion within pcap_dispatch()).

A suggestion i have received (but not tried yet) is to make buffers
(2k-64) bytes so they make a better use of the cache.

Right now, with 2k netmap buffers and 64-byte cache lines, the
initial part of the packet maps to 1/16 of the available cache
space, and data touching ops might cause extra cache misses (i am
not using explicit prefetch, so read misses are going to be there
anyways; but writes could be affected). Making the buffers shorter
would spread them more evenly among cache lines.

I don't remember how large are the packet buffers in Click and
whether this can make a difference. I was also not particularly
excited to make a change just to look good in benchmarks, but
perhaps there is a point because the initial part of the packet is
the one with headers, so there is going to be a lot more contention
there than in other parts of the buffer.

cheers
luigi

> Eddie
> 
> 
> On 1/20/12 5:12 PM, Luigi Rizzo wrote:
> >On Fri, Jan 20, 2012 at 10:21:15PM +0200, Beyers Cronje wrote:
> >>Hi Luigi,
> >>
> >>Have you guys released the driver for linux yet, or is it still a work in
> >>progress? How does the libpcap emulation compare performance wise with a
> >>native netmap driver?
> >
> >the linux driver is not ready yet, though today we managed to
> >send the first packets with linux-netmap. I am hoping we can get
> >something ready for distribution  in 2-3 weeks.
> >
> >As for native vs libpcap, here are the most recent numbers i have are
> >for something equivalent to FromDevice ->  Queue ->  ToDevice :
> >
> >   1.	pure netmap (zero copy)		14.2 Mpps @ 1.6  GHz
> >   2.	pcap_dispatch(pcap_inject())	 8.5 Mpps @ 2.93 GHz
> >   3.	click userspace+netmap/libpcap	 3.95Mpps @ 2.93 GHz
> >   4.	click userspace+bpf/libpcap 	 0.49Mpps @ 2.93 GHz
> >
> >In terms of data copies and memory allocations:
> >#1 is really zero-copy, #2 does one copy, #3 does two copies and
> >one allocation (and the allocator was the bottleneck, fixed last
> >summer with Eddie), #4 is copies plus one Packet allocation plus
> >one syscall plus two mbuf allocations.
> >
> >cheers
> >luigi
> >
> >	
> >
> >>Beyers
> >>
> >>On Fri, Jan 20, 2012 at 8:54 PM, Luigi Rizzo<rizzo at iet.unipi.it>  wrote:
> >>
> >>>On Fri, Jan 20, 2012 at 09:19:43AM -0800, Roman Chertov wrote:
> >>>>>
> >>>>>Kinda wonder how prepared click is for the netmap work being done and
> >>>>>10Gb performance, and if it will help
> >>>i
> >
> >>>>I recall reading on the list that netmap maintains a TX ring and as such
> >>>>it is not friendly to Click configs that can hold on to some packets for
> >>>>a while.
> >>>>
> >>>>FromDevice(ethX) ->  cl::Classifier(...)[0]
> >>>>                   ->  Queue
> >>>>                   ->  DelayShaper(DELAY)
> >>>>                   ->  outQ;
> >>>>
> >>>>cl[1] ->  outQ;
> >>>>
> >>>>Any clarifications from people familiar with netmap?
> >>>
> >>>Click uses the libpcap emulation to run on top of netmap,
> >>>so it doesn't know anything about the netmap rings.
> >>>Incoming frames are copied into Click's Packets in the callback
> >>>passed tp pcap_dispatch(), and they
> >>>can be stored for as long as you like provided there is
> >>>enough (userspace) memory.
> >>>When it is time to transmit, pcap_inject() copies the payload
> >>>from the Packet to the netmap ring and the packet goes out.
> >>>
> >>>The two copies are moderately expensive but we are still talking
> >>>3-4Mpps for FromDevice ->  EtherSwitch ->  ToDevice .
> >>>
> >>>Anything that takes a timestamp is going to hurt much more
> >>>than the memory copies.
> >>>
> >>>cheers
> >>>luigi
> >>>_______________________________________________
> >>>click mailing list
> >>>click at amsterdam.lcs.mit.edu
> >>>https://amsterdam.lcs.mit.edu/mailman/listinfo/click
> >>>
> >_______________________________________________
> >click mailing list
> >click at amsterdam.lcs.mit.edu
> >https://amsterdam.lcs.mit.edu/mailman/listinfo/click