[Click] Poor forwarding performance with click in a paravirtualized Xen environment

[Giorgio Calarco]

Hi Richard,

I understand your motivations, and that's why I suggested to slow down
your generator and see what happens since I had the
same suspect introduced by Luigi Rizzo, (receive livelock).
Probably if you increase the packet rate progressively,
you should be able to clarify this. I would also try (during
these tests) to assign a predefined cpu core to the vm
and follow its load.
This seems similar to my brief experience with
 Click virtualized within Linux Containers,
I couldn't overcome 60.000pps. When I rised up the rate
further, the forwarding performance went down very quickly.

A curiosity: which hardware platform are you using exactly ?

Ciao, Giorgio


On Thu, Mar 21, 2013 at 12:31 PM, Richard Neumann
<mail at richard-neumann.de>wrote:

> Hello again,
>
> @Giorgio Calarco:
>         Hi Richard, what happens if you decrease the input rate at lower
>         values? For instance at 40000pps? (maybe you have to change your
>         packet generator to perform this experiment ). And, where is
>         your brctl-based bridge created ? Within dom0?
>         Let me know,
>
> I'll try to run a test with a slower packet generator and will let you
> know how it performs then. But the problem is that we actually do want
> to do high-speed packet processing here.
> The brige is created inside the domU environment, connecting the two 10G
> interfaces which have been passed through from the dom0.
>
> @Luigi Rizzo:
> > At first sight it seems a case of receive livelock, which does not occur
> > so badly with the linux bridge as it (probably) is helped by NAPI.
> >
> > It is not clear if your userspace click is also using netmap (which
> > may have its own bugs) and/or whether the two interfaces eth1 and
> > eth2 are using pci-passthrough or are emulated/paravirtualized.
>
> in the respective setup, click is is not using netmap and even was
> compiled without the --with-netmap stuff.
> The 10G Cards are using pci-passthrough, because I expected better
> performance from this than from emulation or paravirtualization  (I did
> not actually compare it yet).
>
> > In any case the 530Kpps you get with linux bridge is probably close
> > to the peak performance you can get, unless (a) Xen gives you access
> > to the real hw (via virtual functions and/or pci-passthrough), and
>
> So, for I am actually am using PCI passthrough, the performance should
> be better even using a linux bridge.
>
> > (b) you are using netmap or some very fast os stack bypass to talk
> > to the network interfaces within the virtual machine.
> >
> > I managed to go quite fast within qemu+kvm, see below
> >
> >     http://info.iet.unipi.it/~luigi/netmap/talk-google-2013.html
> >     http://info.iet.unipi.it/~luigi/papers/20130206-qemu.pdf
> >
> > but that needed a little bit of tweaking here and there
> > (not that i doubt that the Xen folks are smart enough to do
> > something similar).
>
> Thank you both for your help so far.
>
> Best regards,
>
> Richard
>
>
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>