[Click] Poor forwarding performance with click in a paravirtualized Xen environment

[Richard Neumann]

Hello again,

@Giorgio Calarco:
        Hi Richard, what happens if you decrease the input rate at lower
        values? For instance at 40000pps? (maybe you have to change your
        packet generator to perform this experiment ). And, where is
        your brctl-based bridge created ? Within dom0? 
        Let me know,
        
I'll try to run a test with a slower packet generator and will let you
know how it performs then. But the problem is that we actually do want
to do high-speed packet processing here.
The brige is created inside the domU environment, connecting the two 10G
interfaces which have been passed through from the dom0.

@Luigi Rizzo:
> At first sight it seems a case of receive livelock, which does not occur
> so badly with the linux bridge as it (probably) is helped by NAPI.
> 
> It is not clear if your userspace click is also using netmap (which
> may have its own bugs) and/or whether the two interfaces eth1 and
> eth2 are using pci-passthrough or are emulated/paravirtualized.

in the respective setup, click is is not using netmap and even was
compiled without the --with-netmap stuff.
The 10G Cards are using pci-passthrough, because I expected better
performance from this than from emulation or paravirtualization  (I did
not actually compare it yet).

> In any case the 530Kpps you get with linux bridge is probably close
> to the peak performance you can get, unless (a) Xen gives you access
> to the real hw (via virtual functions and/or pci-passthrough), and

So, for I am actually am using PCI passthrough, the performance should
be better even using a linux bridge.

> (b) you are using netmap or some very fast os stack bypass to talk
> to the network interfaces within the virtual machine.
> 
> I managed to go quite fast within qemu+kvm, see below
> 
>     http://info.iet.unipi.it/~luigi/netmap/talk-google-2013.html
>     http://info.iet.unipi.it/~luigi/papers/20130206-qemu.pdf
> 
> but that needed a little bit of tweaking here and there
> (not that i doubt that the Xen folks are smart enough to do
> something similar).

Thank you both for your help so far.

Best regards,

Richard