Click performance benchmark results
Joe Elliott
joe at inetd.com
Mon Mar 4 09:33:07 EST 2002
Hello,
From the numbers below it appears the results are constrained by lack
of CPU cycles for the user level process. As the click kernel module is
capable of handling many multiples of typical 100BaseT performance:
ie a typical packet size of 512bytes gives around 25k PPS,
it would suggest that click.o kernel mod could give a little
CPU time back. It must be it a tight loop consuming all the CPU mostly
polling an empty interface.
Can you supply a switch or /proc interface to set a sleep/yield call that
goes in the core loop to reduce the CPU utilzation? You could use select()
to do a very small delay. I have used this technique sucessfully in the
past.
You can set the logic to only do the delay if the result of the last poll
was null. This way it does not sleep when there is traffic present, only
when there are quiet times.
If you cant do this, can you suggest a location for me to place the code?
I want to constrain click.o to only consume 50% of the CPU. Is the
standard select() call available in the kernel space? are any special
libs required to link against?
The config I want to perfect is:
FromLinux(joe0, 10.0.0.1/8) -> Idle;
PollDevice(eth1, 1) -> ToLinuxSniffers(joe0);
# tcpdump -s 1600 -i joe0 -w testJoe.bin
I want to avoid going to a dual proc system if possible.
Is there a way to insert a large buffer for packets to be processed
by ToLinuxSniffers()?
Has anyone else built a successful solution for this config or similar
variation? I am looking for better performance than vanilla libpcap can
provide.
Joe.
--
__o _~o __o
`\<, `\<, `\<,
______________________________________(*)/_(*)__(*)/_(*)__(*)/_(*)________
Im a 21st Century Digital Boy ... I aint got a life, but I got lotsa toys.
******************* Joe Elliott joe at inetd.com ********************
Phone:(650)961-6631 Cell:(650)714-3932 Inetd.Com http://inetd.com
--------------------------------------------------------------------------
On Fri, 1 Mar 2002, Brecht Vermeulen wrote:
> Date: Fri, 01 Mar 2002 19:46:58 +0100
> From: Brecht Vermeulen <brecht.vermeulen at rug.ac.be>
> To: Joe Elliott <joe at inetd.com>
> Cc: click at amsterdam.lcs.mit.edu
> Subject: Re: Click performance benchmark results #1
>
>
> Hi Joe,
>
>
> There must be something wrong (I don't know what yet though, gcc
> compiler on redhat buggy ?).
>
> On a K6 500 MHz, we can route 135000 pps.
> With the PollDevice(eth1,1)->Counter->Discard; we can without any drop
> sniff 150000pps (which was link speed at 100 Mbit/s, 64 byte packets).
> So probably more.
>
> As traffic generator we use Smartbits equipment (hardware), for routing
> it can count the packets received also.
>
> On a Athlon 1GHz, we can route 180000pps with very little loss, see
> chart in attachment. (measurements are during 10s). Load is in % of link
> speed (100 MBit/s). And we used 2 streams through 4 interfaces.
>
> We use a Dlink 570 card (4port card with tulip driver).
>
> However on the K6 500 MHz, with
> PollDevice->Counter->ToLinuxSniffers;
> and tcpdump -i eth1 -w /dev/null (so pure transfer from kernel to
> user)
>
> we get the following results:
> (sorry for the layout)
> first number is the number of packets sent in kpps
> second number is the number reported by Counter
> third number is the number of packets counted by tcpdump
> measurements are 10 seconds.
>
> thus e.g.
>
> 3 # 3000 pps
> 30000 # Counter in click
> 29961 # tcpdump
>
> 3.3
> 33000
> 32793
>
> 3.5
> 35000
> 32747
>
> 4
> 40000
> 32636
>
> 4.5
> 45000
> 32622
>
> 5
> 50000
> 32512
>
> 10
> dump 99980
> tcpdump 31896
>
> 20
> 199866
> 30744
>
> 30
> 299742
> 29520
>
> 40
> 399404
> 28260
>
> 50
> 499353
> 27036
>
> 60
> 599188
> 25812
>
> 70
> 696885
> 24516
>
> 80
> 798858
> 23292
>
> 90
> 898141
> 22068
>
> 130
> 1289775
> 17496
>
> are you sure of the rates of your software traffic generator ?
>
> I'm waiting for a dual processor machine to run click on one processor
> and tcpdump on the other. Other possible improvements are just
> transporting the header to user space.
>
> tcpdump via pcap without click still transfers the whole packet I think
> and filtering is in user space (I saw in the source code about a bug
> that the packet size is set by Linux otherwise to 68). With latest
> versions of libpcap and 2.2.x kernels filtering is in kernel space, but
> still the whole packet is transferred.
>
> I've got the same problem with our 8139too ethernet cards and FromDevice
> regarding the manual ifconfig ethx promisc.
>
> I don't know what is wrong with your setup really, are the results the
> same for test #1 with tulip and intel ?
>
> (we use 2.2.19).
>
> regards,
> Brecht
>
> >
> > S/W traffic generator sending 1M pkts at 50Mb/s with 96Byte pings (~ 71K pps)
> > All tests use the same traffic source and config.
> >
> > Click host - Dell Dimension 8100 1.4GHZ/256MB
> >
> > Test #1 - Just counting packets with kernel module.
> > -------
> > PollDevice(eth1,1) -> Counter -> Discard - 715368 (~28% loss)
> > FromDevice(eth1,1) -> Counter -> Discard - 358281 (~64% loss)
> > PollDevice(eth2,1) -> Counter -> Discard - 707318 (~29% loss)
> > FromDevice(eth2,1) -> Counter -> Discard - would not work? - count=0
> > (same problem as below in test 3.4?)
> >
> > Test #2 - Using pcap - no click code loaded.
> > -------
> > tcpdump -pi eth1 -w test1.bin - 259502 packets received (~74% loss)
> > tcpdump -pi eth2 -w test2.bin - 342921 packets received (~65% loss)
> >
> > Test #3 - Copying traffic to user level socket with Poll and From methods.
> > -------
> > # TULIP Poll
> > FromLinux(joe0, 10.0.0.1/8) -> Idle;
> > PollDevice(eth1, 1) -> ToLinuxSniffers(joe0);
> > tcpdump -i joe0 -w testJoe.bin - 74752 packets received (~93% loss)
> >
> > # TULIP From
> > FromLinux(joe0, 10.0.0.1/8) -> Idle;
> > FromDevice(eth1, 1) -> ToLinuxSniffers(joe0);
> > tcpdump -i joe0 -w testJoe.bin - 197672 packets received (~80% loss)
> >
> > # Intel Poll
> > FromLinux(joe0, 10.0.0.1/8) -> Idle;
> > PollDevice(eth2, 1) -> ToLinuxSniffers(joe0);
> > tcpdump -pi joe0 -w test2.bin - 10116 packets received (~99% loss)
> >
> > # Intel From
> > FromLinux(joe0, 10.0.0.1/8) -> Idle;
> > FromDevice(eth2) -> ToLinuxSniffers(joe0);
> > tcpdump -i joe0 -w testJoe.bin - 176512 packets received (~82% loss)
> > for this I have to put eth2 into promisc mode manually - 'ifconfig eth2 promisc'
> > or I get no traffic.
> >
> > All traffic is going thro a Netgear 10/100 hub. clean 100Mbit traffic. No collisions seen.
> >
> > Any ideas on how I can improve performance or test where the problem is?
> > I am willing to run tests if someone can send me sample configs. Am I
> > correct in assuming I should get all traffic with polling to wirespeed?
> > ie up to ~150K PPS per interface?
> >
> > Look forward to any help! -- Joe.
> >
> > --
> > __o _~o __o
> > `\<, `\<, `\<,
> > ______________________________________(*)/_(*)__(*)/_(*)__(*)/_(*)________
> > Im a 21st Century Digital Boy ... I aint got a life, but I got lotsa toys.
> > ******************* Joe Elliott joe at inetd.com ********************
> > Phone:(650)961-6631 Cell:(650)714-3932 Inetd.Com http://inetd.com
> > --------------------------------------------------------------------------
> >
> > On Fri, 1 Mar 2002, Brecht Vermeulen wrote:
> >
> > > Date: Fri, 01 Mar 2002 08:25:44 +0100
> > > From: Brecht Vermeulen <brecht.vermeulen at rug.ac.be>
> > > To: Joe Elliott <joe at inetd.com>
> > > Cc: click at amsterdam.lcs.mit.edu
> > > Subject: Re: Best Click Linux distribution?
> > >
> > >
> > > Hi Joe,
> > >
> > > >
> > > > If I use click module with polling interface on tulip and do the same 100K pkts
> > > > I get far less. I was hoping for a BIG improvement.
> > > >
> > > > # cat > /proc/click/config
> > > > FromLinux(joe0, 10.0.0.1/8) -> Idle;
> > > > FromDevice(eth1, 1) -> Print(ok) -> ToLinuxSniffers(joe0);
> > > >
> > >
> > > mmm, for polling you need to use PollDevice instead of FromDevice and I
> > > think the Print(ok) does also insert a big delay if you do this for each
> > > packet (logging to syslog).
> > >
> > > > # tcpdump -npi joe0 -w dump.bin
> > > > tcpdump: listening on joe0
> > > > 33045 packets received by filter (around 65% loss)
> > > >
> > > > Any ideas? How can I just count packets recieved by click ?
> > > >
> > >
> > > simply use counter, it will also calculate the rate for you.
> > >
> > > Just use e.g. PollDevice(eth1,1) -> Counter -> ToLinuxSniffers();
> > >
> > > We are also interested in your results, so if you could post something
> > > if you have interesting (faster :-) ) results. I have tried the same but
> > > only on a singleprocessor for now (multiprocessor would be done in a few
> > > weeks) and there I thought indeed the click-tcpdump was slower because
> > > click takes all processing power.
> > >
> > > regards,
> > > Brecht
> > >
More information about the click
mailing list