click perform effects with respect to RAM

[Will Stockwell]

This thread began as a discussion of memory, but memory is not particular
pertinent to my current questions.  I'm having performances issues that I
can't seem to get a handle on.  I'm running the click configuration listed
below using 3 e1000s with the polling drivers in a 1.7 Ghz P4 w/ 512 MB of
memory.  Essentially, we're splitting a stream of traffic coming in on one
interface between two outgoing interfaces based on subnet masks with an
IPClassifier.  My configuration includes counters which give me the
following rates:

121304.883
16733.301
19740.43

In packets per second, the first is input and the following two are output
on each of the outgoing interfaces obtained from counters in my
configuration.  The packet rate is much higher than we had originally
presumed (80000 or so), but outgoing rates would still be lacking at 80000
pps.  I have a queue for each outgoing interface, each is showing drops
accumulate very quickly.

I've tried tweaking various things to improve on this:  queue size, burst
rate on my ToDevice elements, a ScheduleInfo element scheduling ToDevice
elements more often than PollDevice element on the incoming interface.
None of these seemed to affect my rates or packet loss much at all.
Adding a PollDevice which discards packets on the outgoing interfaces
seemed to improve things by a couple thousand packets per second (to the
numbers seen above).

An important side note, I also tried mirroring directly from the incoming
interface directly to a single outgoing interface.  My outgoing rates in
that situation were very similar to the sum of the outgoing rates shown
above.

A little more info abut the setup, all three NICs are on the same PCI bus,
which I believe is of the 33mhz variety.  The two outgoing interfaces are
connected by a crossover cable because they wouldn't actually transmit
packets otherwise.  I don't know if that is relevant at all, but just in
case.  I have no other equipment to which I can currently connect those
interfaces for the same effect.

To the point, what's slowing me down?  My configuration seems fairly
simple, and based on my previous exchange with Eddie my sense was that
this type of hardware ought to be able to handle this amount of traffic
well enough, certainly better than ~30,000 packets per second it's
putting out.

Any thoughts or ideas?  Things I might try?


Will

// Begin configuration
// We're going to classify traffic based on a simple subnet mask
// I omitted the actual netmasks
split :: IPClassifier(subnet A mask,
		      subnet B mask,
			-);

// Output queues to device including counters for useful info
// These PollDevice elements seem to speed things up a bit for the
// outgoing streams
PollDevice(eth2) -> Discard;
PollDevice(eth3) -> Discard;
outq1 :: Queue(20000) -> e2ctr :: Counter -> td2 :: ToDevice(eth2, 5);
outq2 :: Queue(20000) -> e3ctr :: Counter -> td3 :: ToDevice(eth3, 5);

// Poll the incoming pipe traffic,
// mark the IP header location
// pass it into the IPClassifier
pd :: PollDevice(eth1, true) -> inctr :: Counter -> MarkIPHeader(14) -> split;

// subnet A to outq1
// subnet B to outq2
// All else discarded
split[0] -> outq1;
split[1] -> outq2;
split[2] -> disctr :: Counter -> Discard;

// End configuration

On Fri, 17 Jan 2003, Eddie Kohler wrote:

> Hi Will,
>
> > In reading the TOCS Click paper, the performance section doesn't mention
> > the amount of memory used in the forwarding system.
>
> The TOCS paper does mention this about memory:
>
>   1295 instructions are retired during the forwarding of a packet, implying
>   that significantly more complex Click configurations could be supported
>   without exhausting the Pentium III's 16K L1 instruction cache.
>
> d-cache usage, of course, is mostly due to packets, and so shows up in the
> forwarding rate graphs.
>
> > Am I to interpret this as, the amount of memory does not significantly
> > affect the perform within reason? Or should I assume that it can have
> > affect, but to what extent it does is dependent largely on the
> > configuration I'm using?
>
> Amount of memory is important -- the performance limitation we see may be
> due to memory system limits (depending on the machine). Your 2nd assumption
> is a good one.
>
> > We're pushing about 80,000 packets per second onto eth0.  The system we're
> > thinking of running click on is a 1.4 Ghz P4 with 512 MB of RAM.  We don't
> > have the memory at present, so I was curious about necessity of such an
> > amount for this kind of use.
>
> I think this system will be totally fine for your usage, as long as your
> Ethernet cards are pollable (tulip-based 100MHz Ethernet, or Intel E1000
> gig-E cards). You may squeak by even without polling.
>
> Let us know if you have problems!
>
> Eddie
>