[Click] profiling / debugging tools or tips?

[Ian Rose]

Hello all,

I need to do some debugging and profiling of my click configuration 
which consists of a large number of both standard and custom (i.e. 
written by me) elements.  Specifically, I would like to get an idea of 
the CPU usage of each element, or at least know what the biggest 
consumers are.

Are there any tools within click to help me do this, or does anyone have 
any tips on how to go about this?  The best idea that I can come up with 
is to take a guess at which elements might be the CPU hogs and 
instrument them with getrusage() calls at every entrance and exit points 
to the element.  Unfortunately this is a bit labor-intensive (and a 
little error-prone) as there are potentially many such points (e.g. 
push(), pull(), run_timer(), run_task()).

Additionally, the resolution is pretty low (looks like 1ms on my system) 
so this *may* result in some inaccuracies.  Or may not... perhaps over 
time random chance will even things out so that over long enough periods 
each element's summed cpu time will be mostly accurate?

It would be handy to implement this instead in Click's core instead of 
in each individual element; you might imagine calling some 
initialization methods to tell Click which elements to profile for you. 
  However I don't think this is possible currently - although Click 
should be able to properly track whenever a timer or task fires (and for 
which element) I don't think click is able to track when a packet (and 
therefore control) is pushed/pulled from one element to another.  If I 
am wrong about this please correct me.

Another half-baked idea I had was something similar to how (some) OSes 
sample the process scheduler to estimate each process' load average.  So 
the idea would be to somehow "generate an interrupt" every so often and 
check which element was currently "active" at that time.  The active 
element gets 1 point and each elements "load average" is the fraction of 
points that it got over some recent time window.  But I don't really 
have an idea of how to actualize this in click; I don't really have an 
idea of how to do either the "generate an interrupt" part or 
(especially) the "figure out which element is 'active'" part.

Anyhow, before I dove into instrumenting individual elements with 
getrusage() calls I thought I'd solicit some advice on whether or not 
the idea made any sense.

cheers,
- Ian