[Click] profiling / debugging tools or tips?

[Ian Rose]

Well now, I'm certainly glad I asked first!  I just happened to find my 
current bug leading to the CPU starvation, but when I get a moment I'm 
interested to try running my configs with this enabled just to see what 
the overhead is like.  I'm running on some moderately weak hardware 
(Soekris net4826) so I don't have a lot of spare cycles, but my packet 
rates are also much less than what many Click users probably deal with 
so maybe it'll still work...

thanks for the pointer!
- Ian


Cliff Frey wrote:
> click already supports exactly what you are talking about!
> 
> if you configure it with
> ./configure --enable-stats=2
> 
> then every element is given a 'cycles' read handler.
> 
> However, using generic profiling tools is likely to also give you 
> feedback (with much lower overhead).  Perhaps take a look at oprofile as 
> a package for doing this.
> 
> At the other end of the spectrum, you can use callgrind (which is part 
> of the valgrind package) along with kcachegrind to give you a break down 
> of the exact number of CPU _instructions_ spent in each of your 
> elements, along with per task/stack information.  That information can 
> be very colorful and interesting to look at, however it tends not 
> not-very-closely line up with reality because instruction count is not a 
> great proxy for wallclock time.  Also, running through callgrind with 
> slow down your application by 50x or so, but it it can still be very 
> interesting to see.
> 
> Cliff
> 
> On Wed, Mar 17, 2010 at 5:08 PM, Ian Rose <ianrose at eecs.harvard.edu 
> <mailto:ianrose at eecs.harvard.edu>> wrote:
> 
>     Hello all,
> 
>     I need to do some debugging and profiling of my click configuration
>     which consists of a large number of both standard and custom (i.e.
>     written by me) elements.  Specifically, I would like to get an idea of
>     the CPU usage of each element, or at least know what the biggest
>     consumers are.
> 
>     Are there any tools within click to help me do this, or does anyone have
>     any tips on how to go about this?  The best idea that I can come up with
>     is to take a guess at which elements might be the CPU hogs and
>     instrument them with getrusage() calls at every entrance and exit points
>     to the element.  Unfortunately this is a bit labor-intensive (and a
>     little error-prone) as there are potentially many such points (e.g.
>     push(), pull(), run_timer(), run_task()).
> 
>     Additionally, the resolution is pretty low (looks like 1ms on my system)
>     so this *may* result in some inaccuracies.  Or may not... perhaps over
>     time random chance will even things out so that over long enough periods
>     each element's summed cpu time will be mostly accurate?
> 
>     It would be handy to implement this instead in Click's core instead of
>     in each individual element; you might imagine calling some
>     initialization methods to tell Click which elements to profile for you.
>      However I don't think this is possible currently - although Click
>     should be able to properly track whenever a timer or task fires (and for
>     which element) I don't think click is able to track when a packet (and
>     therefore control) is pushed/pulled from one element to another.  If I
>     am wrong about this please correct me.
> 
>     Another half-baked idea I had was something similar to how (some) OSes
>     sample the process scheduler to estimate each process' load average.  So
>     the idea would be to somehow "generate an interrupt" every so often and
>     check which element was currently "active" at that time.  The active
>     element gets 1 point and each elements "load average" is the fraction of
>     points that it got over some recent time window.  But I don't really
>     have an idea of how to actualize this in click; I don't really have an
>     idea of how to do either the "generate an interrupt" part or
>     (especially) the "figure out which element is 'active'" part.
> 
>     Anyhow, before I dove into instrumenting individual elements with
>     getrusage() calls I thought I'd solicit some advice on whether or not
>     the idea made any sense.
> 
>     cheers,
>     - Ian
>     _______________________________________________
>     click mailing list
>     click at amsterdam.lcs.mit.edu <mailto:click at amsterdam.lcs.mit.edu>
>     https://amsterdam.lcs.mit.edu/mailman/listinfo/click
> 
>