[Click] SMP SpinLock and Deadlocks

[Paine, Thomas Asa]

Beyers,

The last configuration used was...

./configure --prefix=/backups/staging/click --disable-userlevel
--enable-multithread=2 --enable-intel-cpu --disable-dynamic-linking

Adding the additional output in sync.hh of course generated what I would
suspect...
.
.
releasing someone else's lock - owner 0 current cpu 1
releasing someone else's lock - owner 1 current cpu 0
releasing someone else's lock - owner 0 current cpu 1
releasing someone else's lock - owner 1 current cpu 0
.
.
 
	As of now, I am not using any additional click-install
parameters when the module is installed.  Adding threads would just
aggravate the situation.  Likewise, since there is only one thread, I am
not using the StaticThreadSched option to bind elements to threads.
	Right now I am using the sched_setaffinity() system call from
another process to set the affinity for the click PID, gathered from
/proc/click/threads after click is running, to keep the thread on one
virtual CPU, and the system stable.

Thanks, 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
   Thomas Paine (paineta at uwec.edu) 
   University of Wisconsin - Eau Claire 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
 
 

________________________________

From: Beyers Cronje [mailto:bcronje at gmail.com] 
Sent: Saturday, September 16, 2006 6:27 AM
To: Paine, Thomas Asa
Cc: click at pdos.csail.mit.edu
Subject: Re: [Click] SMP SpinLock and Deadlocks


HI Thomas,

What is your click './configure' line used and also what click-install
parameters are you specifying? Are you using StaticThreadSched in your
click configuration? 

If you are using StaticThreadSched change the Spinlock::Release line to:


click_chatter("releasing someone else's lock - owner %i current cpu %i",
_owner, my_cpu);

This should help narrow down if the problem is in the push/pull path or
timer/handler related.

Beyers 


On 9/12/06, Paine, Thomas Asa <PAINETA at uwec.edu> wrote: 


	        I have a click package, which contains my custom
elements.
	Those elements basically maintain maps of IP and data.  To
protect the
	maps during read/write handler operations, I have wrapped the
critical
	sections using the SpinLock found in the sync.hh.
	
	My typical usage...
	
	   ...
	   _lock.acquire();
	   bleMap->clear();
	   _lock.release();
	   ...
	
	
	   ...
	   _lock.acquire();
	   bleMap->insert(n, type); 
	   _lock.release();
	   ...
	
	
	
	However, I'm experiencing periodic deadlocks and am seeing these
types
	of messages very frequently...
	
	        "chatter: releasing someone else's lock" ...which is
coming from 
	the SpinLock
	
	I'm seeing these messages even when handlers are not being
called and I
	nice it a bit...  "echo -19 > /click/priority", so I'm not sure
of the
	scope of the problem.  Clearly calling _depth-- on the wrong
lock would 
	lead to accounting problems with the semaphore and deadlock.
	        The configuration is currently running on an appliance
using the
	latest CVS code and the 2.6.16.3 kernel.  The appliance is using
an 
	Intel(R) Pentium(R) 4 CPU 3.20GHz with hyperthreading, and the
kernel is
	running with SMP support.  This configuration has been run on
SMP
	enabled machines before, but with 2 physical processors, and was
still
	using the 2.4 kernel, so I don't want to make too many
comparisons
	there.  I can say I have not experienced any behavior like this
in the
	past with the my elements or with click.
	
	The click and custom packages were last configured using the
following, 
	and I've tried several variations of config options to see if I
could at
	least cause a change, but I haven't seemed to find what's
broken.
	
	"./configure --prefix=/backups/staging/click
--disable-userlevel" 
	
	At this point, I'm not quite sure where the problem may lie.  Is
this an
	affinity problem with the click kernel module, SpinLock,
hyperthreading,
	kernel compilation option, or am I just not configuring/using
something 
	correctly?  Has anyone else seen these messages unexpectedly?
	
	
	
	Thanks,
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	   Thomas Paine (paineta at uwec.edu)
	   University of Wisconsin - Eau Claire 
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	
	
	_______________________________________________
	click mailing list
	click at amsterdam.lcs.mit.edu
	https://amsterdam.lcs.mit.edu/mailman/listinfo/click