[Click] Click Spinlock Question

[David Zats]

Hi
I am installing click using:
./configure --enable-multithread --enable-kassert --enable-warp9 ; make ; sudo make install

And running it in kernel mode using:
sudo click-install -t 4 --cpu=0 router.click

With a configuration that is derived from the sample router creation one (ie. /conf/make-ip-conf.pl). Additionally, I have added BalancedThreadSched.

The code implements a queue that primary has the following locking strategy:

/* header file */
Spinlock lock;

/* queue module */
push() {
	lock.acquire();
	//Add to queue
	lock.release();
	if(need to contact another queue) {
		ptr_other_queue->enqueue_special_packet() /* this calls locks on the other queue */
	}
}

pull() {
	Packet* p;
	lock.acquire()
	p = dequeue();
	lock.release();
	return p
}

run_task() {
	lock.acquire()
	//Perform maintenance
	lock.release()
}

When running this code, I experience soft Lockup. For example:
[ 3122.764275] Pid: 4185, comm: kclick Not tainted (2.6.24.7 #1)
[ 3122.764277] EIP: 0060:[<c03b78e7>] EFLAGS: 00000286 CPU: 1
[ 3122.764279] EIP is at _spin_lock+0x7/0x10
[ 3122.764281] EAX: f7f1a200 EBX: 00000001 ECX: c0356514 EDX: df44b218
[ 3122.764283] ESI: f7f1a000 EDI: f7f1a218 EBP: f7c3a000 ESP: f598be80
[ 3122.764285]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 3122.764286] CR0: 8005003b CR2: bfedbfdc CR3: 0056b000 CR4: 00000690
[ 3122.764288] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 3122.764290] DR6: ffff0ff0 DR7: 00000400
[ 3122.764291]  [<c0356cc9>] dev_watchdog+0x19/0xd0
[ 3122.764298]  [<c0135c49>] run_timer_softirq+0x169/0x1e0
[ 3122.764302]  [<c0356cb0>] dev_watchdog+0x0/0xd0
[ 3122.764307]  [<c0356cb0>] dev_watchdog+0x0/0xd0
[ 3122.764311]  [<c0131de2>] __do_softirq+0x82/0x110
[ 3122.764316]  [<c0131ec5>] do_softirq+0x55/0x60
[ 3122.764320]  [<c01321ad>] irq_exit+0x6d/0x80
[ 3122.764323]  [<c0118885>] smp_apic_timer_interrupt+0x55/0x80
[ 3122.764329]  [<c01054b8>] apic_timer_interrupt+0x28/0x30
[ 3122.764334]  [<c014007b>] sys_timer_create+0x32b/0x330
[ 3122.764338]  [<fbb7f366>] _ZN10DelayQueue8run_taskEP4Task+0x96/0x180 [click]
[ 3122.764554]  [<fbb1988a>] _ZN12RouterThread6driverEv+0x25a/0x5b0 [click]
[ 3122.764759]  [<fbaf0640>] _ZN6VectorIPvE7reserveEi+0x80/0xb0 [click]
[ 3122.764957]  [<fbbb8146>] _Z11click_schedPv+0xc6/0x250 [click]
[ 3122.765158]  [<fbbb8080>] _Z11click_schedPv+0x0/0x250 [click]
[ 3122.765359]  [<c0105657>] kernel_thread_helper+0x7/0x10
[ 3122.765364]  =======================
[ 3123.804546] BUG: soft lockup - CPU#3 stuck for 11s! [kclick:4187]
[ 3123.810649] 
[ 3123.810651] Pid: 4187, comm: kclick Not tainted (2.6.24.7 #1)
[ 3123.810653] EIP: 0060:[<fbb7e066>] EFLAGS: 00000202 CPU: 3
[ 3123.810853] EIP is at _ZN10DelayQueue4pullEi+0x96/0xd0 [click]
[ 3123.810856] EAX: 00000001 EBX: df918800 ECX: df918840 EDX: 00000003
[ 3123.810859] ESI: 00000001 EDI: 00000000 EBP: f7f1a000 ESP: f59d1f44
[ 3123.810863]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 3123.810866] CR0: 8005003b CR2: b7d3c104 CR3: 0056b000 CR4: 00000690
[ 3123.810869] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 3123.810873] DR6: ffff0ff0 DR7: 00000400
[ 3123.810878]  [<fbb6fabc>] _ZN8ToDevice8run_taskEP4Task+0xdc/0x580 [click]
[ 3123.811085]  [<fbb1988a>] _ZN12RouterThread6driverEv+0x25a/0x5b0 [click]
[ 3123.811287]  [<c013682b>] switch_uid+0x5b/0x70
[ 3123.811295]  [<fbbb8146>] _Z11click_schedPv+0xc6/0x250 [click]
[ 3123.811495]  [<fbbb8080>] _Z11click_schedPv+0x0/0x250 [click]
[ 3123.811693]  [<c0105657>] kernel_thread_helper+0x7/0x10
[ 3123.811699]  =======================

Also, low latencies are really important to me, so I really do not want to run this in user space. Thanks in advance for any help you can provide.

Sincerely,
David Zats



On Feb 28, 2011, at 1:28 PM, Beyers Cronje wrote:

> Hi David,
> 
> Can you post some of your code and a simple config that procudes problems for you? I'm running user-level click in multithreaded mode with Spinlocks quite successfully when running the latest code from git that includes recent multithreading fixes.
> 
> Beyers
> 
> On Mon, Feb 28, 2011 at 8:45 PM, David Zats <dzats at eecs.berkeley.edu> wrote:
> Hi,
> I have been trying to use Spinlocks in Click and have been running into problems. I recently realized that I was running 4 threads when I only had 2 cores. Could this be a potential problem? Also, does hyperthreading need to be disabled for Click to work? If neither of these are potential problems, could you suggest some other common pitfalls? I am using just one lock, so re-ordering cannot be an issue and am sure that I am releasing it after every acquisition. Furthermore, SpinlockIRQ seems to work well as does running click using only one thread.
> Thank you very much for the help!
> Sincerely,
> David Zats
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>