[Click] Kernel Panic
Ritun Patney
ritun at cse.wustl.edu
Mon Jun 15 16:41:18 EDT 2009
Hey,
I have been doing some experiments using click on emulab and have been
running into this problem where the kernel panics and crashes. Initially I
thought its one of my modules, but now I have noticed it just happens
without my modules also.
The setup I have is machines with 1G ethernet ports (e1000 intel) running
click patched kernel 2.6.24.7. The version of click is 1.7.0rc1. There
are just 2 nodes involved in the experiment, one transmitter and the
other is receiving that traffic. Both are running click modules to send
and receive. The source has the following configuration:
RatedSource(DATA ...., RATE 500000)->ToDevice(ethx)
The destination has the following configuration:
FromDevice(ethx)->Counter->Discard
The kernel seems to panic at data rates where either the FromDevice
module or the interface starts to drop packets. Also, the
error seems to be to memory related. I have attached 2
error reports from 2 different runs printed on the console. In a separate
run (call it the third run), I got a huge call trace but it started
with the error "BUG: unable to handle kernel paging request at virtual
address xxxxxx". I would appreciate any help.
Thanks
-Ritun
----------------------------------------------------------------------------
First Run
----------------------------------------------------------------------------
click: starting router thread pid 4227 (f6bf1e40)
------------[ cut here ]------------
kernel BUG at mm/slab.c:3007!
invalid opcode: 0000 [#1] SMP
Modules linked in: click proclikefs i2c_dev i2c_core tulip e100 3c59x
dm_mirror nvram e1000 button
Pid: 6, comm: ksoftirqd/1 Not tainted (2.6.24.7-1.2944.fc6.emulab-1 #1)
EIP: 0060:[<c01655c9>] EFLAGS: 00010046 CPU: 1
EIP is at cache_alloc_refill+0xf3/0x49d
EAX: 00000008 EBX: f78515c0 ECX: 00000016 EDX: f7800880
ESI: f78b0c80 EDI: 00000005 EBP: f78022c0 ESP: f7871e88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process ksoftirqd/1 (pid: 6, ti=f7870000 task=f784b0c0 task.ti=f7870000)
Stack: f7808400 00000008 00000020 f7800880 f78515c0 00000000 f696b680
00000246
f8cfe4dd f8e92fe4 ffffffff 00000000 f696b580 f7800880 00000246
00000020
c01659e2 f696b580 00000080 f696b580 f787cdc0 c0318cca 00000000
00000020
Call Trace:
[<f8cfe4dd>] packet_notifier_hook+0x5d/0x80 [click]
[<c01659e2>] __kmalloc+0x6f/0xa5
[<c0318cca>] __alloc_skb+0x53/0xff
[<c031996c>] __netdev_alloc_skb+0x1c/0x35
[<f88793b8>] e1000_clean_rx_irq+0x1b1/0x44c [e1000]
[<f8879207>] e1000_clean_rx_irq+0x0/0x44c [e1000]
[<f8876dab>] e1000_clean+0x59/0x1e6 [e1000]
[<c031efed>] net_rx_action+0x8d/0x174
[<c01288ed>] run_timer_softirq+0x2f/0x156
[<c012566e>] __do_softirq+0x5d/0xc1
[<c0125954>] ksoftirqd+0x0/0xa4
[<c0125704>] do_softirq+0x32/0x36
[<c0125996>] ksoftirqd+0x42/0xa4
[<c0131448>] kthread+0x38/0x60
[<c0131410>] kthread+0x0/0x60
[<c01050c7>] kernel_thread_helper+0x7/0x10
=======================
Code: 8b 75 00 39 ee 75 15 8b 75 10 8d 45 10 c7 45 34 01 00 00 00 39 c6 0f
84 a5 00 00 00 8b 54 24 0c 8b 82 98 00 00 00 39 46 10 72 37 <0f> 0b eb fe
8b
44 24 0c 8b 5c 24 10 8b 90 8c 00 00 00 8b 44 24
EIP: [<c01655c9>] cache_alloc_refill+0xf3/0x49d SS:ESP 0068:f7871e88
Kernel panic - not syncing: Fatal exception in interrupt
----------------------------------------------------------------------------
Second Run
----------------------------------------------------------------------------
Fedora Core release 6 (Zod)
Kernel 2.6.24.7-1.2944.fc6.emulab-1 on an i686
node0.eval.forest.emulab.net login: click: starting router thread pid 3889
(f576dc40)
------------[ cut here ]------------
kernel BUG at mm/slab.c:602!
invalid opcode: 0000 [#1] SMP
Modules linked in: click proclikefs i2c_dev i2c_core tulip e100 3c59x
dm_mirror nvram e1000 button
Pid: 3889, comm: kclick Not tainted (2.6.24.7-1.2944.fc6.emulab-1 #1)
EIP: 0060:[<c0165034>] EFLAGS: 00010046 CPU: 0
EIP is at free_block+0x4d/0x104
EAX: 40000000 EBX: 0000001c ECX: f5621800 EDX: c16ac420
ESI: f5405dc0 EDI: f78022c0 EBP: f7800880 ESP: f73b7ee8
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process kclick (pid: 3889, ti=f73b6000 task=f5592330 task.ti=f73b6000)
Stack: 00000007 00000000 0000001b 00000018 f78054f4 f78022c0 000000d8
f7800880
0000001b c0164e7d 00000000 f7805480 f7805480 00000286 f540b800
0047b8a4
c0164f0b f56b4a80 f702d180 00000001 c0318570 f8e936e0 f8d31991
00000000
Call Trace:
[<c0164e7d>] cache_flusharray+0x71/0x98
[<c0164f0b>] kfree+0x67/0x76
[<c0318570>] __kfree_skb+0x8/0xd
[<f8d31991>] _ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
[<f8ca7286>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click]
[<f8cfe628>] _ZN10FromDevice8run_taskEP4Task+0x78/0xf0 [click]
[<f8cbfe35>] _ZN12RouterThread6driverEv+0x235/0x470 [click]
[<c011e788>] sched_move_task+0xc1/0xc7
[<c01291cb>] switch_uid+0x5c/0x69
[<f8ca5a54>] click_lalloc+0x34/0x60 [click]
[<f8d2dc59>] _Z11click_schedPv+0xc9/0x1b0 [click]
[<f8d2db90>] _Z11click_schedPv+0x0/0x1b0 [click]
[<c01050c7>] kernel_thread_helper+0x7/0x10
=======================
Code: 8b 08 8d 91 00 00 00 40 c1 ea 0c c1 e2 05 03 15 80 f6 51 c0 8b 02 25
00 40 02 00 3d 00 40 02 00 75 03 8b 52 0c
8b 02 84 c0 78 04 <0f> 0b eb fe 8b 72 1c 8b 54 24 28 8b 46 04 8b bc 95 c8
00 00 00
EIP: [<c0165034>] free_block+0x4d/0x104 SS:ESP 0068:f73b7ee8
---[ end trace e44c05cd35086d7b ]---
More information about the click
mailing list