[Click] Kernel Panic

Tue Mar 9 16:51:59 EST 2010

Hi Ritun,

Thanks for this long-ago bug report.  A number of fixes have been committed to 
the tree that may have some impact on the bug.  If you are still working on 
this problem, give the newer device drivers & Click core a try.

E

Ritun Patney wrote:
> Hey,
> I have been doing some experiments using click on emulab and have been 
> running into this problem where the kernel panics and crashes. Initially I 
> thought its one of my modules, but now I have noticed it just happens 
> without my modules also.
> 
> The setup I have is machines with 1G ethernet ports (e1000 intel) running
> click patched kernel 2.6.24.7. The version of click is 1.7.0rc1. There 
> are just 2 nodes involved in the experiment, one transmitter and the 
> other is receiving that traffic. Both are running click modules to send 
> and receive. The source has the following configuration:
> 
> RatedSource(DATA ...., RATE 500000)->ToDevice(ethx)
> 
> The destination has the following configuration:
> 
> FromDevice(ethx)->Counter->Discard
> 
> The kernel seems to panic at data rates where either the FromDevice 
> module or the interface starts to drop packets. Also, the 
> error seems to be to memory related. I have attached 2
> error reports from 2 different runs printed on the console. In a separate
> run (call it the third run), I got a huge call trace but it started 
> with the error "BUG: unable to handle kernel paging request at virtual 
> address xxxxxx". I would appreciate any help.
> Thanks
> -Ritun
> 
> ----------------------------------------------------------------------------
> First Run
> ----------------------------------------------------------------------------
> click: starting router thread pid 4227 (f6bf1e40)
> ------------[ cut here ]------------
> kernel BUG at mm/slab.c:3007!
> invalid opcode: 0000 [#1] SMP
> Modules linked in: click proclikefs i2c_dev i2c_core tulip e100 3c59x
> dm_mirror nvram e1000 button
> 
> Pid: 6, comm: ksoftirqd/1 Not tainted (2.6.24.7-1.2944.fc6.emulab-1 #1)
> EIP: 0060:[<c01655c9>] EFLAGS: 00010046 CPU: 1
> EIP is at cache_alloc_refill+0xf3/0x49d
> EAX: 00000008 EBX: f78515c0 ECX: 00000016 EDX: f7800880
> ESI: f78b0c80 EDI: 00000005 EBP: f78022c0 ESP: f7871e88
>   DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Process ksoftirqd/1 (pid: 6, ti=f7870000 task=f784b0c0 task.ti=f7870000)
> Stack: f7808400 00000008 00000020 f7800880 f78515c0 00000000 f696b680
> 00000246
>         f8cfe4dd f8e92fe4 ffffffff 00000000 f696b580 f7800880 00000246
> 00000020
>         c01659e2 f696b580 00000080 f696b580 f787cdc0 c0318cca 00000000
> 00000020
> Call Trace:
>   [<f8cfe4dd>] packet_notifier_hook+0x5d/0x80 [click]
>   [<c01659e2>] __kmalloc+0x6f/0xa5
>   [<c0318cca>] __alloc_skb+0x53/0xff
>   [<c031996c>] __netdev_alloc_skb+0x1c/0x35
>   [<f88793b8>] e1000_clean_rx_irq+0x1b1/0x44c [e1000]
>   [<f8879207>] e1000_clean_rx_irq+0x0/0x44c [e1000]
>   [<f8876dab>] e1000_clean+0x59/0x1e6 [e1000]
>   [<c031efed>] net_rx_action+0x8d/0x174
>   [<c01288ed>] run_timer_softirq+0x2f/0x156
>   [<c012566e>] __do_softirq+0x5d/0xc1
>   [<c0125954>] ksoftirqd+0x0/0xa4
>   [<c0125704>] do_softirq+0x32/0x36
>   [<c0125996>] ksoftirqd+0x42/0xa4
> [<c0131448>] kthread+0x38/0x60
>   [<c0131410>] kthread+0x0/0x60
>   [<c01050c7>] kernel_thread_helper+0x7/0x10
>   =======================
> Code: 8b 75 00 39 ee 75 15 8b 75 10 8d 45 10 c7 45 34 01 00 00 00 39 c6 0f
> 84 a5 00 00 00 8b 54 24 0c 8b 82 98 00 00 00 39 46 10 72 37 <0f> 0b eb fe 
> 8b
> 44 24 0c 8b 5c 24 10 8b 90 8c 00 00 00 8b 44 24
> EIP: [<c01655c9>] cache_alloc_refill+0xf3/0x49d SS:ESP 0068:f7871e88
> Kernel panic - not syncing: Fatal exception in interrupt
> 
> 
> 
> ----------------------------------------------------------------------------
> Second Run
> ----------------------------------------------------------------------------
> 
> Fedora Core release 6 (Zod)
> Kernel 2.6.24.7-1.2944.fc6.emulab-1 on an i686
> 
> node0.eval.forest.emulab.net login: click: starting router thread pid 3889 
> (f576dc40)
> ------------[ cut here ]------------
> kernel BUG at mm/slab.c:602!
> invalid opcode: 0000 [#1] SMP
> Modules linked in: click proclikefs i2c_dev i2c_core tulip e100 3c59x 
> dm_mirror nvram e1000 button
> 
> Pid: 3889, comm: kclick Not tainted (2.6.24.7-1.2944.fc6.emulab-1 #1)
> EIP: 0060:[<c0165034>] EFLAGS: 00010046 CPU: 0
> EIP is at free_block+0x4d/0x104
> EAX: 40000000 EBX: 0000001c ECX: f5621800 EDX: c16ac420
> ESI: f5405dc0 EDI: f78022c0 EBP: f7800880 ESP: f73b7ee8
>   DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Process kclick (pid: 3889, ti=f73b6000 task=f5592330 task.ti=f73b6000)
> Stack: 00000007 00000000 0000001b 00000018 f78054f4 f78022c0 000000d8 
> f7800880
>         0000001b c0164e7d 00000000 f7805480 f7805480 00000286 f540b800 
> 0047b8a4
>         c0164f0b f56b4a80 f702d180 00000001 c0318570 f8e936e0 f8d31991 
> 00000000
> Call Trace:
>   [<c0164e7d>] cache_flusharray+0x71/0x98
>   [<c0164f0b>] kfree+0x67/0x76
>   [<c0318570>] __kfree_skb+0x8/0xd
>   [<f8d31991>] _ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
>   [<f8ca7286>] _ZN7Element4pushEiP6Packet+0x26/0x40 [click]
>   [<f8cfe628>] _ZN10FromDevice8run_taskEP4Task+0x78/0xf0 [click]
>   [<f8cbfe35>] _ZN12RouterThread6driverEv+0x235/0x470 [click]
>   [<c011e788>] sched_move_task+0xc1/0xc7
>   [<c01291cb>] switch_uid+0x5c/0x69
>   [<f8ca5a54>] click_lalloc+0x34/0x60 [click]
>   [<f8d2dc59>] _Z11click_schedPv+0xc9/0x1b0 [click]
>   [<f8d2db90>] _Z11click_schedPv+0x0/0x1b0 [click]
>   [<c01050c7>] kernel_thread_helper+0x7/0x10
>   =======================
> Code: 8b 08 8d 91 00 00 00 40 c1 ea 0c c1 e2 05 03 15 80 f6 51 c0 8b 02 25 
> 00 40 02 00 3d 00 40 02 00 75 03 8b 52 0c
> 8b 02 84 c0 78 04 <0f> 0b eb fe 8b 72 1c 8b 54 24 28 8b 46 04 8b bc 95 c8 
> 00 00 00
> EIP: [<c0165034>] free_block+0x4d/0x104 SS:ESP 0068:f73b7ee8
> ---[ end trace e44c05cd35086d7b ]---
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click