[Click] Help! 2.6.18 kernel panic with click 1.6.0
Raj Sidh
rajaram_s at yahoo.com
Fri Jun 19 23:05:12 EDT 2009
Hi,
I am running click kernel module over centos 5 which I patched to accept click 1.6.0. I am getting kernel panics time to time when the traffic is going on. Please see couple oops dumps below. I had customized the click to expect VLAN packets and process, though I am not sure whether that is causing the issue. A newbie to the kernel space and oops, I will appreciate any leads on what could be going wrong and advice. Thanks a lot! - Rajaram
Oops #1
=======
BUG: unable to handle kernel NULL pointer dereference at virtual address 00000001
printing eip:
c04d0cba
*pde = 00000000
Oops: 0000 [#1]
last sysfs file: /class/net/fake4/address
Modules linked in: click(U) proclikefs(U) deflate(U) zlib_deflate(U) af_key(U) pk_drv(PU) ipv6(U) xfrm_nalgo(U) crypto_api(U) autofs4(U) dm_mirror(U) dm_log(U) dm_multipath(U) scsi_dh(U) dm_mod(U) video(U) sbs(U) backlight(U) i2c_ec(U) i2c_core(U) button(U) battery(U) asus_acpi(U) ac(U) parport_pc(U) lp(U) parport(U) joydev(U) ata_piix(U) i5000_edac(U) edac_mc(U) pcspkr(U) libata(U) e1000e(U) bnx2(U) ide_cd(U) serio_raw(U) cdrom(U) cciss(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
CPU: 0
EIP: 0060:[<c04d0cba>] Tainted: P VLI
EFLAGS: 00010082 (2.6.18-click.patched.1 #1)
EIP is at list_del+0x6/0x5c
eax: 00000001 ebx: f61a6d40 ecx: f7fff0c0 edx: 00000000
esi: f7ffd440 edi: f675a000 ebp: f7fff7c0 esp: c06eaf6c
ds: 007b es: 007b ss: 0068
Process kclick (pid: 2277, ti=c06ea000 task=f7a17000 task.ti=f7924000)
Stack: f61a6d40 c045a5f6 00000000 0000000c 0000000b f7ffc180 0000000c 00000000
f7fff7c0 f7ffc140 c045a43e 00000000 f7ffc140 f672d000 00000286 0000000a
c045a4ab f665c550 f7e63b5c 00000007 c04183a0 c0719060 c042690d c07190dc
Call Trace:
[<c045a5f6>] free_block+0x67/0xda
[<c045a43e>] cache_flusharray+0x53/0x6e
[<c045a4ab>] kfree+0x52/0x61
[<c04183a0>] free_task+0x16/0x23
[<c042690d>] __rcu_process_callbacks+0xcf/0x128
[<c0426975>] rcu_process_callbacks+0xf/0x1e
[<c041de13>] tasklet_action+0x58/0x97
[<c041e13a>] __do_softirq+0x5d/0xc6
[<c04054bd>] do_softirq+0x3b/0x85
[<c041dff2>] local_bh_enable+0x66/0x71
[<f946f0ff>] _ZN8ToDevice8run_taskEP4Task+0x13f/0x400 [click]
[<f942f0e7>] _ZN12RouterThread6driverEv+0x177/0x290 [click]
[<f94d1802>] _Z11click_schedPv+0x82/0x130 [click]
[<f94d1780>] _Z11click_schedPv+0x0/0x130 [click]
[<c0404093>] kernel_thread_helper+0x7/0x10
=======================
Code: 8d 4b 04 8b 51 04 8d 46 04 e8 73 00 00 00 8d 4b 0c 8b 51 04 8d 46 0c 5b 5e 5f e9 62 00 00 00 89 c3 eb eb 90 90 53 89 c3 8b 40 04 <8b> 00 39 d8 74 17 50 53 68 a9 62 61 c0 e8 93 9c f4 ff 0f 0b 41
EIP: [<c04d0cba>] list_del+0x6/0x5c SS:ESP 0068:c06eaf6c
<0>Kernel panic - not syncing: Fatal exception in interrupt
Oops #2
=======
------------[ cut here ]------------
kernel BUG at mm/slab.c:1605!
invalid opcode: 0000 [#1]
last sysfs file: /class/net/fake2/address
Modules linked in: click(U) proclikefs(U) deflate(U) zlib_deflate(U) af_key(U) pk_drv(PU) ipv6(U) xfrm_nalgo(U) crypto_api(U) autofs4(U) dm_mirror(U) dm_log(U) dm_multipath(U) scsi_dh(U) dm_mod(U) video(U) sbs(U) backlight(U) i2c_ec(U) i2c_core(U) button(U) battery(U) asus_acpi(U) ac(U) parport_pc(U) lp(U) parport(U) joydev(U) pcspkr(U) ata_piix(U) i5000_edac(U) edac_mc(U) libata(U) e1000e(U) bnx2(U) ide_cd(U) serio_raw(U) cdrom(U) cciss(U) sd_mod(U) scsi_mod(U) ext3(U) jbd(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U)
CPU: 0
EIP: 0060:[<c045a2e3>] Tainted: P VLI
EFLAGS: 00010046 (2.6.18-click.patched.1 #1)
EIP is at kmem_freepages+0x49/0x97
eax: c0080000 ebx: f7fff7c0 ecx: c5800000 edx: 00000000
esi: 00000001 edi: 00000000 ebp: 00000000 esp: c06eaf50
ds: 007b es: 007b ss: 0068
Process kclick (pid: 2289, ti=c06ea000 task=f7fb0000 task.ti=f619e000)
Stack: f7fff7c0 f5e25680 00000000 c045a576 f5e25680 f7ffd440 f5e37fff f7fff7c0
c045a630 00000aa0 0000000c 00000007 f7ffc170 0000000c 00000000 f7fff7c0
f7ffc140 c045a43e 00000000 f7ffc140 f5caf000 00000286 00000001 c045a4ab
Call Trace:
[<c045a576>] slab_destroy+0x53/0x6c
[<c045a630>] free_block+0xa1/0xda
[<c045a43e>] cache_flusharray+0x53/0x6e
[<c045a4ab>] kfree+0x52/0x61
[<c04183a0>] free_task+0x16/0x23
[<c042690d>] __rcu_process_callbacks+0xcf/0x128
[<c0426975>] rcu_process_callbacks+0xf/0x1e
[<c041de13>] tasklet_action+0x58/0x97
[<c041e13a>] __do_softirq+0x5d/0xc6
[<c04054bd>] do_softirq+0x3b/0x85
[<c041dff2>] local_bh_enable+0x66/0x71
[<f94710ff>] _ZN8ToDevice8run_taskEP4Task+0x13f/0x400 [click]
[<f94310e7>] _ZN12RouterThread6driverEv+0x177/0x290 [click]
[<f94d3802>] _Z11click_schedPv+0x82/0x130 [click]
[<f94d3780>] _Z11click_schedPv+0x0/0x130 [click]
[<c0404093>] kernel_thread_helper+0x7/0x10
=======================
Code: c1 e1 05 03 0d b0 53 72 c0 8b 01 c1 e8 1e 8b 04 85 84 f6 6a c0 01 90 0c 01 00 00 01 15 2c fd 71 c0 89 f2 eb 15 8b 01 84 c0 78 08 <0f> 0b 45 06 d8 cb 60 c0 0f ba 31 07 83 c1 20 4a 83 fa ff 75 e5
EIP: [<c045a2e3>] kmem_freepages+0x49/0x97 SS:ESP 0068:c06eaf50
<0>Kernel panic - not syncing: Fatal exception in interrupt
Note:
====
The tainted "P" is due to Version info missing for a module. The crash happens with or without that module.
In case helpful: snippet of the change in tohost.cc to accomodate vlan as follows in ToHost::push()
=========================================================================
if ((skb->nh.raw - skb->data) > 14) {
skb_pull(skb, skb->nh.raw - skb->data);
}
else
skb_pull(skb, 14);
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 0)
const ethhdr *eth = eth_hdr(skb);
#else
const ethhdr *eth = skb->mac.ethernet;
#endif
unsigned short proto = ntohs(eth->h_proto);
if (proto == ETH_P_8021Q) {
const vlan_ethhdr* veth = vlan_eth_hdr(skb);
proto = ntohs(veth->h_vlan_encapsulated_proto);
}
More information about the click
mailing list