[Click] kernel crash with dev_queue_xmit()
Noonan, Patrick J.
pnoonan at swri.org
Tue Mar 6 18:08:02 EST 2012
Hi All,
I'm running patchless Click (pulled from git today) on Ubuntu 10.04
(kernel 2.6.32-38). Using the make-ip-conf.pl script I set up a 2-port
router config with the following topology.
HostA::eth0 <==> eth1:Router:eth0 <==> HostB::eth0
What I found is that if the ARP tables are empty in both end hosts and in
the router's ARPQuerier elements then the kernel will crash when trying to
send packets that have been queued in the ARPQuerier's ARPTable. Using
wireshark on the Click router I'm able to see the ARP query sent by Click;
however, it never makes it to the end host. This happens a few times and
ARPQuerier queues up packets. Eventually an ARP query makes it to the end
host, the end host replies, ARPQuerier begins emptying its queue and
finally the kernel crashes.
After quite a bit of searching on the mailing list I decided to revert the
following ToDevice patch, and now ARP queries always make it to the end
host and the kernel no longer crashes.
https://github.com/kohler/click/commit/d65bfe4054a23468c808e95df4ab6b4a5ef9
d9a3
Not sure why my setup is crashing with dev_queue_xmit(), or if no longer
using dev_queue_xmit() will cause other problems. Any help would be
greatly appreciated. Thanks.
[ 8023.115770] click: starting router thread pid 31116 (ed0fcd80)
[ 8483.153052] BUG: unable to handle kernel paging request at 0200a8c0
[ 8483.157020] IP: [<0200a8c0>] 0x200a8c0
[ 8483.157020] *pde = 00000000
[ 8483.157020] Oops: 0000 [#1] SMP
[ 8483.157020] last sysfs file:
/sys/devices/pci0000:00/0000:00:1f.2/host2/target2:0:0/2:0:0:0/block/sda/ue
vent
[ 8483.157020] Modules linked in: click proclikefs binfmt_misc
snd_hda_codec_realtek fbcon tileblit font bitblit softcursor vga16fb
vgastate snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss
snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi
snd_seq_midi_event snd_seq snd_timer snd_seq_device ppdev snd i915
drm_kms_helper parport_pc drm i2c_algo_bit video psmouse serio_raw
intel_agp agpgart soundcore output snd_page_alloc lp parport 8139too
floppy r8169 mii e1000e
[ 8483.157020]
[ 8483.157020] Pid: 31116, comm: kclick Not tainted (2.6.32-38-generic
#83-Ubuntu) MS-7529
[ 8483.157020] EIP: 0060:[<0200a8c0>] EFLAGS: 00010246 CPU: 0
[ 8483.157020] EIP is at 0x200a8c0
[ 8483.157020] EAX: ed0dec00 EBX: ed0dec00 ECX: 000000e9 EDX: 0200a8c0
[ 8483.157020] ESI: c04c7f95 EDI: 00000000 EBP: f1b21e60 ESP: f1b21e58
[ 8483.157020] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 8483.157020] Process kclick (pid: 31116, ti=f1b20000 task=e6d172c0
task.ti=f1b20000)
[ 8483.157020] Stack:
[ 8483.157020] c04be005 ed0dec00 f1b21e6c c04bdd50 ed0dec00 f1b21e80
c04bde50 ed0dec00
[ 8483.157020] <0> 00000000 f7001b40 f1b21ea4 c04c7f95 ed0de3c0 f6a68000
f8088380 f6a68000
[ 8483.157020] <0> f19d5b00 f7001b40 00000000 f1b21ed0 c04dcc25 ed0de474
f6a68000 00000000
[ 8483.157020] Call Trace:
[ 8483.157020] [<c04be005>] ? skb_release_head_state+0x45/0xe0
[ 8483.157020] [<c04bdd50>] ? __kfree_skb+0x10/0x90
[ 8483.157020] [<c04bde50>] ? kfree_skb+0x40/0x80
[ 8483.157020] [<c04c7f95>] ? dev_hard_start_xmit+0x215/0x240
[ 8483.157020] [<c04dcc25>] ? sch_direct_xmit+0x105/0x170
[ 8483.157020] [<c04cad6f>] ? dev_queue_xmit+0x37f/0x4b0
[ 8483.157020] [<fb17c2cf>] ?
_ZN8ToDevice12queue_packetEP6PacketP12netdev_queue+0xef/0x1b0 [click]
[ 8483.157020] [<fb19e948>] ? _ZN3Tee4pushEiP6Packet+0x68/0x70 [click]
[ 8483.157020] [<fb17c485>] ? _ZN8ToDevice8run_taskEP4Task+0xf5/0x220
[click]
[ 8483.157020] [<fb17770d>] ? _ZN10FromDevice8run_taskEP4Task+0x8d/0xe0
[click]
[ 8483.157020] [<fb130e7a>] ? _ZN12RouterThread6driverEv+0x31a/0x580
[click]
[ 8483.157020] [<fb1bbd2c>] ? _ZL11click_schedPv+0x10c/0x200 [click]
[ 8483.157020] [<fb1bbc20>] ? _ZL11click_schedPv+0x0/0x200 [click]
[ 8483.157020] [<c0104087>] ? kernel_thread_helper+0x7/0x10
[ 8483.157020] Code: Bad EIP value.
[ 8483.157020] EIP: [<0200a8c0>] 0x200a8c0 SS:ESP 0068:f1b21e58
[ 8483.157020] CR2: 000000000200a8c0
[ 8483.392554] ---[ end trace 7fcecc4ba07683ce ]---
[ 8483.397171] Kernel panic - not syncing: Fatal exception in interrupt
[ 8483.403525] Pid: 31116, comm: kclick Tainted: G D
2.6.32-38-generic #83-Ubuntu
[ 8483.411434] Call Trace:
[ 8483.413887] [<c058ea74>] ? printk+0x1d/0x21
[ 8483.418158] [<c058e9aa>] panic+0x48/0xf5
[ 8483.422170] [<c059211d>] oops_end+0xcd/0xd0
[ 8483.426445] [<c012b9cc>] no_context+0xbc/0xe0
[ 8483.430891] [<c012ba2c>] __bad_area_nosemaphore+0x3c/0x160
[ 8483.436464] [<c0591433>] ? _read_unlock_bh+0x13/0x20
[ 8483.441520] [<c012bb67>] bad_area_nosemaphore+0x17/0x20
[ 8483.446834] [<c05938a6>] do_page_fault+0x2f6/0x3a0
[ 8483.451716] [<c04c7f95>] ? dev_hard_start_xmit+0x215/0x240
[ 8483.457292] [<c05935b0>] ? do_page_fault+0x0/0x3a0
[ 8483.462170] [<c05915b3>] error_code+0x73/0x80
[ 8483.466614] [<c04c7f95>] ? dev_hard_start_xmit+0x215/0x240
[ 8483.472323] [<fb11007b>] ? _ZlsR11StringAccumRK9Timestamp+0xdb/0x380
[click]
[ 8483.479456] [<c04be005>] ? skb_release_head_state+0x45/0xe0
[ 8483.485113] [<c04bdd50>] __kfree_skb+0x10/0x90
[ 8483.489645] [<c04bde50>] kfree_skb+0x40/0x80
[ 8483.494007] [<c04c7f95>] dev_hard_start_xmit+0x215/0x240
[ 8483.499412] [<c04dcc25>] sch_direct_xmit+0x105/0x170
[ 8483.504466] [<c04cad6f>] dev_queue_xmit+0x37f/0x4b0
[ 8483.509590] [<fb17c2cf>]
_ZN8ToDevice12queue_packetEP6PacketP12netdev_queue+0xef/0x1b0 [click]
[ 8483.518443] [<fb19e948>] ? _ZN3Tee4pushEiP6Packet+0x68/0x70 [click]
[ 8483.524955] [<fb17c485>] _ZN8ToDevice8run_taskEP4Task+0xf5/0x220
[click]
[ 8483.531899] [<fb17770d>] ? _ZN10FromDevice8run_taskEP4Task+0x8d/0xe0
[click]
[ 8483.539171] [<fb130e7a>] _ZN12RouterThread6driverEv+0x31a/0x580 [click]
[ 8483.546023] [<fb1bbd2c>] _ZL11click_schedPv+0x10c/0x200 [click]
[ 8483.552183] [<fb1bbc20>] ? _ZL11click_schedPv+0x0/0x200 [click]
[ 8483.558190] [<c0104087>] kernel_thread_helper+0x7/0x10
[ 8483.563418] [drm:drm_fb_helper_panic] *ERROR* panic occurred, switching
back to text console
[ 8483.567408] BUG: scheduling while atomic: kclick/31116/0x10000200
[ 8483.567408] Modules linked in: click proclikefs binfmt_misc
snd_hda_codec_realtek fbcon tileblit font bitblit softcursor vga16fb
vgastate snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss
snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi
snd_seq_midi_event snd_seq snd_timer snd_seq_device ppdev snd i915
drm_kms_helper parport_pc drm i2c_algo_bit video psmouse serio_raw
intel_agp agpgart soundcore output snd_page_alloc lp parport 8139too
floppy r8169 mii e1000e
[ 8483.567408] Pid: 31116, comm: kclick Tainted: G D
2.6.32-38-generic #83-Ubuntu
[ 8483.567408] Call Trace:
[ 8483.567408] [<c013f6e2>] __schedule_bug+0x72/0x80
[ 8483.567408] [<c058f2a4>] schedule+0x644/0x870
[ 8483.567408] [<c0356211>] ? __const_udelay+0x31/0x40
[ 8483.567408] [<c058f612>] _cond_resched+0x32/0x50
[ 8483.567408] [<c01ffe30>] __kmalloc+0xc0/0x190
[ 8483.567408] [<c012a958>] ? default_spin_lock_flags+0x8/0x10
[ 8483.567408] [<f80e56f2>] ? T.813+0x12/0x20 [drm_kms_helper]
[ 8483.567408] [<f80e56f2>] T.813+0x12/0x20 [drm_kms_helper]
[ 8483.567408] [<f80e57e7>] drm_crtc_helper_set_config+0xe7/0x870
[drm_kms_helper]
[ 8483.567408] [<f80e34cd>] drm_fb_helper_force_kernel_mode+0x4d/0x90
[drm_kms_helper]
[ 8483.567408] [<f80e3574>] drm_fb_helper_panic+0x24/0x30 [drm_kms_helper]
[ 8483.567408] [<c0593993>] notifier_call_chain+0x43/0x60
[ 8483.567408] [<c0593a02>] atomic_notifier_call_chain+0x22/0x30
[ 8483.567408] [<c058e9ca>] panic+0x68/0xf5
[ 8483.567408] [<c059211d>] oops_end+0xcd/0xd0
[ 8483.567408] [<c012b9cc>] no_context+0xbc/0xe0
[ 8483.567408] [<c012ba2c>] __bad_area_nosemaphore+0x3c/0x160
[ 8483.567408] [<c0591433>] ? _read_unlock_bh+0x13/0x20
[ 8483.567408] [<c012bb67>] bad_area_nosemaphore+0x17/0x20
[ 8483.567408] [<c05938a6>] do_page_fault+0x2f6/0x3a0
[ 8483.567408] [<c04c7f95>] ? dev_hard_start_xmit+0x215/0x240
[ 8483.567408] [<c05935b0>] ? do_page_fault+0x0/0x3a0
[ 8483.567408] [<c05915b3>] error_code+0x73/0x80
[ 8483.567408] [<c04c7f95>] ? dev_hard_start_xmit+0x215/0x240
[ 8483.567408] [<fb11007b>] ? _ZlsR11StringAccumRK9Timestamp+0xdb/0x380
[click]
[ 8483.567408] [<c04be005>] ? skb_release_head_state+0x45/0xe0
[ 8483.567408] [<c04bdd50>] __kfree_skb+0x10/0x90
[ 8483.567408] [<c04bde50>] kfree_skb+0x40/0x80
[ 8483.567408] [<c04c7f95>] dev_hard_start_xmit+0x215/0x240
[ 8483.567408] [<c04dcc25>] sch_direct_xmit+0x105/0x170
[ 8483.567408] [<c04cad6f>] dev_queue_xmit+0x37f/0x4b0
[ 8483.567408] [<fb17c2cf>]
_ZN8ToDevice12queue_packetEP6PacketP12netdev_queue+0xef/0x1b0 [click]
[ 8483.567408] [<fb19e948>] ? _ZN3Tee4pushEiP6Packet+0x68/0x70 [click]
[ 8483.567408] [<fb17c485>] _ZN8ToDevice8run_taskEP4Task+0xf5/0x220
[click]
[ 8483.567408] [<fb17770d>] ? _ZN10FromDevice8run_taskEP4Task+0x8d/0xe0
[click]
[ 8483.567408] [<fb130e7a>] _ZN12RouterThread6driverEv+0x31a/0x580 [click]
[ 8483.567408] [<fb1bbd2c>] _ZL11click_schedPv+0x10c/0x200 [click]
[ 8483.567408] [<fb1bbc20>] ? _ZL11click_schedPv+0x0/0x200 [click]
[ 8483.567408] [<c0104087>] kernel_thread_helper+0x7/0x10
[ 8483.567408] NOHZ: local_softirq_pending 28a
~Patrick
More information about the click
mailing list