[Click] click router on centos 5.5 kernel

sri bskmohan at gmail.com
Sun Oct 9 23:50:25 EDT 2011


Hello Experts,

I was using click-1.6.0 on centos 5.3 kernel and recently* ported that to
centos5.5 kerne*l.
(Though the version of click I am using is very old, this has customized
elements to suit my purposes. Hence not click version is not upgraded.)
After porting, am getting the (*low stack detected by irq handler)  and the
*following errors on the console frequently and causing the system
unresponsive totally.
Even after reboot, this issue is re-occurring some time later.
Am looking for pointers, just pointers, to start with so that I can
investigate. I tried to do web search on this but not useful.
Thanks in advance.

######################## *START OF LOGS* ############################
Oct  1 14:16:56 server03 kernel: low stack detected by irq handler
Oct  1 14:16:56 server03 kernel:  [<c0405e9b>] do_IRQ+0x86/0xc3
Oct  1 14:16:56 server03 kernel:  [<c04046e6>] common_interrupt+0x1a/0x20
Oct  1 14:16:56 server03 kernel:  [<c05a5c06>] neigh_lookup+0x23/0x7f
Oct  1 14:16:56 server03 kernel:  [<c05d6800>] arp_bind_neighbour+0x50/0x71
Oct  1 14:16:56 server03 kernel:  [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct  1 14:16:56 server03 kernel:  [<c05b8352>]
__ip_route_output_key+0x700/0x79e
Oct  1 14:16:56 server03 kernel:  [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct  1 14:16:57 server03 kernel:  [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct  1 14:16:57 server03 kernel:  [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct  1 14:16:57 server03 kernel:  [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct  1 14:16:57 server03 kernel:  [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct  1 14:16:57 server03 kernel:  [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct  1 14:16:57 server03 kernel:  [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct  1 14:16:57 server03 kernel:  [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct  1 14:16:57 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:16:57 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:16:57 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:16:57 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:16:57 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:16:57 server03 kernel:  [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct  1 14:16:57 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:16:57 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:16:57 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:16:57 server03 kernel:  [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct  1 14:16:57 server03 kernel:  [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct  1 14:16:57 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:16:57 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:16:57 server03 kernel:  [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct  1 14:16:57 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:16:57 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:16:57 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:16:58 server03 kernel:  [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct  1 14:16:58 server03 kernel:  [<c0417f22>] enqueue_task+0x29/0x39
Oct  1 14:16:58 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:16:58 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:16:58 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:16:58 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:58 server03 kernel:  [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct  1 14:16:58 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:16:58 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:58 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:16:58 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:16:58 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:16:58 server03 kernel:  [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct  1 14:16:58 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:16:58 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:16:58 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:16:58 server03 kernel:  [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct  1 14:16:58 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:16:58 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:16:58 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:16:58 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:58 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:16:59 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1b1853>]
_ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:16:59 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:16:59 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct  1 14:16:59 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:16:59 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:16:59 server03 kernel:  [<c05a5c59>] neigh_lookup+0x76/0x7f
Oct  1 14:16:59 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct  1 14:16:59 server03 kernel:  [<c05bf82d>] ip_output+0x1f0/0x227
Oct  1 14:16:59 server03 kernel:  [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct  1 14:16:59 server03 kernel:  [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1dcd23>]
_ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:00 server03 kernel:  [<c05d065b>] tcp_v4_do_rcv+0x22/0x269
Oct  1 14:17:00 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:00 server03 kernel:  [<fb1bfb12>]
_ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click]
Oct  1 14:17:00 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:00 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:00 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:00 server03 kernel:  [<c05d7777>] arp_rcv+0xd0/0xe1
Oct  1 14:17:00 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:00 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:00 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:00 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:00 server03 kernel:  [<c0411f41>]
smp_apic_timer_interrupt+0x32/0x45
Oct  1 14:17:00 server03 kernel:  [<c040470b>]
apic_timer_interrupt+0x1f/0x24
Oct  1 14:17:00 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:00 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:01 server03 kernel:  [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct  1 14:17:01 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:01 server03 kernel:  [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct  1 14:17:01 server03 kernel:  [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct  1 14:17:01 server03 kernel:  [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct  1 14:17:01 server03 kernel:  [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct  1 14:17:01 server03 kernel:  [<c0404987>] kernel_thread_helper+0x7/0x10
Oct  1 14:17:01 server03 kernel:  =======================
Oct  1 14:17:01 server03 kernel: low stack detected by irq handler
Oct  1 14:17:01 server03 kernel:  [<c0405e9b>] do_IRQ+0x86/0xc3
Oct  1 14:17:01 server03 kernel:  [<c04046e6>] common_interrupt+0x1a/0x20
Oct  1 14:17:01 server03 kernel:  [<c05a5c06>] neigh_lookup+0x23/0x7f
Oct  1 14:17:01 server03 kernel:  [<c05d6800>] arp_bind_neighbour+0x50/0x71
Oct  1 14:17:01 server03 kernel:  [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct  1 14:17:01 server03 kernel:  [<c05b8352>]
__ip_route_output_key+0x700/0x79e
Oct  1 14:17:01 server03 kernel:  [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct  1 14:17:01 server03 kernel:  [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct  1 14:17:01 server03 kernel:  [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct  1 14:17:01 server03 kernel:  [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct  1 14:17:01 server03 kernel:  [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct  1 14:17:01 server03 kernel:  [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct  1 14:17:01 server03 kernel:  [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct  1 14:17:01 server03 kernel:  [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct  1 14:17:01 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:02 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:02 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:02 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:02 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:02 server03 kernel:  [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct  1 14:17:02 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:02 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:02 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:02 server03 kernel:  [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct  1 14:17:02 server03 kernel:  [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct  1 14:17:02 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:02 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:02 server03 kernel:  [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct  1 14:17:02 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:02 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:02 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:02 server03 kernel:  [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct  1 14:17:02 server03 kernel:  [<c0417f22>] enqueue_task+0x29/0x39
Oct  1 14:17:02 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:02 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:02 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:02 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:02 server03 kernel:  [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:03 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:03 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:03 server03 kernel:  [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct  1 14:17:03 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:03 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:03 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct  1 14:17:03 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:03 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:03 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:03 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:03 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1b1853>]
_ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct  1 14:17:03 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:03 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:04 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct  1 14:17:04 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:04 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:04 server03 kernel:  [<c05a5c59>] neigh_lookup+0x76/0x7f
Oct  1 14:17:04 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct  1 14:17:04 server03 kernel:  [<c05bf82d>] ip_output+0x1f0/0x227
Oct  1 14:17:04 server03 kernel:  [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1dcd23>]
_ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click]
Oct  1 14:17:04 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:04 server03 kernel:  [<c05d065b>] tcp_v4_do_rcv+0x22/0x269
Oct  1 14:17:04 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:04 server03 kernel:  [<fb1bfb12>]
_ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click]
Oct  1 14:17:05 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:05 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:05 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:05 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:05 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:05 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:05 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:05 server03 kernel:  [<c05d7777>] arp_rcv+0xd0/0xe1
Oct  1 14:17:05 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:05 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:05 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:05 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:05 server03 kernel:  [<c0411f41>]
smp_apic_timer_interrupt+0x32/0x45
Oct  1 14:17:05 server03 kernel:  [<c040470b>]
apic_timer_interrupt+0x1f/0x24
Oct  1 14:17:05 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:05 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:05 server03 kernel:  [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct  1 14:17:05 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:05 server03 kernel:  [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct  1 14:17:05 server03 kernel:  [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct  1 14:17:05 server03 kernel:  [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct  1 14:17:05 server03 kernel:  [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct  1 14:17:05 server03 kernel:  [<c0404987>] kernel_thread_helper+0x7/0x10
Oct  1 14:17:05 server03 kernel:  =======================
Oct  1 14:17:06 server03 kernel: low stack detected by irq handler
Oct  1 14:17:06 server03 kernel:  [<c0405e9b>] do_IRQ+0x86/0xc3
Oct  1 14:17:06 server03 kernel:  [<c04046e6>] common_interrupt+0x1a/0x20
Oct  1 14:17:06 server03 kernel:  [<c05a5c06>] neigh_lookup+0x23/0x7f
Oct  1 14:17:06 server03 kernel:  [<c05d6800>] arp_bind_neighbour+0x50/0x71
Oct  1 14:17:06 server03 kernel:  [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct  1 14:17:06 server03 kernel:  [<c05b8352>]
__ip_route_output_key+0x700/0x79e
Oct  1 14:17:06 server03 kernel:  [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct  1 14:17:06 server03 kernel:  [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct  1 14:17:06 server03 kernel:  [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct  1 14:17:06 server03 kernel:  [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct  1 14:17:06 server03 kernel:  [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct  1 14:17:06 server03 kernel:  [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct  1 14:17:06 server03 kernel:  [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct  1 14:17:06 server03 kernel:  [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct  1 14:17:06 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:06 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:06 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:06 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:06 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:06 server03 kernel:  [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct  1 14:17:07 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:07 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:07 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:07 server03 kernel:  [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct  1 14:17:07 server03 kernel:  [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct  1 14:17:07 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:07 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:07 server03 kernel:  [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct  1 14:17:07 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:07 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:07 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:07 server03 kernel:  [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct  1 14:17:07 server03 kernel:  [<c0417f22>] enqueue_task+0x29/0x39
Oct  1 14:17:07 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:07 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:07 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:07 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:07 server03 kernel:  [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct  1 14:17:07 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:08 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:08 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:08 server03 kernel:  [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct  1 14:17:08 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:08 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:08 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct  1 14:17:08 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:08 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:08 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:08 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:08 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1b1853>]
_ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct  1 14:17:08 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:09 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:09 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct  1 14:17:09 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:09 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:09 server03 kernel:  [<c05a5c59>] neigh_lookup+0x76/0x7f
Oct  1 14:17:09 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct  1 14:17:09 server03 kernel:  [<c05bf82d>] ip_output+0x1f0/0x227
Oct  1 14:17:09 server03 kernel:  [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1dcd23>]
_ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:09 server03 kernel:  [<c05d065b>] tcp_v4_do_rcv+0x22/0x269
Oct  1 14:17:09 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:09 server03 kernel:  [<fb1bfb12>]
_ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click]
Oct  1 14:17:09 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:10 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:10 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:10 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:10 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:10 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:10 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:10 server03 kernel:  [<c05d7777>] arp_rcv+0xd0/0xe1
Oct  1 14:17:10 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:10 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:10 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:10 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:10 server03 kernel:  [<c0411f41>]
smp_apic_timer_interrupt+0x32/0x45
Oct  1 14:17:10 server03 kernel:  [<c040470b>]
apic_timer_interrupt+0x1f/0x24
Oct  1 14:17:10 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:10 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:10 server03 kernel:  [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct  1 14:17:10 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:10 server03 kernel:  [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct  1 14:17:10 server03 kernel:  [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct  1 14:17:10 server03 kernel:  [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct  1 14:17:10 server03 kernel:  [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct  1 14:17:10 server03 kernel:  [<c0404987>] kernel_thread_helper+0x7/0x10
Oct  1 14:17:10 server03 kernel:  =======================
Oct  1 14:17:11 server03 kernel: low stack detected by irq handler
Oct  1 14:17:11 server03 kernel:  [<c0405e9b>] do_IRQ+0x86/0xc3
Oct  1 14:17:11 server03 kernel:  [<c04046e6>] common_interrupt+0x1a/0x20
Oct  1 14:17:11 server03 kernel:  [<c05a5c06>] neigh_lookup+0x23/0x7f
Oct  1 14:17:11 server03 kernel:  [<c05d6800>] arp_bind_neighbour+0x50/0x71
Oct  1 14:17:11 server03 kernel:  [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct  1 14:17:11 server03 kernel:  [<c05b8352>]
__ip_route_output_key+0x700/0x79e
Oct  1 14:17:11 server03 kernel:  [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct  1 14:17:11 server03 kernel:  [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct  1 14:17:11 server03 kernel:  [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct  1 14:17:11 server03 kernel:  [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct  1 14:17:11 server03 kernel:  [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct  1 14:17:11 server03 kernel:  [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct  1 14:17:11 server03 kernel:  [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct  1 14:17:11 server03 kernel:  [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct  1 14:17:11 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:11 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:11 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:11 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:11 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:11 server03 kernel:  [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct  1 14:17:11 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:11 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:12 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:12 server03 kernel:  [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct  1 14:17:12 server03 kernel:  [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct  1 14:17:12 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:12 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:12 server03 kernel:  [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct  1 14:17:12 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:12 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:12 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:12 server03 kernel:  [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct  1 14:17:12 server03 kernel:  [<c0417f22>] enqueue_task+0x29/0x39
Oct  1 14:17:12 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:12 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:12 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:12 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:12 server03 kernel:  [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct  1 14:17:12 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:12 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:12 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:12 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:12 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:12 server03 kernel:  [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct  1 14:17:12 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:13 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:13 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct  1 14:17:13 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:13 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:13 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1b1853>]
_ZN11ARPQuerier214send_query_forERK9IPAddressitit+0x133/0x3f0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:13 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:14 server03 kernel:  [<c05a5c59>] neigh_lookup+0x76/0x7f
Oct  1 14:17:14 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct  1 14:17:14 server03 kernel:  [<c05bf82d>] ip_output+0x1f0/0x227
Oct  1 14:17:14 server03 kernel:  [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1dcd23>]
_ZN14HashMACLookup514push_extern_ipEP6Packet+0x1b3/0x4d0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:14 server03 kernel:  [<c05d065b>] tcp_v4_do_rcv+0x22/0x269
Oct  1 14:17:14 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:14 server03 kernel:  [<fb1bfb12>]
_ZN11ARPQuerier415handle_responseEP6Packet+0x32/0x8f0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:14 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:14 server03 kernel:  [<c05d7777>] arp_rcv+0xd0/0xe1
Oct  1 14:17:15 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:15 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:15 server03 kernel:  [<c043f334>] __do_IRQ+0x0/0x9f
Oct  1 14:17:15 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:15 server03 kernel:  [<c0411f41>]
smp_apic_timer_interrupt+0x32/0x45
Oct  1 14:17:15 server03 kernel:  [<c040470b>]
apic_timer_interrupt+0x1f/0x24
Oct  1 14:17:15 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:15 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:15 server03 kernel:  [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct  1 14:17:15 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:15 server03 kernel:  [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct  1 14:17:15 server03 kernel:  [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct  1 14:17:15 server03 kernel:  [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct  1 14:17:15 server03 kernel:  [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct  1 14:17:15 server03 kernel:  [<c0404987>] kernel_thread_helper+0x7/0x10
Oct  1 14:17:15 server03 kernel:  =======================
Oct  1 14:17:15 server03 kernel: device eth1 left promiscuous mode
Oct  1 14:17:15 server03 kernel: type=1700 audit(1317500115.581:9): dev=eth1
prom=0 old_prom=256 auid=4294967295 ses=4294967295
Oct  1 14:17:15 server03 kernel: device eth0 left promiscuous mode
Oct  1 14:17:15 server03 kernel: type=1700 audit(1317500115.680:10):
dev=eth0 prom=0 old_prom=256 auid=4294967295 ses=4294967295
Oct  1 14:17:15 server03 kernel: device eth1 entered promiscuous mode
Oct  1 14:17:15 server03 kernel: type=1700 audit(1317500122.770:11):
dev=eth1 prom=256 old_prom=0 auid=4294967295 ses=4294967295
Oct  1 14:17:15 server03 kernel: device eth0 entered promiscuous mode
Oct  1 14:17:15 server03 kernel: type=1700 audit(1317500122.872:12):
dev=eth0 prom=256 old_prom=0 auid=4294967295 ses=4294967295
Oct  1 14:17:16 server03 kernel: low stack detected by irq handler
Oct  1 14:17:16 server03 kernel:  [<c0405e9b>] do_IRQ+0x86/0xc3
Oct  1 14:17:16 server03 kernel:  [<c04046e6>] common_interrupt+0x1a/0x20
Oct  1 14:17:16 server03 kernel:  [<c05d67f5>] arp_bind_neighbour+0x45/0x71
Oct  1 14:17:16 server03 kernel:  [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct  1 14:17:16 server03 kernel:  [<c05b8352>]
__ip_route_output_key+0x700/0x79e
Oct  1 14:17:16 server03 kernel:  [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct  1 14:17:16 server03 kernel:  [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct  1 14:17:16 server03 kernel:  [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct  1 14:17:16 server03 kernel:  [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct  1 14:17:16 server03 kernel:  [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct  1 14:17:16 server03 kernel:  [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct  1 14:17:16 server03 kernel:  [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct  1 14:17:16 server03 kernel:  [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct  1 14:17:16 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:16 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:16 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:16 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:16 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:16 server03 kernel:  [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct  1 14:17:16 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:16 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:16 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:17 server03 kernel:  [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct  1 14:17:17 server03 kernel:  [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct  1 14:17:17 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:17 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:17 server03 kernel:  [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct  1 14:17:17 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:17 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:17 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:17 server03 kernel:  [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct  1 14:17:17 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:17 server03 kernel:  [<c0417f22>] enqueue_task+0x29/0x39
Oct  1 14:17:17 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:17 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:17 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:17 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:17 server03 kernel:  [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct  1 14:17:17 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:17 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:17 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:17 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:17 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:17 server03 kernel:  [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct  1 14:17:18 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:18 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:18 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:18 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1c86a3>]
_ZN10DnsHandler10get_dns_rrEPKcS1_RS1_R14perfigo_dns_rrR6String+0x43/0xb0
[click]
Oct  1 14:17:18 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:18 server03 kernel:  [<fb151a6f>]
_ZN6Packet19expensive_uniqueifyEiib+0x18f/0x1c0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct  1 14:17:18 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:18 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:18 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct  1 14:17:18 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:18 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:19 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:19 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1a11d1>]
_ZN14LinearIPLookup4pushEiP6Packet+0x71/0x100 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:19 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:19 server03 kernel:  [<fb2038ac>]
_ZN14DropBroadcastsD1Ev+0xc/0x10 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1c3b4a>]
_ZN13ARPResponder34pushEiP6Packet+0x4a/0x100 [click]
Oct  1 14:17:19 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:19 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:19 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:19 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:19 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:20 server03 kernel:  [<c05d7777>] arp_rcv+0xd0/0xe1
Oct  1 14:17:20 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:20 server03 kernel:  [<f88ab061>]
tg3_start_xmit_dma_bug+0x461/0x77f [tg3]
Oct  1 14:17:20 server03 kernel:  [<f88ab2ed>]
tg3_start_xmit_dma_bug+0x6ed/0x77f [tg3]
Oct  1 14:17:20 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:20 server03 kernel:  [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct  1 14:17:20 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:20 server03 kernel:  [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct  1 14:17:20 server03 kernel:  [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct  1 14:17:20 server03 kernel:  [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct  1 14:17:20 server03 kernel:  [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct  1 14:17:20 server03 kernel:  [<c0404987>] kernel_thread_helper+0x7/0x10
Oct  1 14:17:20 server03 kernel:  =======================
Oct  1 14:17:20 server03 kernel: low stack detected by irq handler
Oct  1 14:17:20 server03 kernel:  [<c0405e9b>] do_IRQ+0x86/0xc3
Oct  1 14:17:20 server03 kernel:  [<c04046e6>] common_interrupt+0x1a/0x20
Oct  1 14:17:20 server03 kernel:  [<c05d67f5>] arp_bind_neighbour+0x45/0x71
Oct  1 14:17:20 server03 kernel:  [<c05b7b44>] rt_intern_hash+0x372/0x480
Oct  1 14:17:20 server03 kernel:  [<c05b8352>]
__ip_route_output_key+0x700/0x79e
Oct  1 14:17:20 server03 kernel:  [<c05b83d4>]
__ip_route_output_key+0x782/0x79e
Oct  1 14:17:20 server03 kernel:  [<c05b8403>]
ip_route_output_flow+0x13/0x1d4
Oct  1 14:17:20 server03 kernel:  [<c05b85cd>] ip_route_output_key+0x9/0xb
Oct  1 14:17:20 server03 kernel:  [<c05bdd5d>] ip_send_reply+0xf8/0x209
Oct  1 14:17:20 server03 kernel:  [<c045f1a6>]
cache_alloc_refill+0x122/0x3e4
Oct  1 14:17:21 server03 kernel:  [<c05b7c2f>] rt_intern_hash+0x45d/0x480
Oct  1 14:17:21 server03 kernel:  [<c05cfc56>] tcp_v4_send_reset+0x104/0x11c
Oct  1 14:17:21 server03 kernel:  [<c05d0879>] tcp_v4_do_rcv+0x240/0x269
Oct  1 14:17:21 server03 kernel:  [<c05d28cb>] tcp_v4_rcv+0x7f7/0x848
Oct  1 14:17:21 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:21 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:21 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:21 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:21 server03 kernel:  [<fb1ce85d>]
_ZN12EnsureEther24pushEiP6Packet+0x1d/0x20 [click]
Oct  1 14:17:21 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:21 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:21 server03 kernel:  [<c05baff2>] ip_rcv+0x3cb/0x3f5
Oct  1 14:17:21 server03 kernel:  [<fb19e42d>]
_ZN4IPRw7Mapping9make_pairEiRK8IPFlowIDS3_iiPS0_S4_+0xad/0xd0 [click]
Oct  1 14:17:21 server03 kernel:  [<fb19fcfb>]
_ZN4IPRw7Pattern14create_mappingEiRK8IPFlowIDiiPNS_7MappingES5_RK7HashMapIS1_S5_E+0x1cb/0x1f0
[click]
Oct  1 14:17:21 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:21 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:21 server03 kernel:  [<fb1ea535>]
_ZN11MacThrottle10l2PushProcEiP6Packet+0x55/0x3b0 [click]
Oct  1 14:17:21 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:21 server03 kernel:  [<fb218ce3>]
_ZN10IPRewriter13apply_patternEPN4IPRw7PatternEiRK8IPFlowIDii+0x153/0x160
[click]
Oct  1 14:17:21 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:21 server03 kernel:  [<fb1eab96>]
_ZN11MacThrottle4pushEiP6Packet+0x306/0x1890 [click]
Oct  1 14:17:21 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:22 server03 kernel:  [<c0417f22>] enqueue_task+0x29/0x39
Oct  1 14:17:22 server03 kernel:  [<c041805a>] __activate_task+0x4a/0x59
Oct  1 14:17:22 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:22 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:22 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1ee075>]
_ZN17NtlmLoginDetector4pushEiP6Packet+0x65/0x15d0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:22 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb21f881>]
_ZN15RecycledSkbPool7recycleEP7sk_buff+0x31/0xf0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1fbfdb>]
_ZN14PerfigoDiscard4pushEiP6Packet+0x4b/0x1c0 [click]
Oct  1 14:17:22 server03 kernel:  [<c05bb173>] ip_local_deliver+0x157/0x1d0
Oct  1 14:17:22 server03 kernel:  [<fb1d8e86>]
_ZN11HashIPTable4pushEiP6Packet+0xf6/0x2d0 [click]
Oct  1 14:17:22 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:22 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:22 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1e4517>]
_ZN12IPPortLookup4pushEiP6Packet+0x47/0x9e0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:22 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1cd5fc>]
_ZN10DnsHandler17is_packet_allowedEPK6Packeti+0x8c/0x5e0 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:22 server03 kernel:  [<fb1c86a3>]
_ZN10DnsHandler10get_dns_rrEPKcS1_RS1_R14perfigo_dns_rrR6String+0x43/0xb0
[click]
Oct  1 14:17:23 server03 kernel:  [<fb1cdbb8>]
_ZN10DnsHandler4pushEiP6Packet+0x68/0xb00 [click]
Oct  1 14:17:23 server03 kernel:  [<fb151a6f>]
_ZN6Packet19expensive_uniqueifyEiib+0x18f/0x1c0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1b1dba>]
_ZN11ARPQuerier29handle_ipEP6Packet+0x2aa/0x5f0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1b31bd>]
_ZN11ARPQuerier24pushEiP6Packet+0xcd/0x110 [click]
Oct  1 14:17:23 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:23 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1c69d8>]
_ZN8BWShaper4pushEiP6Packet+0x58/0xe90 [click]
Oct  1 14:17:23 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1e234f>]
_ZN13IPFilterGroup4pushEiP6Packet+0x19f/0xa80 [click]
Oct  1 14:17:23 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:23 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:23 server03 kernel:  [<fb1b121e>]
_ZN6ToHost4pushEiP6Packet+0xee/0x210 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1d3492>]
_ZN13HashIPLookup34pushEiP6Packet+0x112/0x1c0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1e66cf>]
_ZN15LinearIPFilter24pushEiP6Packet+0x4f/0x3f0 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1a11d1>]
_ZN14LinearIPLookup4pushEiP6Packet+0x71/0x100 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1b80e8>]
_ZN11ARPQuerier420handle_arp_retaggingEP6Packett+0x58/0x940 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1dd289>]
_ZN14HashMACLookup54pushEiP6Packet+0x249/0x890 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1d45b7>]
_ZN13HashIPLookup54pushEiP6Packet+0xc7/0x110 [click]
Oct  1 14:17:23 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:24 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:24 server03 kernel:  [<fb2038ac>]
_ZN14DropBroadcastsD1Ev+0xc/0x10 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1c3b4a>]
_ZN13ARPResponder34pushEiP6Packet+0x4a/0x100 [click]
Oct  1 14:17:24 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:24 server03 kernel:  [<fb207674>]
_ZN11PaintSwitch4pushEiP6Packet+0x44/0xe0 [click]
Oct  1 14:17:24 server03 kernel:  [<fb192904>]
_ZN8IPFilter4pushEiP6Packet+0x64/0xf0 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1c0895>]
_ZN11ARPQuerier44pushEiP6Packet+0x4c5/0x2370 [click]
Oct  1 14:17:24 server03 kernel:  [<fb218d9e>]
_ZN10IPRewriter4pushEiP6Packet+0xae/0x260 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1d762e>]
_ZN12HashIPTable24pushEiP6Packet+0x18e/0x580 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1fb79e>]
_ZN6Passer4pushEiP6Packet+0x12e/0x860 [click]
Oct  1 14:17:24 server03 kernel:  [<c05d7777>] arp_rcv+0xd0/0xe1
Oct  1 14:17:24 server03 kernel:  [<c05a141d>]
__netif_receive_skb+0x3b3/0x3d5
Oct  1 14:17:24 server03 kernel:  [<f88ab061>]
tg3_start_xmit_dma_bug+0x461/0x77f [tg3]
Oct  1 14:17:24 server03 kernel:  [<f88ab2ed>]
tg3_start_xmit_dma_bug+0x6ed/0x77f [tg3]
Oct  1 14:17:24 server03 kernel:  [<c0405ecc>] do_IRQ+0xb7/0xc3
Oct  1 14:17:24 server03 kernel:  [<fb1fe516>]
_ZN10Classifier4pushEiP6Packet+0x76/0xe0 [click]
Oct  1 14:17:24 server03 kernel:  [<fb155806>]
_ZN7Element4pushEiP6Packet+0x26/0x40 [click]
Oct  1 14:17:24 server03 kernel:  [<fb1ab2a5>]
_ZN10FromDevice8run_taskEP4Task+0xa5/0x120 [click]
Oct  1 14:17:24 server03 kernel:  [<fb16c127>]
_ZN12RouterThread6driverEv+0x177/0x290 [click]
Oct  1 14:17:24 server03 kernel:  [<fb21c822>] _Z11click_schedPv+0x82/0x130
[click]
Oct  1 14:17:24 server03 kernel:  [<fb21c7a0>] _Z11click_schedPv+0x0/0x130
[click]
Oct  1 14:17:25 server03 kernel:  [<c0404987>] kernel_thread_helper+0x7/0x10
Oct  1 14:17:25 server03 kernel:  =======================

#######################* END OF LOGS #*
#####################################################
-- 
--
 Krishna Mohan B


More information about the click mailing list