[Click] Repeatable way to kernel panic via Click?

Roman Chertov rchertov at purdue.edu
Wed Aug 22 17:14:28 EDT 2007 

I would setup a serial console server to the click server.  That way you 
can induce a reboot in case the machine locks up.  (deadlock w/o a 
panic).  I heard it is also possible to do serial console over Ethernet 
but I have never tried that.

Roman

dmoore7 at nd.edu wrote:
> Kernel Version: vanilla 2.6.16.13 from kernel.org w/ click patch applied
> Click Version: 1.5.0
> I can go hunting for a stack trace the next time the machine is rebooted... as I
> said lack of physical access means all I get is what I get through my ssh
> connection when it dies.
> I'm setting up another machine with physical and root access so I can further
> narrow down the source of this problem.
> Thanks,
>  - David Moore
> 
> Quoting Beyers Cronje <bcronje at gmail.com>:
> 
>> Doh, as David pointed out use echo instead of cat :)
>>
>> On 8/22/07, dmoore7 at nd.edu <dmoore7 at nd.edu> wrote:
>>> I did not know this... the IT guys will be happy though.
>>> Although, the command is:
>>> echo 60 > ....
>>> Because of course I have no file named '60' kicking around for cat to
>>> print out
>>> of.
>>> Thanks so much for the tip, you are my hero.
>>> - David
>>>
>>> Quoting Beyers Cronje <bcronje at gmail.com>:
>>>
>>>> Hi David,
>>>>
>>>> I'm sure you know this already, what I always do on remote servers is to
>>> set
>>>> panic and panic_on_oops proc setting which will reboot the server on an
>>>> oops:
>>>>
>>>> cat 60 > /proc/sys/kernel/panic
>>>> cat 1 > /proc/sys/kernel/panic_on_oops
>>>>
>>>> So if the kernel oopses at least the server reboots, saved me a lot of
>>>> headaches.
>>>>
>>>> Beyers
>>>>
>>>> On 8/22/07, dmoore7 at nd.edu <dmoore7 at nd.edu> wrote:
>>>>> Hello all,
>>>>> I've been working with click most of the summer and seem to have found
>>> a
>>>>> repeatable way to make a kernel panic (something not-so-fun as my
>>> servers
>>>>> are
>>>>> not physically accessible).
>>>>> It is avoidable with a little care, but I presume folks will want to
>>> know
>>>>> about
>>>>> it.
>>>>>
>>>>> The machine in question has 2 NIC's, and if I ssh in on NIC1, and then
>>>>> attempt
>>>>> to run this click file, without bringing up the second NIC via
>>> ifconfig, I
>>>>> get
>>>>> the following error:
>>>>>
>>>>> [root at filius ~]# click-install ./Filius.click
>>>>>
>>>>> Message from syslogd at filius at Wed Aug 22 10:03:54 2007 ...
>>>>> filius kernel: BUG: spinlock wrong CPU on CPU#2, click-install/12444
>>>>>
>>>>> Message from syslogd at filius at Wed Aug 22 10:03:54 2007 ...
>>>>> filius kernel:  lock: f8c79c44, .magic: dead4ead, .owner:
>>>>> click-install/12444,
>>>>> .owner_cpu: 1
>>>>>
>>>>> At this stage the ssh window stops responding, the machine no longer
>>>>> responds to
>>>>> pings, and hard reset is needed.  I can avoid this if I take care to
>>>>> "ifconfig
>>>>> up" my eth's ahead of time.
>>>>> For those interested, the click file in question is included
>>>>> below.  Basically
>>>>> it performs To and From Device and Host on an eth that was not up at
>>> the
>>>>> time,
>>>>> in addition to a bunch of other stuff.
>>>>> It also spoofs a bunch of IP's for fun and games.
>>>>>
>>>>> To recap: I can work around this issue, but I thought the community
>>> might
>>>>> benefit from me showing this.
>>>>> Thanks,
>>>>> - David Moore
>>>>>
>>>>> The File:
>>>>> ---------------------------------------------------
>>>>> //Routing Configuration for Host Filius
>>>>> // Routing for Device eth2
>>>>> // All from's and to's this real device and its spoof children
>>>>> // Real Device Connection
>>>>> RealFromDevice_eth2 :: FromDevice(eth2);
>>>>> RealToDevice_eth2 :: Queue -> ToDevice(eth2);
>>>>> // Real Host Spoof
>>>>> RealFromHost_eth2 :: FromHost(fake_eth2, 192.168.10.5/24, ETHER
>>>>> 00:04:23:C2:8A:EC);
>>>>> RealToHost_eth2 :: ToHost(fake_eth2);
>>>>>
>>>>> // Spoof Host spoof_eth2_0
>>>>> FromHost(spoof_eth2_0, 192.168.10.11/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_0 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_0 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_0);
>>>>>
>>>>> // Spoof Host spoof_eth2_1
>>>>> FromHost(spoof_eth2_1, 192.168.10.12/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_1 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_1 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_1);
>>>>>
>>>>> // Spoof Host spoof_eth2_2
>>>>> FromHost(spoof_eth2_2, 192.168.10.13/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_2 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_2 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_2);
>>>>>
>>>>> // Spoof Host spoof_eth2_3
>>>>> FromHost(spoof_eth2_3, 192.168.10.14/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_3 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_3 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_3);
>>>>>
>>>>> // Spoof Host spoof_eth2_4
>>>>> FromHost(spoof_eth2_4, 192.168.10.15/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_4 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_4 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_4);
>>>>>
>>>>> // Spoof Host spoof_eth2_5
>>>>> FromHost(spoof_eth2_5, 192.168.10.16/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_5 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_5 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_5);
>>>>>
>>>>> // Spoof Host spoof_eth2_6
>>>>> FromHost(spoof_eth2_6, 192.168.10.17/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_6 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_6 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_6);
>>>>>
>>>>> // Spoof Host spoof_eth2_7
>>>>> FromHost(spoof_eth2_7, 192.168.10.18/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_7 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_7 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_7);
>>>>>
>>>>> // Spoof Host spoof_eth2_8
>>>>> FromHost(spoof_eth2_8, 192.168.10.19/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_8 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_8 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_8);
>>>>>
>>>>> // Spoof Host spoof_eth2_9
>>>>> FromHost(spoof_eth2_9, 192.168.10.20/24, ETHER 00:04:23:C2:8A:EC) ->
>>> Queue
>>>>> ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_9 ::
>>>>> DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_9 :: Queue -> BandwidthRatedUnqueue(1000000000
>>> bps)
>>>>> ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_9);
>>>>>
>>>>> // Spoof Host spoof_eth2_10
>>>>> FromHost(spoof_eth2_10, 192.168.10.21/24, ETHER 00:04:23:C2:8A:EC) ->
>>>>> Queue ->
>>>>> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>>>> SpoofFromHost_spoof_eth2_10
>>>>> :: DelayUnqueue(100 ms)
>>>>> SpoofToHost_spoof_eth2_10 :: Queue -> BandwidthRatedUnqueue(1000000000
>>>>> bps) ->
>>>>> Queue -> DelayUnqueue(100 ms) -> ToHost(spoof_eth2_10);
>>>>>
>>>>>
>>>>> // Now we define all globally used Pipe's and Switches
>>>>> // Entrance for all inc on real devices traffic, splits IP and
>>> 'others'
>>>>> YePipeInc_eth2 :: Classifier( 12/0800,// IP packets
>>>>>                          -);    // Any others
>>>>>
>>>>> // Entrance for all inc on host traffic, splits IP and 'others'
>>>>> YePipeOut_eth2 :: Classifier( 12/0800,// IP packets
>>>>>                          -);    // Any others
>>>>>
>>>>> UnknownOutTee_eth2 :: Tee; // For non-IP traffic to pass unhindered
>>>>> UnknownIncTee_eth2 :: Tee; // For non-IP traffic to pass unhindered
>>>>>
>>>>> PipeOutRealToDevice_eth2_0 :: Print(OutPipe) -> Strip(14) ->
>>>>> EtherEncap(0x0800,
>>>>> 00:04:23:C2:8A:EC, 00:04:23:C2:8C:1E) ->
>>>>> Queue -> BandwidthRatedUnqueue(1000000000 bps) -> Queue ->
>>> DelayUnqueue(0
>>>>> ms) ->
>>>>> RealToDevice_eth2;
>>>>>
>>>>> //This Adds pipe IP and click_name to the routing table for external
>>>>> routes
>>>>>
>>>>> // Splitter to grab any packet destined for a local real or spoof
>>> device
>>>>> LocalDstSplitter_eth2 :: IPClassifier(dst 192.168.10.5, // dst for
>>>>> RealToHost_eth2
>>>>>                                  dst 192.168.10.11, // dst for
>>>>> SpoofToHost_spoof_eth2_0
>>>>>                                  dst 192.168.10.12, // dst for
>>>>> SpoofToHost_spoof_eth2_1
>>>>>                                  dst 192.168.10.13, // dst for
>>>>> SpoofToHost_spoof_eth2_2
>>>>>                                  dst 192.168.10.14, // dst for
>>>>> SpoofToHost_spoof_eth2_3
>>>>>                                  dst 192.168.10.15, // dst for
>>>>> SpoofToHost_spoof_eth2_4
>>>>>                                  dst 192.168.10.16, // dst for
>>>>> SpoofToHost_spoof_eth2_5
>>>>>                                  dst 192.168.10.17, // dst for
>>>>> SpoofToHost_spoof_eth2_6
>>>>>                                  dst 192.168.10.18, // dst for
>>>>> SpoofToHost_spoof_eth2_7
>>>>>                                  dst 192.168.10.19, // dst for
>>>>> SpoofToHost_spoof_eth2_8
>>>>>                                  dst 192.168.10.20, // dst for
>>>>> SpoofToHost_spoof_eth2_9
>>>>>                                  dst 192.168.10.21, // dst for
>>>>> SpoofToHost_spoof_eth2_10
>>>>>                                  -); // All other IP packets
>>>>> // Splitter to grab any packet destined for a connected real device
>>>>> RemoteDstSplitter_eth2 :: IPClassifier(dst 192.168.10.3, // dst for
>>>>> PipeOutRealToDevice_eth2_0
>>>>>                                   dst 192.168.10.4, // dst for
>>>>> PipeOutRealToDevice_eth2_0
>>>>>                                   -); // All other IP packets
>>>>>
>>>>> // Routing Setup
>>>>> // All packets coming in on Real Devices must be passed to YePipeInc
>>>>> // All packets going out on From all real and spoof interfaces must be
>>>>> passed to
>>>>> YePipeOut
>>>>> // Now you see why we call them Ye Pipes!
>>>>> RealFromDevice_eth2 -> Print(FromDevice) -> YePipeInc_eth2
>>>>> RealFromHost_eth2 -> Print(FromRealHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_0 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_1 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_2 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_3 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_4 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_5 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_6 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_7 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_8 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_9 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>> SpoofFromHost_spoof_eth2_10 -> Print(FromSpoofHost) -> YePipeOut_eth2
>>>>>
>>>>> YePipeInc_eth2[1] -> UnknownIncTee_eth2; // Non-IP traffic heads off
>>>>> unhindered
>>>>> YePipeOut_eth2[1] -> UnknownOutTee_eth2; // Non-IP traffic heads off
>>>>> unhindered
>>>>>
>>>>> YePipeInc_eth2[0] -> Print(PipeIncToLclSplitter) ->
>>> LocalDstSplitter_eth2;
>>>>> //
>>>>> All IP traffic hits the first splitter
>>>>> YePipeOut_eth2[0] -> Print(PipeOutToLclSplitter) ->
>>> LocalDstSplitter_eth2;
>>>>> //
>>>>> All IP traffic hits the first splitter
>>>>>
>>>>> // Here all non-IP traffic is passed on: outgoing traffic on all
>>> outgoing
>>>>> devices,
>>>>> // and incoming traffic to all registered hosts.
>>>>> UnknownIncTee_eth2[0] -> RealToHost_eth2
>>>>> UnknownOutTee_eth2[0] -> RealToDevice_eth2
>>>>> UnknownIncTee_eth2[1] -> SpoofToHost_spoof_eth2_0
>>>>> UnknownIncTee_eth2[2] -> SpoofToHost_spoof_eth2_1
>>>>> UnknownIncTee_eth2[3] -> SpoofToHost_spoof_eth2_2
>>>>> UnknownIncTee_eth2[4] -> SpoofToHost_spoof_eth2_3
>>>>> UnknownIncTee_eth2[5] -> SpoofToHost_spoof_eth2_4
>>>>> UnknownIncTee_eth2[6] -> SpoofToHost_spoof_eth2_5
>>>>> UnknownIncTee_eth2[7] -> SpoofToHost_spoof_eth2_6
>>>>> UnknownIncTee_eth2[8] -> SpoofToHost_spoof_eth2_7
>>>>> UnknownIncTee_eth2[9] -> SpoofToHost_spoof_eth2_8
>>>>> UnknownIncTee_eth2[10] -> SpoofToHost_spoof_eth2_9
>>>>> UnknownIncTee_eth2[11] -> SpoofToHost_spoof_eth2_10
>>>>> UnknownOutTee_eth2[1] -> UnknownIncTee_eth2; // For the benefit of
>>> Spoof
>>>>> ARP's,
>>>>> we send non-IP stuff back around
>>>>>
>>>>> // Time to process the IP traffic: which was sent through the filter
>>> for
>>>>> local
>>>>> destinations
>>>>> LocalDstSplitter_eth2[0] -> Print(ToHost) -> RealToHost_eth2;
>>>>> LocalDstSplitter_eth2[1] -> Print(ToHost) -> SpoofToHost_spoof_eth2_0;
>>>>> LocalDstSplitter_eth2[2] -> Print(ToHost) -> SpoofToHost_spoof_eth2_1;
>>>>> LocalDstSplitter_eth2[3] -> Print(ToHost) -> SpoofToHost_spoof_eth2_2;
>>>>> LocalDstSplitter_eth2[4] -> Print(ToHost) -> SpoofToHost_spoof_eth2_3;
>>>>> LocalDstSplitter_eth2[5] -> Print(ToHost) -> SpoofToHost_spoof_eth2_4;
>>>>> LocalDstSplitter_eth2[6] -> Print(ToHost) -> SpoofToHost_spoof_eth2_5;
>>>>> LocalDstSplitter_eth2[7] -> Print(ToHost) -> SpoofToHost_spoof_eth2_6;
>>>>> LocalDstSplitter_eth2[8] -> Print(ToHost) -> SpoofToHost_spoof_eth2_7;
>>>>> LocalDstSplitter_eth2[9] -> Print(ToHost) -> SpoofToHost_spoof_eth2_8;
>>>>> LocalDstSplitter_eth2[10] -> Print(ToHost) ->
>>> SpoofToHost_spoof_eth2_9;
>>>>> LocalDstSplitter_eth2[11] -> Print(ToHost) ->
>>> SpoofToHost_spoof_eth2_10;
>>>>> // This final output is sent on to the remote splitter
>>>>> LocalDstSplitter_eth2[12] -> Print(ToRemoteSplitter) ->
>>>>> RemoteDstSplitter_eth2;
>>>>> // Not local destination, on to routing fun
>>>>>
>>>>> // Now we process all IP traffic that is NOT going to a local
>>> destination
>>>>> RemoteDstSplitter_eth2[0] -> PipeOutRealToDevice_eth2_0;
>>>>> RemoteDstSplitter_eth2[1] -> PipeOutRealToDevice_eth2_0;
>>>>> RemoteDstSplitter_eth2[2] -> Print(Discarded) -> Discard; // No idea
>>> who
>>>>> its
>>>>> going to, drop it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> click mailing list
>>>>> click at amsterdam.lcs.mit.edu
>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>>>>>
>>>
>>>
>>>
>>>
>>>
> 
> 
> 
> 
> 
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>

More information about the click mailing list