[Click] patchless click patches

[Joonwoo Park]

Hi Cliff.

I quite didn't understand what you said.
I guess you was seeing panic from a tree which had misapplied patch..
Are you still seeing panic below with your tree that you updated yesterday?

Joonwoo

On Tue, Nov 16, 2010 at 11:43 PM, Cliff Frey <cliff at meraki.com> wrote:
> I think that the crash that I was seeing earlier was from a misapplied
> patch.  I can still get it to patch by switching between TYPE ETHER and TYPE
> IP though:
> qemu-t:~/co/click/build-linuxmodule$ sudo
> ./tools/click-install/click-install -C ./linuxmodule/ -e 'FromHost(foo, TYPE
> ETHER, ETHER 00:02:03:03:03:03) -> Print -> Discard;'
> qemu-t:~/co/click/build-linuxmodule$ sudo
> ./tools/click-install/click-install -C ./linuxmodule/ -e 'FromHost(foo, TYPE
> IP) -> Print -> Discard;'
> qemu-t:~/co/click/build-linuxmodule$ sudo
> ./tools/click-install/click-install -C ./linuxmodule/ -e 'FromHost(foo, TYPE
> ETHER, ETHER 00:02:03:03:03:03) -> Print -> Discard;'
> Killed
> qemu-t:~/co/click/build-linuxmodule$ dmesg | tail -100|c++filt
> ...
> [ 1525.432234] chatter:   70 | 33330000 00020002 03030303 86dd6000 00000010
> 3afffe80
> [ 1525.732426] chatter:  319 | 33330000 00fb0002 03030303 86dd6000 00000109
> 11fffe80
> [ 1526.170181] foo: Disabled Privacy Extensions
> [ 1526.170921] config:1: While initializing 'FromHost at 1 :: FromHost':
> [ 1526.170925]   cannot set ip address for FromHost devices on this kernel
> [ 1526.170929] Router could not be initialized!
> [ 1532.763774] foo: Disabled Privacy Extensions
> [ 1535.668981] BUG: unable to handle kernel paging request at 2f203ab8
> [ 1535.668988] IP: [<f9ac2903>] AnyDeviceMap::insert(AnyDevice*,
> bool)+0x33/0x90 [click]
> [ 1535.669122] *pdpt = 0000000034ddf001 *pde = 0000000000000000
> [ 1535.669128] Oops: 0000 [#1] SMP
> [ 1535.669131] last sysfs file: /sys/devices/virtual/net/foo/type
> [ 1535.669147] Modules linked in: click proclikefs binfmt_misc ppdev
> parport_pc psmouse i2c_piix4 serio_raw lp parport floppy 8139too 8139cp mii
> [last unloaded: click]
> [ 1535.669162]
> [ 1535.669167] Pid: 2127, comm: click-install Not tainted
> 2.6.35-22-generic-pae #35-Ubuntu /Bochs
> [ 1535.669176] EIP: 0060:[<f9ac2903>] EFLAGS: 00010002 CPU: 1
> [ 1535.669298] EIP is at AnyDeviceMap::insert(AnyDevice*, bool)+0x33/0x90
> [click]
> [ 1535.669301] EAX: 2f203a68 EBX: f9b5e500 ECX: 00000000 EDX: f4c5cc00
> [ 1535.669304] ESI: 00000282 EDI: 2f203ab8 EBP: f3cd1cc4 ESP: f3cd1cb0
> [ 1535.669307]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> [ 1535.669311] Process click-install (pid: 2127, ti=f3cd0000 task=f4adbf70
> task.ti=f3cd0000)
> [ 1535.669313] Stack:
> [ 1535.669315]  f9ac9200 f4c5cc00 f4c5cc00 f3cd1dac f4c5cc3c f3cd1d80
> f9aca3f0 00000000
> [ 1535.669322] <0> f4c5cc00 f3cd1dac f9b1ebcc 00000003 f9b1b5b4 f4c5cc3c
> f9b1ebc5 00000002
> [ 1535.669329] <0> f9b1b73a f4c5cc70 f4c5cc74 f9b1ebc0 00000000 f9b1b5bb
> f3cd1d64 f9b1ebba
> [ 1535.669337] Call Trace:
> [ 1535.669473]  [<f9ac9200>] ? dev_updown(net_device*, int,
> ErrorHandler*)+0x70/0xd0 [click]
> [ 1535.669599]  [<f9aca3f0>] ? FromHost::configure(Vector<String>&,
> ErrorHandler*)+0x3e0/0x500 [click]
> [ 1535.669707]  [<f9a86781>] ? Router::initialize(ErrorHandler*)+0x1f1/0x4b0
> [click]
> [ 1535.669875]  [<f9b058bf>] ? write_config(String const&, Element*, void*,
> ErrorHandler*)+0x11f/0x1e0 [click]
> [ 1535.669983]  [<f9a8422e>] ? Handler::call_write(String const&, Element*,
> ErrorHandler*) const+0xae/0x100 [click]
> [ 1535.670029]  [<c01e2116>] ? unlock_page+0x46/0x50
> [ 1535.670046]  [<c01fba78>] ? __do_fault+0x468/0x5e0
> [ 1535.670168]  [<f9b08007>] ? handler_do_write+0x227/0x710 [click]
> [ 1535.670268]  [<f9a641e5>] ? click_lalloc+0x35/0x60 [click]
> [ 1535.670363]  [<f9a5882e>] ? String::create_memo(char*, int,
> int)+0x3e/0x50 [click]
> [ 1535.670458]  [<f9a599a2>] ? String::append_garbage(int)+0xc2/0x150
> [click]
> [ 1535.670480]  [<c0133558>] ? default_spin_lock_flags+0x8/0x10
> [ 1535.670509]  [<c05efbef>] ? _raw_spin_lock_irqsave+0x2f/0x50
> [ 1535.670524]  [<c016e26f>] ? finish_wait+0x4f/0x70
> [ 1535.670645]  [<f9b07a8a>] ? lock_config(char const*, int, int)+0x9a/0xc0
> [click]
> [ 1535.670766]  [<f9b08555>] ? handler_flush+0x65/0xa0 [click]
> [ 1535.670784]  [<c0220821>] ? filp_close+0x31/0x80
> [ 1535.670789]  [<c02208e5>] ? sys_close+0x75/0xc0
> [ 1535.670794]  [<c05f00f4>] ? syscall_call+0x7/0xb
> [ 1535.670796] Code: 08 e8 12 71 64 c6 84 c9 89 c3 74 4c 8b 42 48 85 c0 74
> 65 8b 40 54 85 c0 78 5e 83 e0 3f 8d 7c 83 04 8b 44 83 04 eb 07 90 8d 78 50
> <8b> 40 50 85 c0 75 f6 c7 42 50 00 00 00 00 89 17 80 4a 4f 01 84
> [ 1535.670837] EIP: [<f9ac2903>] AnyDeviceMap::insert(AnyDevice*,
> bool)+0x33/0x90 [click] SS:ESP 0068:f3cd1cb0
> [ 1535.670961] CR2: 000000002f203ab8
> [ 1535.670971] ---[ end trace b4828044537e6f22 ]---
> Cliff
> On Tue, Nov 16, 2010 at 11:12 PM, Joonwoo Park <joonwpark81 at gmail.com>
> wrote:
>>
>> This is cool. Thanks so much Cliff.
>>
>> Interesting.. I'm almost using the same environment with yours but was
>> not able to reproduce oops by reconfiguring config.
>> I ran this on ubuntu 9.10 + linux 2.6.35 inside kvm virtual machine.
>> On the top of your today's branch, I also applied the patch that I
>> posted to set mac address for IP type but it shouldn't be difference I
>> think.
>>
>> If you have chance, please let me see call back trace I'll have a look.
>>
>> Joonwoo.
>>
>> On Tue, Nov 16, 2010 at 10:23 PM, Cliff Frey <cliff at meraki.com> wrote:
>> > I'm not sure at all what the current best-guess set of patches is to
>> > make
>> > patchless fully work.  I cobbled together enough patches from the list
>> > and
>> > did very very basic testing on my machine, and came up with this branch
>> >
>> > https://github.com/clifffrey/click/tree/working-fromhost-fixincludes
>> >
>> > It roughly works for me on first, but it crashed when I reconfigured it.
>> >  I'm running ubuntu 10.10, with 2.6.35-22-generic.  It is running inside
>> > of
>> > a qemu virtual machine.
>> >
>> > Perhaps I will spend more time debugging, not sure if I'll have time
>> > soon or
>> > not.  I will keep the branch up to date with anything that I fix though.
>> >
>> > Cliff
>> >
>> > On Tue, Nov 16, 2010 at 5:29 PM, Sascha Alexander Jopen <
>> > jopen at informatik.uni-bonn.de> wrote:
>> >
>> >> Hey,
>> >>
>> >> today i tried FromHost with your patch applied. The ethernet address is
>> >> now set as expected. I could not yet test the element because of
>> >> possible errors in my click script. I will check this tomorrow.
>> >>
>> >> I tried to fix the problem with not being able to set the ip address as
>> >> well. Since devinet_ioctl seems no longer available in recent kernels,
>> >> i
>> >> tried to use inet_ioctl instead, which passes through to devinet_ioctl
>> >> for certain commands. inet_ioctl expects a struct socket as an argument
>> >> however, which i do not know how to create and bind to the correct
>> >> interface. I tried to fill in only the struct sock *sk part, obtained
>> >> through inet_ctl_sock_create, as the socket itself should never be used
>> >> by inet_ioctl. The machine crashed with a kernel panic on testing.
>> >>
>> >> Further investigation will follow. As i'm new to kernel level
>> >> programming, figuring out the problem may take some time, however.
>> >>
>> >> Sascha
>> >>
>> >>
>> >> Am 15.11.2010 22:50, schrieb Joonwoo Park:
>> >> > CC'ing Eddie.
>> >> >
>> >> > On Sun, Nov 14, 2010 at 2:32 PM, Joonwoo Park <joonwpark81 at gmail.com>
>> >> wrote:
>> >> >> Hi,
>> >> >>
>> >> >> Thanks for testing.  Can you try this patch once again?
>> >> >> It seems to me linuxmodule FromHost never allowed to set ethernet
>> >> >> address by it's argument if type is IP.
>> >> >> It has been clearing ethernet address with 0 always after parsing
>> >> argument.
>> >> >>
>> >> >> Eddie,
>> >> >> Would you mind to shed on me why FromHost is doing that?
>> >> >>
>> >> >> Thanks,
>> >> >> Joonwoo
>> >> >>
>> >> >> ---
>> >> >> diff --git a/elements/linuxmodule/fromhost.cc
>> >> b/elements/linuxmodule/fromhost.cc
>> >> >> index 8b76672..6bde049 100644
>> >> >> --- a/elements/linuxmodule/fromhost.cc
>> >> >> +++ b/elements/linuxmodule/fromhost.cc
>> >> >> @@ -205,9 +205,7 @@ FromHost::configure(Vector<String> &conf,
>> >> >> ErrorHandler *errh)
>> >> >>     }
>> >> >>
>> >> >>     // set type
>> >> >> -    if (type == "IP")
>> >> >> -       _macaddr = EtherAddress();
>> >> >> -    else if (type != "ETHER" && type != "")
>> >> >> +    if (type != "ETHER" && type != "IP")
>> >> >>        return errh->error("bad TYPE");
>> >> >>
>> >> >>     // set up queue
>> >> >> ---
>> >> >>
>> >> >> On Sun, Nov 14, 2010 at 2:37 AM, Sascha Alexander Jopen
>> >> >> <jopen at informatik.uni-bonn.de> wrote:
>> >> >>> Hey,
>> >> >>>
>> >> >>> thank you for the patch, Joonwoo.
>> >> >>> Now i can start the click instance without further crashes.
>> >> >>> I have problems setting the encap type and mac address of the fake
>> >> >>> device, though. Although i specified a mac address in the click
>> >> >>> script,
>> >> >>> the encap type of the tun/tap interface is UNSPEC with no mac
>> >> >>> address
>> >> set.
>> >> >>> Is it possible to set these parameters manually after the click
>> >> >>> kernel
>> >> >>> module is installed? Will click detect those changes?
>> >> >>> Not being able to set the fake interfaces ip address shouldn'z be
>> >> >>> that
>> >> >>> much of a problem, i think. Is this due to missing ioctl's in newer
>> >> >>> kernels or what is the problem here?
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Sascha
>> >> >>>
>> >> >>> Am 14.11.2010 07:34, schrieb Joonwoo Park:
>> >> >>>> Hello,
>> >> >>>>
>> >> >>>> Can you please try attached patch to see if it works for you?
>> >> >>>> You have to revert patch 07/12 if you applied already to apply
>> >> attached patch.
>> >> >>>>
>> >> >>>> p.s I couldn't find a nice way to set ip address from kernel
>> >> >>>> module
>> >> >>>> without patching kernel therefore you have to set ip address of
>> >> >>>> your
>> >> >>>> device after configuring FromHost. (You won't be able to set it by
>> >> >>>> FromHost argument)
>> >> >>>> Setting ethernet address by FromHost argument should be still
>> >> >>>> working.
>> >> >>>>
>> >> >>>> Joonwoo.
>> >> >>>>
>> >> >>>> On Tue, Nov 9, 2010 at 9:25 PM, Joonwoo Park
>> >> >>>> <joonwpark81 at gmail.com>
>> >> wrote:
>> >> >>>>> Hi,
>> >> >>>>>
>> >> >>>>> Thanks for call trace and simplified config.
>> >> >>>>> I should have allocated netdev_ops and assign it to
>> >> >>>>> dev->netdev_ops.
>> >> >>>>> It seem to me dev->netdev_ops is NULL.
>> >> >>>>> My bad.  I'll fix and submit new patch as soon as I have chance.
>> >> >>>>>
>> >> >>>>> Joonwoo
>> >> >>>>>
>> >> >>>>> On Tue, Nov 9, 2010 at 3:30 PM, Sascha Alexander Jopen
>> >> >>>>> <jopen at informatik.uni-bonn.de> wrote:
>> >> >>>>>> Hi,
>> >> >>>>>>
>> >> >>>>>> i finally removed the mentioned 'false' requirement and the
>> >> >>>>>> element
>> >> was
>> >> >>>>>> compiled and integrated into the kernel module. When installing
>> >> >>>>>> the
>> >> >>>>>> module with a click script using click-install, the module
>> >> >>>>>> crashes
>> >> with
>> >> >>>>>> a null pointer dereference, however. I didn't have the time to
>> >> >>>>>> do
>> >> >>>>>> further tests with simpler click scripts. I will check this
>> >> >>>>>> within
>> >> the
>> >> >>>>>> next days.
>> >> >>>>>> Attached you will find the kernel call trace for this null
>> >> >>>>>> pointer
>> >> >>>>>> dereference. The corresponding click part:
>> >> >>>>>>
>> >> >>>>>> AddressInfo(localDevice DEVNAME:ip/24 DEVNAME:eth);
>> >> >>>>>> FromHost(fake0, localDevice:ipnet, ETHER localDevice:eth, TYPE
>> >> >>>>>> IP)
>> >> -> ...
>> >> >>>>>>
>> >> >>>>>> DEVNAME will be replaced by an existing network device.
>> >> >>>>>> Maybe this is sufficient for someone to detect the problem.
>> >> >>>>>>
>> >> >>>>>> Sascha
>> >> >>>>>>
>> >> >>>>>> Am 09.11.2010 22:27, schrieb Sascha Alexander Jopen:
>> >> >>>>>>> Hello,
>> >> >>>>>>>
>> >> >>>>>>> the FromHost element simply does not exist after compiling.
>> >> >>>>>>> ToHost,
>> >> >>>>>>> ToHostSniffers and ToDevice, as well as FromDevice, are
>> >> >>>>>>> compiled,
>> >> >>>>>>> FromHost not.
>> >> >>>>>>> When feeding a click script to click-install the following
>> >> >>>>>>> error is
>> >> emitted:
>> >> >>>>>>> <stdin>:144: unknown element class 'FromHost'
>> >> >>>>>>>
>> >> >>>>>>> fromhost.cc contains
>> >> >>>>>>> ELEMENT_REQUIRES(AnyDevice linuxmodule false)
>> >> >>>>>>> The 'false' requirement, may be this prevents compiling this
>> >> element?
>> >> >>>>>>>
>> >> >>>>>>> Sascha
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> Am 09.11.2010 22:06, schrieb Joonwoo Park:
>> >> >>>>>>>> Hi Sascha,
>> >> >>>>>>>>
>> >> >>>>>>>> I guess I'm not quite following FromHost issue here.
>> >> >>>>>>>>
>> >> >>>>>>>> Are you having build failure or any other problem?
>> >> >>>>>>>> When I submitted patchless patches, I also built and fixes
>> >> FromHost
>> >> >>>>>>>> element and confirmed it's building.  I haven't tested element
>> >> though.
>> >> >>>>>>>> Did you apply '07/12 FromHost: netdev_ops for linux 2.6.33+'?
>> >> >>>>>>>>
>> >> http://www.mail-archive.com/click@amsterdam.lcs.mit.edu/msg04242.html
>> >> >>>>>>>>
>> >> >>>>>>>> Thanks,
>> >> >>>>>>>> Joonwoo
>> >> >>>>>>>>
>> >> >>>>>>>> On Tue, Nov 9, 2010 at 11:42 AM, Sascha Alexander Jopen
>> >> >>>>>>>> <jopen at informatik.uni-bonn.de> wrote:
>> >> >>>>>>>>> Hey,
>> >> >>>>>>>>>
>> >> >>>>>>>>> i compiled for a recent kernel from ubuntu, namely
>> >> 2.6.35-22-generic. I
>> >> >>>>>>>>> didn't apply your NETREG_REGISTERED patch, but it seems,
>> >> everything
>> >> >>>>>>>>> compiled fine.
>> >> >>>>>>>>> I think this patch part is included in Joonwoos "[PATCH
>> >> >>>>>>>>> 04/12]
>> >> >>>>>>>>> Configury: prepend net_device namespace for
>> >> >>>>>>>>> NETREG_REGISTERED"
>> >> >>>>>>>>> As the enum in question is part of the net_device struct,
>> >> everything
>> >> >>>>>>>>> should work as expected. Applying your patch leads to
>> >> >>>>>>>>> compiler
>> >> errors on
>> >> >>>>>>>>> my system.
>> >> >>>>>>>>>
>> >> >>>>>>>>> So basically i am right, that FromHost is completely missing,
>> >> currently?
>> >> >>>>>>>>> It would be nice, if someone could point me to the right
>> >> direction. What
>> >> >>>>>>>>> is the problematic part in implementing this element?
>> >> >>>>>>>>>
>> >> >>>>>>>>> Regards,
>> >> >>>>>>>>> Sascha
>> >> >>>>>>>>>
>> >> >>>>>>>>> Am 09.11.2010 19:35, schrieb Roman Chertov:
>> >> >>>>>>>>>> Sascha,
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> What kernel version did you use?  It appears that for 2.6.35
>> >> >>>>>>>>>> and
>> >> above the
>> >> >>>>>>>>>> NETREG_REGISTERED fix is required as well.  If you start on
>> >> updating FromHost, I
>> >> >>>>>>>>>> will be glad to give you a hand.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Roman
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> On Fri, 05 Nov 2010 20:02:38 +0100 Sascha Alexander Jopen
>> >> >>>>>>>>>> <jopen at informatik.uni-bonn.de> wrote
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>> Hello,
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> i tried click patchless today. I used those patches from
>> >> Joonwoo, which
>> >> >>>>>>>>>>> didn't make it into the repository till now. Especially the
>> >> patches to
>> >> >>>>>>>>>>> fixincludes and the configure scripts were necessary.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> It seems the FromHost element is not included during build.
>> >> >>>>>>>>>>> I
>> >> know this
>> >> >>>>>>>>>>> was the last element which had to be done, but is it really
>> >> still not
>> >> >>>>>>>>>>> finished? What are the missing parts for this element to
>> >> >>>>>>>>>>> work?
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> We are using click userlevel and kernel level isn't really
>> >> necessary but
>> >> >>>>>>>>>>> still nice to have. So maybe i could use some spare time to
>> >> finish the
>> >> >>>>>>>>>>> FromHost element, if there isn't that much missing.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Regards,
>> >> >>>>>>>>>>> Sascha Jopen
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> On 11/03/10 17:42, Roman Chertov wrote:
>> >> >>>>>>>>>>>> This will also require Joonwoo's PATCH1/12.  It removes
>> >> #include
>> >> >>>>>>>>>>>> <linux/autoconf.h>  from configure.in
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> On Tue, 02 Nov 2010 18:44:32 -0700 "Roman Chertov"<
>> >> rchertov at cs.ucsb.edu>
>> >> >>>>>>>>>>>> wrote
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Hello,
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> This works on Fedora 13 running 2.6.35.8 SMP kernel.  The
>> >> second patch is
>> >> >>>>>>>>>>>>> based
>> >> >>>>>>>>>>>>> on patches 2/12 and 3/12 that Joonwoo released.  I had to
>> >> >>>>>>>>>>>>> add
>> >> the
>> >> >>>>>>>>>>>>> NETREG_REGISTERED patch due to scoping issues when an
>> >> >>>>>>>>>>>>> enum
>> >> declared in
>> >> >>>>>>>>>>>>> net_device is used.  Otherwise, the code is same as in
>> >> >>>>>>>>>>>>> github
>> >> master
>> >> >>>>>>>>>>>>> branch.
>> >> >>>>>>>>>>>>> I
>> >> >>>>>>>>>>>>> am not sure if there is a more elegant way to solve that
>> >> issue.
>> >> >>>>>>>>>>>>>
>> >> >>>>>>>>>>>>> Roman
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> _______________________________________________
>> >> >>>>>>>>>>>> click mailing list
>> >> >>>>>>>>>>>> click at amsterdam.lcs.mit.edu
>> >> >>>>>>>>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> _______________________________________________
>> >> >>>>>>>>>>> click mailing list
>> >> >>>>>>>>>>> click at amsterdam.lcs.mit.edu
>> >> >>>>>>>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>
>> >> >>>>>>>>> _______________________________________________
>> >> >>>>>>>>> click mailing list
>> >> >>>>>>>>> click at amsterdam.lcs.mit.edu
>> >> >>>>>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >> >>>>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> _______________________________________________
>> >> >>>>>>> click mailing list
>> >> >>>>>>> click at amsterdam.lcs.mit.edu
>> >> >>>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> _______________________________________________
>> >> >>>>>> click mailing list
>> >> >>>>>> click at amsterdam.lcs.mit.edu
>> >> >>>>>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> click mailing list
>> >> >>> click at amsterdam.lcs.mit.edu
>> >> >>> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >> >>>
>> >> >>
>> >>
>> >> _______________________________________________
>> >> click mailing list
>> >> click at amsterdam.lcs.mit.edu
>> >> https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >>
>> > _______________________________________________
>> > click mailing list
>> > click at amsterdam.lcs.mit.edu
>> > https://amsterdam.lcs.mit.edu/mailman/listinfo/click
>> >
>
>