[Click] Extending lib/master.cc to have better kqueue support

[Pekka Nikander]

Forgot the references, sorry about that:

[1] http://wiki.hiit.fi/display/psirpcode/Blackhawk

[2] www.psirp.org

--Pekka Nikander

On 2010-08 -30, at 15:54 , Pekka Nikander wrote:

> We are trying to interface the Blackhawk memory-sharing publish/subscribe-based IPC system [1], developed in the PSIRP project [2], with Click, initially at the user level.  
> 
> One particular problem that we are facing is that the Blackhawk IPC is heavily based on using EVFILT_VNODE kevents, even extending them at the kernel side.  Now, Click as it is currently only supports EVFILT_READ and EVFILT_WRITE kevents.  While it would be possible to change Blackhawk to support them, too, in addition to EVFILT_VNODE kevents, it would not be sufficient since Blackhawk also uses the kevent->udata field for houskeeping, and Click's Master class does not support it.
> 
> Thus, we have ended up looking at how to extend the Master class to have better kevent support, in a way that would also benefit others and not only our project.  Right now we have made some minimal changes to the Master class, allowing it to be more easily subclassed, and created a subclass of Master, called KqueueMaster.  However, this approach has the problem that we also need to modify the click.cc itself, to instantiate a KqueueMaster instead of the regular Master, an approach that I don't consider very elegant.  It works for now, but it is not nice.
> 
> Hence, I'm looking for advice and opinions on how to approach the issue.  Our initial patch (the master.{hh,cc} part) included below.
> 
> --Pekka Nikander
> 
> diff --git a/include/click/master.hh b/include/click/master.hh
> index f0cc360..e2c4861 100644
> --- a/include/click/master.hh
> +++ b/include/click/master.hh
> @@ -69,6 +69,15 @@ class Master { public:
>     static volatile sig_atomic_t signals_pending;
> #endif
> 
> +#if CLICK_KQUEUE_MASTER
> +  protected:
> +    void select_lock_acquire();
> +    void select_lock_release();
> +    void register_kqueue(int fd, short filter, u_int fflags, 
> +                        intptr_t d, void *ud);
> +    virtual void call_kqueued(int fd, struct kevent *event);
> +#endif
> +
>   private:
> 
>     // stick _timer_expiry here so it will most likely fit in a cache line,
> @@ -360,5 +369,19 @@ Element::master() const
>     return _router->master();
> }
> 
> +#if CLICK_KQUEUE_MASTER
> +inline void Master::select_lock_acquire() {
> +    _select_lock.acquire();
> +}    
> +
> +inline void Master::select_lock_release() {
> +    _select_lock.release();
> +}
> +
> +inline void Master::call_kqueued(int /*fd*/, struct kevent */*event*/) {
> +    // Do nothing
> +}
> +#endif
> +
> CLICK_ENDDECLS
> #endif
> 
> diff --git a/lib/master.cc b/lib/master.cc
> index 95ee585..1c80282 100644
> --- a/lib/master.cc
> +++ b/lib/master.cc
> @@ -535,6 +535,26 @@ Master::run_timers()
> 
> #if CLICK_USERLEVEL
> 
> +# if CLICK_KQUEUE_MASTER
> +void
> +Master::register_kqueue(int fd, short filter, u_int fflags, intptr_t d, void *ud) {
> +    if (_kqueue >= 0) {
> +       // Add events to the kqueue
> +       struct kevent kev[2];
> +       int nkev = 0;
> +       EV_SET(&kev[nkev], fd, filter, EV_ADD, fflags, d, ud);
> +       nkev++;
> +       int r = kevent(_kqueue, &kev[0], nkev, 0, 0, 0);
> +       if (r < 0) {
> +           // Not all file descriptors are kqueueable.  So if we encounter
> +           // a problem, fall back to select() or poll().
> +           close(_kqueue);
> +           _kqueue = -1;
> +       }
> +    }
> +}
> +# endif
> +
> namespace {
> enum { SELECT_READ = Element::SELECT_READ, SELECT_WRITE = Element::SELECT_WRITE };
> }
> @@ -560,6 +580,14 @@ Master::register_select(int fd, bool add_read, bool add_write)
>        _pollfds[pi].events |= POLLOUT;
> 
> #if HAVE_USE_KQUEUE
> +# if CLICK_KQUEUE_MASTER
> +    if (add_read) {
> +       register_kqueue(fd, EVFILT_READ, 0, 0, EV_SET_UDATA_CAST ((intptr_t) 0));
> +    }
> +    if (add_write) {
> +       register_kqueue(fd, EVFILT_WRITE, 0, 0, EV_SET_UDATA_CAST ((intptr_t) 0));
> +    }
> +# else
>     if (_kqueue >= 0) {
>        // Add events to the kqueue
>        struct kevent kev[2];
> @@ -580,6 +608,7 @@ Master::register_select(int fd, bool add_read, bool add_write)
>            _kqueue = -1;
>        }
>     }
> +# endif
> #endif
> #if !HAVE_POLL_H || HAVE_USE_SELECT
> @@ -839,6 +868,10 @@ Master::run_selects_kqueue(RouterThread *thread, bool more_tasks)
>                    mask |= Element::SELECT_READ;
>                else if (p->filter == EVFILT_WRITE)
>                    mask |= Element::SELECT_WRITE;
> +# if CLICK_KQUEUE_MASTER
> +               else 
> +                   call_kqueued(fd, p);
> +# endif
>            call_selected(fd, mask);
>        }
>     }
> 
> 
> _______________________________________________
> click mailing list
> click at amsterdam.lcs.mit.edu
> https://amsterdam.lcs.mit.edu/mailman/listinfo/click