Subject: Re: integration of c-ares into various file descriptor based main loops [reposted]

Re: integration of c-ares into various file descriptor based main loops [reposted]

From: Pavel Simerda <psimerda_at_redhat.com>
Date: Thu, 14 Nov 2013 06:32:32 -0500 (EST)

Any further comments?

Pavel

----- Original Message -----
> From: "Pavel Simerda" <psimerda_at_redhat.com>
> To: c-ares_at_cool.haxx.se
> Sent: Thursday, November 7, 2013 7:14:56 PM
> Subject: integration of c-ares into various file descriptor based main loops [reposted]
>
> Hello,
>
> apparently posting to c-ares mailing list is not possible for unsubscribed
> people. Therefore some you only received bits of my original e-mail in
> Jakub's reply. I'm reposting the original e-mail now subscrubed in hope we
> can somehow fix the communication gap. Original text follows...
>
> ****
>
> Hi,
>
> I started a new network name resolver library project (extending the idea of
> glibc's nsswitch) that is using c-ares as its backend. I had the chance to
> design both the external API and the backend API from scratch. While I was
> first searching for inspiration in the c-ares API, I realized that
> `ares_fds()` style API is suboptimal (unless the mainloop is indeed based on
> `select()`).
>
> When designing the APIs, I considered the following well-known options:
>
> * select
> * poll
> * epoll
> * libevent
> * glib main loop
>
> All of them has more or less native support for adding and removing file
> descriptors but for example *epoll* doesn't have support for inspection. My
> APIs take this into account and so they are based on *addition*,
> *modification* and *removal* of file descriptors from the multiplex, all of
> them handled by one function with the following arguments:
>
> * driver: the epoll fd, resolver struct pointer, etc...
> * fd: the file descriptor
> * events: bit array of monitored events, non-zero for *add*/*modify*, zero
> for *delete*
>
> The main difference from `ares_fds()` is that for external API this has to be
> implemented using callbacks (or similar mechanism) bound directly to the
> places where the library needs to start or stop listening to file descriptor
> events. The supplied *callback* then handles addition/modification/removal
> of the file descriptors and you don't have to retrieve the whole file
> descriptor set before each `select()`, `poll()`, `epoll_wait()`,
> `g_main_context_iteration()` nor transform it into a data structure suitable
> for the respective main loop waiting method.
>
> An *epoll* example can be found in the source code of the library:
>
> 11 void
> 12 watch_fd(netresolve_t resolver, int fd, int events, void *user_data)
> 13 {
> 14 int epoll_fd = *(int *) user_data;
> 15
> 16 struct epoll_event event = { .events = events, .data = { .fd =
> fd} };
> 17
> 18 if (epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, &event) == -1 &&
> errno != ENOENT) {
> 19 perror("epoll_ctl");
> 20 abort();
> 21 }
> 22 if (events && epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &event) ==
> -1) {
> 23 perror("epoll_ctl");
> 24 abort();
> 25 }
> 26 }
>
> https://sourceware.org/git/?p=netresolve.git;a=blob;f=tests/test-async.c;h=fc146cd41a45ca744e382ab0258319c1a35d1881;hb=HEAD#l11
>
> On the other hand, the added complexity of `ares_fds()` approach can be
> easily seen in the DNS backend:
>
> 17 void
> 18 register_fds(Data *data)
> 19 {
> 20 netresolve_backend_t resolver = data->resolver;
> 21 fd_set rfds;
> 22 fd_set wfds;
> 23 int nfds, fd;
> 24
> 25 FD_ZERO(&rfds);
> 26 FD_ZERO(&wfds);
> 27 nfds = ares_fds(data->channel, &rfds, &wfds);
> 28
> 29 for (fd = 0; fd < nfds || fd < data->nfds; fd++) {
> 30 if (!FD_ISSET(fd, &rfds) && FD_ISSET(fd, &data->rfds)) {
> 31 FD_CLR(fd, &data->rfds);
> 32 netresolve_backend_watch_fd(resolver, fd, 0);
> 33 } else if (FD_ISSET(fd, &rfds) && !FD_ISSET(fd,
> &data->rfds)) {
> 34 FD_SET(fd, &data->rfds);
> 35 netresolve_backend_watch_fd(resolver, fd,
> POLLIN);
> 36 }
> 37 if (!FD_ISSET(fd, &wfds) && FD_ISSET(fd, &data->wfds)) {
> 38 FD_CLR(fd, &data->wfds);
> 39 netresolve_backend_watch_fd(resolver, fd, 0);
> 40 } else if (FD_ISSET(fd, &wfds) && !FD_ISSET(fd,
> &data->wfds)) {
> 41 FD_SET(fd, &data->wfds);
> 42 netresolve_backend_watch_fd(resolver, fd,
> POLLOUT);
> 43 }
> 44 }
> 45
> 46 data->nfds = nfds;
> 47
> 48 if (!nfds)
> 49 netresolve_backend_finished(resolver);
> 50 }
>
> https://sourceware.org/git/?p=netresolve.git;a=blob;f=backends/dns.c;h=7a006f497d0b24652285f9e3d5d55a463481c6ae;hb=HEAD#l17
>
> Also `ares_process_fd()` seems to assume `select()`.
>
> 110 void
> 111 dispatch(netresolve_backend_t resolver, int fd, int events)
> 112 {
> 113 Data *data = netresolve_backend_get_data(resolver);
> 114
> 115 int rfd = events & POLLIN ? fd : ARES_SOCKET_BAD;
> 116 int wfd = events & POLLOUT ? fd : ARES_SOCKET_BAD;
> 117
> 124 ares_process_fd(data->channel, rfd, wfd);
> 125 register_fds(data);
> 126 }
>
> https://sourceware.org/git/?p=netresolve.git;a=blob;f=backends/dns.c;h=7a006f497d0b24652285f9e3d5d55a463481c6ae;hb=HEAD#l110
>
> Would you consider an additional API as an alternative to `ares_fds()`,
> `ares_timeout()` and `ares_process_fd()` consisting of functions similar to
> the following ones useful?
>
> * void ares_set_fd_callback(ares_channel channel, int (*callback)(int fd,
> int events, void *user_data), void *user_data)
> * void ares_set_timeout_callbacks(ares_channel channel, int (*create)(int
> seconds, void *user_data), int (*remove)(int timeout_id), void *user_data)
> * int ares_dispatch_fd(ares_channel channel, int fd, int events)
>
> Apart from possible improvements of *c-ares*, I will be happy for any
> feedback for the API I'm using in the library. It's in very early stage of
> development and needs more eyes than just the two of mine.
>
> Cheers,
>
> Pavel
>
Received on 2013-11-14