Basics of Asynchronous I/O

I’ve been meaning to get into at least the basics of asynchronous I/O because I keep seeing odd system calls when running async programs and want to understand how they work. So I did a bunch of reading and experimentation, and this post is the result!

I’ll go over what I’ve learned about the various I/O multiplexing system calls on *nix: select(), poll(), epoll() (Linux-specific), and even kqueue() (BSD-specific)!

Sidenote: I recently stumbled upon the <details> HTML tag and have incorporated it into my blog theme. Occasionally I will have some side discussions or tangents to talk about. They will be neatly folded into a box like the one below, so that I don’t spend too much time talking about a different topic.

What’s the plan and why?

The plan is to create a simple service: a TCP echo server. Anything you send it, it will send back. The twist is that instead of spawning a new process or new thread for each incoming connection, everything will be handled in the same process and thread. This way, the thread is not wasting cycles checking if there are new connections, or if there is new data on existing connections. As long as there is work to do, the thread will do it. Asynchronous I/O might be a bit of a misnomer in the title, and I’m mostly talking about non-blocking I/O. But I might elaborate more in a future post and make this into a series.

A truly production-ready async service would be able to put things on thread-pools, delegate tasks, not just I/O, and various other things you can see in your favorite async frameworks. To keep the scope manageable, we’ll just stick to the description I gave above. Also most code in the article will skip error-checking.

So the plan is to open a TCP socket in a non-blocking way, accept new connections and make sure those are non-blocking as well, and basically just loop over our sockets when we receive data/a new connection. Non-blocking is important. Otherwise we’d have to wait for data on each socket in turn, which is fine if you spawn a thread/process for each socket. But for our use-case, we want to move on to the next socket if the current one doesn’t have any data yet. Otherwise the whole system stalls.

Also, this whole post will be in C, which is a detour from my usual Rust-based antics. This is because I want to understand this stuff with as few abstractions as possible. The only thing I will accept axiomatically is how the kernel implements this stuff. It’s somewhat straightforward to imagine the implementation when seeing the usage anyway.

One service, four programs

There are four different ways of achieving what I’m trying to do, and they all revolve around their own system call. I’ll go roughly in chronological order of their availability (flipping epoll and kqueue simply because I’m less familiar with BSD).

In all programs, I want the same echo service implementation, a simple way to open a TCP socket, log info and error messages, and also handle some dynamic array stuff. As such I created some helper files:

 1#ifndef LOG_H
 2#define LOG_H
 3
 4#include <stdio.h>
 5#include <time.h>
 6
 7#define _LOG_IMPL(fmt, ...) do {       \
 8    struct timespec t;                 \
 9    clock_gettime(CLOCK_REALTIME, &t); \
10    fprintf(                           \
11        stderr,                        \
12        "%lu.%09d " fmt "\n",          \
13        t.tv_sec,                      \
14        (int)t.tv_nsec                 \
15         __VA_OPT__(,)                 \
16        __VA_ARGS__ );                 \
17} while (0)
18
19#define INFO(fmt, ...) _LOG_IMPL("[INFO] " fmt __VA_OPT__(,) __VA_ARGS__)
20#define WARN(fmt, ...) _LOG_IMPL("[WARN] " fmt __VA_OPT__(,) __VA_ARGS__)
21#define ERRO(fmt, ...) _LOG_IMPL("[ERR!] " fmt __VA_OPT__(,) __VA_ARGS__)
22
23#endif // LOG_H

1789588582.670519197 [INFO] An informative message

struct DynamicArray<type> {
    size_t n;
    size_t capacity;
    type *elems;
};

// Declare new array type called `type_name`, holding `type`s
arr_decl(type_name, type);
// Initalize an array of `type`s
arr_init(type);
// Free an array
arr_free(arr);
// Index the array
arr_at(arr, index);
// Get the raw data pointer, for passing to APIs
arr_raw(arr);
// Get the length
arr_len(arr);
// Push an element to the end
arr_push(arr, x);
// Pop an element from the end into `x`
arr_pop(arr, x);
// Concatenate two arrays
arr_concat(dst, src);
// Remove an element from the array when `expr` is true
arr_remove(arr, it, expr);
// Clear elements from array
arr_clear(arr);
// Helper macro to iterate over an array
arr_foreach(arr, it) { /* Do something with *it */ }

 1#ifndef SOCKET_UTIL_H
 2#define SOCKET_UTIL_H
 3
 4#include "dynarr.h"
 5
 6// Dynamic array of file descriptors (ints)
 7arr_decl(FdArr, int);
 8
 9int init_socket(short port);
10void accept_new_connections(int socket, FdArr *connections);
11
12#endif // SOCKET_UTIL_H

1#ifndef ECHO_H
2#define ECHO_H
3
4int echo(int fd);
5
6#endif // ECHO_H

 1#!/usr/bin/env/bash
 2
 3set -euo pipefail
 4
 5remote=${IP:-127.0.0.1}
 6port=${PORT:3000}
 7
 8rm -f output.txt
 9
10for i in $(seq $1); do
11    printf "Hello from %d\n" $i \
12        | nc -w1 "${remote}" "${port}" \
13        >>output.txt &
14hone
15
16wait
17
18wc -l <output.txt

The implementation can be found in the companion repo. When I was creating the init_socket() function, I read in the man-pages that you can add the SOCK_NONBLOCK flag to the call to accept() and make the socket non-blocking immediately! This is part of POSIX.1-2024, the latest standard. If I add the compiler definition -D_POSIX_C_SOURCE=202405L, I should get access to this functionality and avoid some extra setup. And since it’s in POSIX, this should just work on BSD later on¹! I needed at least 200809 anyway for the timestamps in my logging.

I think that’s all the setup needed. If you’ve seen the declarations in the helper files above, this should be pretty easy to follow along with.

Structure of each server

All servers will essentially be an infinite loop in which the first step is to wait for something to happen on one of the sockets we’re monitoring. In pseudo-code, it looks something like this:

init_socket();
open_connections = init_array();

while (1) {
    wait_for_events(&open_connections);

    if (event on listening socket) {
        accept_new_connections();
        add_new_connections(&open_connections);
    }

    for (connection in open_connection) {
        if (connection received data)
            echo(data);

        if (error on connection) {
            remove_connection(&open_connections);
            close(connection);
        }
    }
}

They will differ slightly in the setup and event handling, but in essence this is all we will do. Send data back if we got some, close the connection if anything went wrong.

In the beginning, there was select()

Our first stop is at the select(2) system call. Reading the man-page, it takes three sets of descriptors (essentially just which file descriptors/sockets we want to be notified of events on), a timeout value which we’ll set to wait forever, and an int describing the number of file descriptors.

int select(
    int nfds,
    fd_set *readfds,
    fd_set *writefds,
    fd_set *exceptfds,
    struct timeval *timeout
);

We don’t really care about writefds, so that will be NULL. We’ll just assume that the socket is ready to accept writes if we get a read. We’ll keep a dynamic array of active open connections around, and use that to populate readfds and exceptfds. These arguments correspond to which file descriptors we want to monitor for the possibility to read data from them, and for exceptional events (errors mostly). In our case we want to monitor all open connections for both.

That nfds argument is not the number of file descriptors we’re watching. It’s actually supposed to be set to the highest-numbered file descriptor in any of the three sets, plus one. Kind of a weird setup. According to the man-page, the maximum this can be is defined by FD_SETSIZE, which in glibc is defined to be 1024. So we will not be able to monitor more than 1024 connections. I’ll just take a shortcut here and set it to the max and call it a day.

Essentially, fd_set is just a bit string (with fixed size 1024 in glibc). We set the bit corresponding to the number of a specific file descriptor we want to monitor for an event. So if we have a connection open with file descriptor 5, we should set bit 5 in our readfds argument. We do this for each active connection we have at the start of each loop.

When select() eventually returns, bits will be set in the readfds and exceptfds according to which event happens. The header provides the FD_ISSET macro to check if a specific bit is set or not.

So our loop body will look something like this:

// Reset the fd_sets
FD_ZERO(&readfds);
FD_ZERO(&exceptfds);

// Monitor the listening socket
FD_SET(socket_fd, &readfds);
FD_SET(socket_fd, &exceptfds);

// Monitor all active connections
arr_foreach(&active_connections, connection) {
    FD_SET(*connection, &readfds);
    FD_SET(*connection, &exceptfds);
}

// Wait for events
int result = select(
    FD_SETSIZE, &readfds, NULL, &exceptfds, NULL);

// Accept new connections if the listening socket
// received a read event
FdArr new_connections = arr_init(int);
if (FD_ISSET(socket_fd, &readfds))
    accept_new_connections(socket_fd, &new_connections);

// Check for data on each open connection
arr_foreach(&active_connections, connection) {
    if (!FD_ISSET(*connection, &readfds))
        continue;

    if (FD_ISSET(*connection, &exceptfds)) {
        // Remove connection later if failed
        arr_push(&to_remove, *connection);
        continue;
    }

    int ret = echo(*connection);
    if (ret < 0)
        arr_push(&to_remove, *connection);
}

// Add new connections for next loop
arr_concat(&active_connections, &new_connections);

// Remove failed connections
arr_foreach(&to_remove, connection) {
    arr_remove(&active_connections, x, *x == *connection);
    close(*connection);
}

This is… fine. It’s a bit finicky to have to have to reset and set which file descriptors we want to monitor in every loop iteration. And we have to loop through every open connection, even ones that didn’t have any events for us to handle. It feels very manual. But it works! running test.sh 1000, it handles 1000 incoming connections completely without issue. Running test.sh 2000, however:

$ ./test.sh 2000
1020

Yeah there we can see that only 1020 connections made it through.

And if we try to scale back and run 1000 connections again?

$ ./test.sh 1000
0

The server has completely locked up. Not great. But at least it works for a small number of descriptors.

Enter poll()

So select() works, but is pretty limited. We’d like to be able to monitor way more connections than 1020. Maybe even 10'000 or so. The man-page for select() on my Linux machine even says this explicitly:

“All modern applications should instead use poll(2) or epoll(7), which do not suffer this limitation.”

Linux man-pages 6.17, select(2)

So let’s try that first suggestion: poll(2)! It’s also specified in POSIX, so we’ll be able to use it in BSD-land. First order of business would be to look at the function signature to get a feel for how to use it:

int poll(struct pollfd *fds, nfds_t nfds, int timeout);

That looks a whole lot simpler than the signature for select! It’s just a pointer to a list of struct pollfds, how big that array is, and a timeout value in milliseconds which we will set to wait forever (-1).

The struct pollfd is a really simple structure:

struct pollfd {
    int fd;         /* file descriptor */
    short events;   /* requested events */
    short revents;  /* returned events  */
};

For each pollfd, you populate the .events with a bitmask of events you want to be notified of (provided by some nice helper macros like POLLIN for input), and when poll() returns, the .revents field will have bits set according to which events happened.

I like it! Much better than having three separate structures you have to juggle with that get overwritten each loop so you have to reset them. With this solution, you can tell the kernel exactly which events you’re interested in, and only the .revents field is modified. Which events you’ve requested stays the same each iteration unless you change it.

With that, it’s pretty simple to modify the loop body for select() to instead use poll(). Structurally they’re pretty much identical.

 1// Wait
 2int poll_result =
 3    poll(arr_raw(&poll_fds), arr_len(&poll_fds), -1);
 4
 5// Accept new connections if any arrived
 6if (arr_get(&poll_fds, 0).revents & POLLIN) {
 7    FdArr new_connections = arr_init(int);
 8    accept_new_connections(sock_fd, &new_connections);
 9    arr_foreach(&new_connections, fd) {
10        // Set up new polling struct for new connection
11        struct pollfd new_pollfd = {
12            .fd = *fd,
13            .events = POLLIN,
14            .revents = 0
15        };
16        arr_push(&poll_fds, new_pollfd);
17    }
18}
19
20// Echo on all connections
21arr_foreach(&poll_fds, poll_fd) {
22    // Skip uninteresting events
23    bool is_listening_socket = (poll_fd->fd == sock_fd);
24    bool is_read_event = (poll_fd->revents & POLLIN);
25
26    if (is_listening_socket || !is_read_event)
27        continue;
28
29    int ret = echo(poll_fd->fd);
30    if (ret < 0)
31        arr_push(&to_remove, *connection);
32}
33
34// Check for failures
35arr_foreach(&poll_fds, poll_fd) {
36    if (poll_fd->fd == sock_fd)
37        continue; // Ignore failures on listening socket
38
39    // Any event other than POLLIN is an error
40    bool failed = (poll_fd->revents & ~POLLIN) != 0;
41
42    if (failed) {
43        INFO(
44            "Connection %d closing:%s%s%s",
45            poll_fd.fd,
46            (poll_fd.revents & POLLHUP) ? " POLLHUP" : "",
47            (poll_fd.revents & POLLNVAL) ? " POLLNVAL" : "",
48            (poll_fd.revents & POLLERR) ? " POLLERR" : ""
49        );
50        arr_push(&to_remove, poll_fd);
51    }
52}
53
54// Remove connections that failed this loop
55arr_foreach(&to_remove, poll_fd) {
56    close(poll_fd->fd);
57    arr_remove(&poll_fds, x, x->fd == poll_fd->fd);
58}
59arr_clear(&to_remove);

As you can see, we’re doing basically the same thing. But instead of having a readfds and an exceptfds, we just tell poll() that we want to monitor for POLLIN (read) events, and we get POLLERR & co. (error) events for free. Then we can just read .revents to see what happened and handle it accordingly. As you can see on lines 24 and 40, it’s just some bit masking against the events we’re interested in.

Alright, so now we have a more powerful server, without the problem of limited sets of file descriptors we can monitor! Surely this can handle 10'000 connections at once, right?

$ ./test.sh 10000
1020

What? I thought the whole point of poll() over select() was that I could have more file descriptors!?

After a bunch of wild goose chasing², thinking there was a bug in my code, it turns out my code was fine. It’s just that by default, Linux limits the number of file descriptors you can have open to… 1024. So all I had to do was increase that limit to something higher than 10'000 and…

$ ./test.sh 10000
10000

Nice!

const struct rlimit new_limit = {
    .rlim_cur = 4096,   /* soft cap */
    .rlim_max = 16384   /* hard cap */
};
setrlimit(RLIMIT_NOFILE, &new_limit);

So now we have essentially the same server, but not limited to 1024 file descriptors. But we also have the same server in that we’re manually keeping track of struct pollfds in a dynamic array, having to juggle memory, and loop through all open descriptors. Even if no events have happened on them. If only someone had invented a better interface for this.

In Linux-land: epoll()

Oh hey look, a better interface for this! epoll(7) is a Linux-specific API for I/O events. Like poll(2), you ask it to keep track of certain events on certain file descriptors by bitmasking a field in a structure. But the way it does it is a lot better. Let’s, as is tradition by now, check the function signature to get a feel for how it works. However, this is not one function, but three. But let’s start with the main one, most similar to select() and poll() and we’ll get to the other two when relevant.

int epoll_wait(
    int epfd,
    struct epoll_event *events,
    int n,
    int timeout
);

timeout we’ll set to -1 to wait forever as before. events and n are obviously related in that n is the size of the events array. But this is not an array we’re setting up and keeping track of ourselves. It’s actually a memory space we provide to the function, which it will write events into for us. It will contain only file descriptors which have had events. So we’re not looping over absolutely everything, only what’s necessary. The return value is the number of events, which we can use for our looping purposes.

The first argument, epfd, is a special file descriptor created by epoll_create1() which is one of the other two functions. The file descriptors we want to monitor are registered to epfd, making it a sort of handle for the group of monitored descriptors. We register them using the last of our set of functions:

int epoll_ctl(
    int epfd,
    int op,
    int fd,
    struct epoll_event *event
);

This function can add, remove, or modify file descriptors registered to epfd. That’s what the op argument is. fd is the descriptor we want to monitor. The event argument contains the file descriptor again for some reason, and which events we want to be notified of, analagous to poll()’s struct pollfd.events.

And that’s it! We don’t have to keep track of the file descriptors ourselves. We don’t even have to remove them from a buffer or call epoll_ctl to remove them when we’re done. If we close() a file descriptor registered to epfd it will automatically be removed from epfd too! This means our loop body can contain a single for-loop:

// Wait
int nfds = epoll_wait(epfd, events, MAX_EVENTS, -1);

// Loop over all events
for (int n = 0; n < nfds; n++) {
    struct epoll_event event = events[n];

    // Accept new connections if any arrived
    if (event.data.fd == sock_fd) {
        FdArr new_connections = arr_init(int);
        accept_new_connections(sock_fd, &new_connections);
        arr_foreach(&new_connections, fd) {
            ev.events = EPOLLIN | EPOLLRDHUP | EPOLLHUP;
            ev.data.fd = *fd;
            epoll_ctl(epfd, EPOLL_CTL_ADD, *fd, &ev);
        }
        arr_free(&new_connections);
        continue;
    }

    INFO("fd: %d, events:%s%s%s%s%s%s", event.data.fd,
            (event.events & EPOLLIN) ? " EPOLLIN" : "",
            (event.events & EPOLLOUT) ? " EPOLLOUT" : "",
            (event.events & EPOLLRDHUP) ? " EPOLLRDHUP" : "",
            (event.events & EPOLLPRI) ? " EPOLLPRI" : "",
            (event.events & EPOLLERR) ? " EPOLLERR" : "",
            (event.events & EPOLLHUP) ? " EPOLLHUP" : ""
    );

    // Handle non-listening event
    if (event.events & EPOLLIN) {
        // Result can be ignored, because echo()
        // already closes the fd and logs why
        // if there is an error
        (void)echo(event.data.fd);
    }

    // Any event other than POLLIN is an error
    bool failed = (event.events & ~EPOLLIN) != 0;
    // Remove connections that failed this loop
    if (failed)
        close(event.data.fd);
}

Now this was nice. One simple inner loop, events are put nicely into the events buffer to be looped over, and when we’re done with a file descriptor we just close() it. No fuss, no buffer juggling, no looping over file descriptors that haven’t had anything happen. Very nice!

Another thing epoll is capable of is edge-triggered events. Unlike select and poll, which only support level-triggered events. I won’t go into detail about that in this article (it’s long enough as it is), but it’s a pretty cool concept. Basically we can subscribe for events to trigger every time they happen, even if their event condition is currently fulfilled, such as a file having data available to read. So if we haven’t read the data already, we’ll get a second event when data is ready again. select and poll only tell us that data is ready, not how many times it has been. Pretty cool, but out of scope for this article.

A journey into BSD: kqueue()

In my initial research of the previous three syscalls, I had heard whispers of a fourth one, implemented in BSD-land: kqueue(). Being curious, I spun up a VM with FreeBSD 15.1 so I could implement the echo service with this fourth syscall as well.

Detour: Getting *nix’ed

Since I knew that I wanted to write a kqueue() implementation of the echo server as well, I figured I should write all my code to POSIX standard. That way I figured I wouldn’t have any problems with the BSD implementation, because the supporting was supposedly portable. I did it by simply slapping _POSIX_C_SOURCE=202405L into my Makefile’s CFLAGS like so:

CFLAGS=-std=c23 -D_POSIX_C_SOURCE=202405L

socket-util.o: socket-util.c socket-util.h log.h dynarr.h
echo.o: echo.c echo.h log.h
%-server.o: %-server.c log.h dynarr.h

%-server: %-server.o socket-util.o
	$(CC) $(CFLAGS) $^ -o $@

I needed that feature test macro for clock_gettime and SOCK_NONBLOCK anyway. This was my entire Makefile when developing the first three echo services, on Linux. Then I figured “Hey, I before I develop the kqueue version, let’s just compile the select and poll versions on FreeBSD to see them work there too!”

cc -std=c23 -D_POSIX_C_SOURCE=202405L select-server.c  -o select-server
ld: error: undefined symbol: init_socket
>>> referenced by select-server.c
>>>               /tmp/select-server-b477ff.o:(main)

ld: error: undefined symbol: accept_new_connections
>>> referenced by select-server.c
>>>               /tmp/select-server-b477ff.o:(main)

ld: error: undefined symbol: echo
>>> referenced by select-server.c
>>>               /tmp/select-server-b477ff.o:(main)
cc: error: linker command failed with exit code 1 (use -v to see invocation)

No such luck. Loads of linker errors when running make. This was really confusing initially, because I was sure this Makefile worked. It worked when I was on Linux, after all!

Oh.

BSD’s make is different from the one on Linux, which is GNU make. Of course. These Makefiles are written in different dialects and have different features. Indeed, inspecting what the compiler command ended up being (and using make’s debug printing functionality with the -d flag) the problem was clear. BSD make doesn’t have the pattern rules that exist in GNU make. I usually use these to write shorter makefiles when I know I’ll have common prefixes/suffixes to multiple targets. So since I wasn’t trying to built a literal %-server, BSD couldn’t find that rule and instead used the standard fallback rule, trying to build select-server from only select-server.c. Of course we’re getting linker errors!

No matter, we can define each of our implementations in an array variable and use BSD make’s .for loop! But then we won’t be able to use the same Makefile on both Linux and BSD. Hm. In the companion repo I’ll include both a GNU and BSD-flavored Makefile. But for the sake of complete portability I will also include a POSIX-flavored one! Either way, let’s try compiling with the updated targets.

cc  -std=c23 -D_POSIX_C_SOURCE=202405L -c socket-util.c -o socket-util.o
socket-util.c:14:51: error: use of undeclared identifier 'SOCK_NONBLOCK'
   14 |     int socket_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
      |                                                   ^
1 error generated.

So that also fails. Turns out SOCK_NONBLOCK is not POSIX? At least not according to FreeBSD. Searching through the system headers, that macro is guarded behind an #ifdef __BSD_VISIBLE directive. Defining that in the Makefile works, but whenever you’re dealing with double underscores you’re probably doing something weird. So instead of doing it like this, I’ll just make the socket non-blocking using fcntl() in a separate step instead, with O_NONBLOCK.³

cc  -std=c23 -D_POSIX_C_SOURCE=202405L -c select-server.c -o select-server.o
cc  -std=c23 -D_POSIX_C_SOURCE=202405L -c socket-util.c -o socket-util.o
cc  -std=c23 -D_POSIX_C_SOURCE=202405L -c echo.c -o echo.o
cc select-server.o socket-util.o echo.o -o select-server
cc  -std=c23 -D_POSIX_C_SOURCE=202405L -c poll-server.c -o poll-server.o
cc poll-server.o socket-util.o echo.o -o poll-server

Success! Now we can get on with the final implementation. Curiously, glibc doesn’t guard this at all. Not with a specific POSIX version or anything. Even though it is POSIX as of 2024, I expected enabling POSIX mode lower than 2024 to cause glibc to complain as well.

Back on track: Implementing with kqueue()

Like epoll, kqueue operates on a special descriptor, returned by the kqueue() function, puts events into a memory buffer provided, and returns how many events were triggered.

As per our time-honored tradition: let’s check the function signature. The special descriptor is created by kqueue() and events are waited for with the kevent() function:

int kevent(
    int kq,
    const struct kevent *changelist,
    int nchanges,
    struct kevent *eventlist,
    int nevents,
    const struct timespec *timeout
);

Essentially the same as epoll_wait, but with two lists instead of one. eventlist is basically the same as for epoll_wait. It’s the list the events will be put into that we loop over. The first one, changelist, essentially bakes the functionality of epoll_ctl into the epoll_wait function. Any changes are applied before filling the eventlist.

Now let’s compare struct kevent to struct epoll_event. The epoll variant had two fields: .fd and .events. Simple. The descriptor and events we’re interested in. How much difference can there be?

struct kevent {
    uintptr_t ident;
    short     filter;
    u_short   flags;
    u_int     fflags;
    int64_t   data;
    void      *udata;
    uint64_t  ext[4];
};

Woah!

The explanation for all of these events is long and best explained by the kqueue(2) man-page.

The gist of it is that kqueue can register multiple kinds of event filters to each file descriptor being monitored. Each filter can put an event in the eventlist when triggered. But each filter can also have a whole lot of extra configuration, making them extremely powerful. Seriously, open the linked man-page and read the configuration you can do to the filters beginning with EVFILT_.

For our purposes, we only care about the filter EVFILT_READ, which will trigger an event when there is data to read on the file descriptor, analogous to POLLIN and EPOLLIN. On each accepted connection, we’ll add a new event to the changelist with this filter set, and the EV_ADD flag to add it to the kqueue. Essentially what we did by calling epoll_ctl. We don’t really care about any extra data returned in the event either, so it will be extremely similar to the epoll implementation.

// Wait
int n_events = kevent(
    kq,
    arr_raw(&changelist),
    arr_len(&changelist),
    arr_raw(&eventlist),
    arr_len(&eventlist),
    NULL
);
arr_clear(&changelist);

// Loop over all events
for (int i = 0; i < n_events; i++) {
    struct kevent event = eventlist[i];

    // Accept new connections if any arrived
    if ((int)event.ident == sock_fd) {
        FdArr new_connections = arr_init(int);
        accept_new_connections(sock_fd, &new_connections);
        arr_foreach(&new_connections, fd) {
            struct kevent new_event = {
                .ident = *fd,
                .filter = EVFILT_READ,
                .flags = EV_ADD,
                .fflags = 0,
                .data = 0,
                .udata = NULL,
                .ext = {0, 0, 0, 0},
            };
            arr_append(&changelist, new_event);
        }
        continue;
    }

    // Closed file descriptors are removed
    // from the kqueue automatically
    (void)echo(event.ident);

    // Any non-read event is an error
    if (event.flags != 0) {
        INFO("fd: %d, events:%s%s", event.ident,
            (event.flags & EV_ERROR) ? " EV_ERROR" : "",
            (event.flags & EV_EOF) ? " EV_EOF" : ""
        );
        close(event.ident);
    }
}

And that’s all, folks! Very similar to epoll but oh, so much more powerful. I have to say that this is probably my favorite version of them all. Everything good about epoll is in kqueue (including edge-triggered events) and more. Baking the changelist into the waiting call instead of having a separate control call feels pretty nice and the sheer amount of filters and options on the filters makes it hard to beat. Reading some of the options in the man-page really makes you wonder how Linux can fare without it all.

Other options?

There are other options than these. There is POSIX’s aio and Linux’s io_uring to learn about as well. However to my understanding these are a bit different to the syscalls covered here. So I’ll leave them for another day.

On Windows you have IOCP. But I don’t do stuff for Windows very much at all, so I left that out of scope for this article.

Summary

All in all, asynchronous I/O in this manner is pretty simple. The kernel does all the dirty work of looping over the file descriptors and just tells us which ones has events we’re interested in. They work in pretty similar ways, but some are clearly better than others. Which one should you use?

• select() – Just don’t use it. The limit on the number of file descriptors makes it unusable. On BSD you can raise it, but why bother when poll() and kqueue() exist?
• poll() – Use it if you want asynchronous I/O but have strict POSIX requirements.
• epoll() – Absolutely use it if you only need Linux support.
• kqueue() – Absolutely use it if you only need BSD support.

And which one is my favorite? Well, I said it only a few paragraphs ago, but kqueue() is the clear winner for me.

The complete code for all these echo service implementations is in the companion repo.