I recently happened upon an implementation of popen() (different API, same idea) using clone(2), and so I opened an issue requesting use of vfork(2) or posix_spawn() for portability. It turns out that on Linux there's an important advantage to using clone(2). I think I should capture the things I wrote there in a better place. A gist, a blog, whatever.

So here goes.

Long ago, I, like many Unix fans, thought that fork(2) and the fork-exec process spawning model were the greatest thing, and the Windows sucked for only having exec*() and spawn*(), the last being a Windows-ism.

After many years of experience, I learned that fork(2) is in fact evil. And vfork(2), long said to be evil, is in fact goodness. A slight variant of vfork(2) that avoids the need to block the parrent would be even better (see below).

Extraordinary statements require explanation, so allow me to explain.

I won't bother explaining what fork(2) is -- if you're reading this, I assume you know. But I'll explain vfork(2) and why it was said to be harmful. vfork(2) is very similar to fork(2), but the new process it creates runs in the same address space as the parent as if it were a thread, even sharing the same stack as the thread that called vfork(2)! Two threads can't share a stack, so the parent is stopped while the child does its thing: either exec*(2) or _exit(2).

Now, 4.3BSD added vfork(2), and a few years later 4.4BSD removed it as it was by then considered harmful. Most subsequent man pages say as much. But the derivatives of 4.4BSD restored it and do not call it harmful. There's a reason for this: vfork(2) is much cheaper than fork(2) -- much, much cheaper. That's because fork(2) has to either copy the parent's address space, or arrange for copy-on-write (which is supposed to be an optimization to avoid unnecessary copies). But even COW is very expensive because it requires modifying memory mappings, taking expensive page faults, and so on. Modern kernels tend to seed the child with a copy of the parent's resident set, but if the parent has a large memory footprint (e.g., is a JVM), then the RSS will be huge. So fork(2) is inescapably expensive except for small programs with small footprints (e.g., a shell).

So you begin to see why fork(2) is evil. And I haven't yet gotten to fork-safety perils! Fork-safety considerations are a lot like thread-safety, but it is harder to make libraries fork-safe than thread-safe. I'm not going to go into fork-safety here: it's not necessary.

Why did I ever think fork(2) was elegant then? It was the same reason that everyone else did and does: spawn() and posix_spawn() and such functions are extremely complex, and they have to be because there is an enormous number of things one might do between fork() and exec() in, say, a shell, but with fork() and exec() one can does not need a language or API that can express all those things. It gave the Unix's creators the ability to move all that complexity into user-land, where it's much easier to develop -- it made them more productive, perhaps much more so. The price Unix's creators paid for that elegance was the need to copy address spaces -- back then programs and processes were small, but now they're huge, and that makes copying even just a parent's resident set, and page table fiddling for the rest, is extremely expensive.

But vfork() has all that elegance, and none of the downsides of fork()!

vfork() does have one bit of inelegance: that the parent and child share a stack, necessitating that the parent be stopped until the child exec()s or _exit()s. This can be forgiven due to vfork(2)'s long preceding threads -- when threads came along the need for a separate stack for each new thread became utterly clear and unavoidable. But blocking is bad because synchronous behavior is bad, especially when it's the only option yet it could have been better. An asynchronous version of vfork() would have to run the child in a new/alternate stack. Let's call it afork(), or avfork(). Now, afork() would have to look a lot like pthread_create(): it has to take a function to call on a new stack, as well as an argument to pass to that function.

An afork() would allow a popen() like API to return very quickly with appropriate pipes for I/O with the child(ren). If anything goes wrong on the child side then the child(ren) will exit and their output pipe (if any) will evince EOF, and/or writes to the child's input will get EPIPE and/or will raise SIGPIPE, at which point the caller of popen() will be able to check for errors.

One might as well borrow the Illumos forkx()/vforkx() flags, and make afork() look like this:

pid_t afork(int (*start_routine)(void *), void *arg);
pid_t aforkx(int flags /* FORK_NOSIGCHLD and/or FORK_WAITPID */, int (*fn)(void *), void *arg);

It turns out that afork() is easy to implement on Linux: it's just a clone(<function>, <stack>, CLONE_VM | CLONE_SETTLS, <argument>) call. (One might want to request that SIGCHLD be sent to the parent when the child dies, but this is decidedly not desirable in a popen() implementation, as otherwise the program might reap it before pclose() can reap it. For more on this go look at Illumos.)

One can also implement something like afork() (minus the Illumos forkx() flags) on POSIX systems by using pthread_create() to start a thread that will block in vfork() while the afork() caller continues its business. Add a taskq to pre-create as many such worker threads as needed, and you'll have a fast afork(). However, an afork() implemented this way won't be able to return a PID unless the threads in the taskq pre-vfork (good idea!), instead it would need a completion callback, something like this:

int emulated_afork(int (*start_routine)(void *), void *arg, void (*cb)(pid_t) /* may be NULL */);

If the threads pre-vfork, then a PID-returning afork() can be implemented, though communicating a task to a pre-vforked thread might be tricky: pthread_cond_wait() might not work in the child, so one would have to use a pipe into which to write a pointer to the dispatched request.

The title also says that clone() is stupid. Linux should have had a thread creation system call -- it would have then saved itself the pain of the first pthread implementation for Linux. clone() has become a swiss army knife -- it has evolved to have zone/jail entering features, but only sort of: Linux doesn't have proper zones/jails, and as new flags are added that old code might wish it had used... one will have to modify and rebuild the clone()-calling world, and that's decidedly not elegant.

I could go on. I could talk about fork-safety. I could discuss all of the functions that are generally, or in specific cases, safe to call in a child of fork(), versus the child of vfork(), versus the child of afork() (if we had one), or a child of a clone() call (but I'd have to consider quite a few flag combinations!). I could go into why 4.4BSD removed vfork() (I'd have to do a bit more digging though). I think this post's length is probably just right, so I'll leave it here.