Skip to content

Instantly share code, notes, and snippets.

@FutureBody
Forked from izabera/pipeline_trick.md
Created January 26, 2024 06:55
Show Gist options
  • Save FutureBody/c7b809fefe4dba4d575b1375c7bb698c to your computer and use it in GitHub Desktop.
Save FutureBody/c7b809fefe4dba4d575b1375c7bb698c to your computer and use it in GitHub Desktop.

Revisions

  1. @izabera izabera revised this gist Jan 25, 2024. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion pipeline_trick.md
    Original file line number Diff line number Diff line change
    @@ -21,7 +21,7 @@ foo | tee fifo1 fifo2 >/dev/null
    This works, but it puts `bar` and `baz` in their own process groups. But that's
    terrible! It fucks up the signals and job control!

    Signals are now only delived to the foreground pipeline. Deadly signals are
    Signals are now only delivered to the foreground pipeline. Deadly signals are
    largely going to be handled correctly, as most well behaved programs terminate
    on EOF, but that's not guaranteed. And you can't suspend/unsuspend the whole
    thing anymore!
  2. @izabera izabera created this gist Jan 25, 2024.
    75 changes: 75 additions & 0 deletions pipeline_trick.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,75 @@
    ## A funky shell thingy that I've never seen before

    So you're in posix sh and you want to do the equivalent of this in bash:

    ```sh
    foo | tee >(bar) >(baz) >/dev/null
    ```
    (Suppose that `bar` and `baz` don't produce output. Add redirections where
    needed if that's not the case.)

    The "standard" way to do this would be to create the named pipes manually, and
    start `bar` and `baz` in background:

    ```sh
    mkfifo fifo1 fifo2
    bar < fifo1 &
    baz < fifo2 &
    foo | tee fifo1 fifo2 >/dev/null
    ```

    This works, but it puts `bar` and `baz` in their own process groups. But that's
    terrible! It fucks up the signals and job control!

    Signals are now only delived to the foreground pipeline. Deadly signals are
    largely going to be handled correctly, as most well behaved programs terminate
    on EOF, but that's not guaranteed. And you can't suspend/unsuspend the whole
    thing anymore!

    You can work around some of the issues with the signals, for instance by adding
    an exit trap and using it to forward any deadly signal to the bg processes:

    ```sh
    trap 'kill "$pid1" "$pid2"' EXIT

    mkfifo fifo1 fifo2
    bar < fifo1 & pid1=$!
    baz < fifo2 & pid2=$!
    foo | tee fifo1 fifo2 >/dev/null
    ```

    Similarly, you can just trap any signals and forward them manually.

    That's cool, but now the signals come from the shell (you can see it in
    `siginfo_t`). You're also relying on the shell to stay alive long enough to
    deliver them, so if the signal was `sigkill` the shell couldn't forward it.

    **Instead, you can create the fifos manually, and then... use a pipeline!**

    ```sh
    mkfifo fifo1 fifo2
    foo | tee fifo1 fifo2 >/dev/null |
    bar < fifo1 |
    baz < fifo2
    ```

    Now all your processes are in the same process group, so any signals sent to
    the group get delivered to all the processes.

    You can even handle cleaning up the fifos this way:

    ```sh
    { rm fifo1; bar; } < fifo1
    ```

    This looks racy, because `tee` needs to open the fifos before `rm` removes
    them, but it's actually entirely safe because the shell is executing the
    redirection first, and opening the fifo in read mode blocks until there's at
    least one writer.

    You can now safely `^C` and forget.

    ----

    Please let me know if this thing has a name. I don't know how to Google it,
    but surely someone came up with it like 40 years ago.