Skip to content

Instantly share code, notes, and snippets.

@waylan
Created April 10, 2012 19:12
Show Gist options
  • Select an option

  • Save waylan/2353749 to your computer and use it in GitHub Desktop.

Select an option

Save waylan/2353749 to your computer and use it in GitHub Desktop.

Revisions

  1. Waylan Limberg revised this gist Apr 11, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -71,7 +71,7 @@ for x in xrange(100):
    try:
    p.stdin.write(line)
    except IOError as e:
    if e.errno = errno.EPIPE or e.errno = errno.EINVAL:
    if e.errno == errno.EPIPE or e.errno == errno.EINVAL:
    # Stop loop on "Invalid pipe" or "Invalid argument".
    # No sense in continuing with broken pipe.
    break
  2. Waylan Limberg revised this gist Apr 11, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -71,7 +71,7 @@ for x in xrange(100):
    try:
    p.stdin.write(line)
    except IOError as e:
    if e.errno = errno.EPIPE and e.errno = errno.EINVAL:
    if e.errno = errno.EPIPE or e.errno = errno.EINVAL:
    # Stop loop on "Invalid pipe" or "Invalid argument".
    # No sense in continuing with broken pipe.
    break
  3. Waylan Limberg revised this gist Apr 11, 2012. 1 changed file with 7 additions and 4 deletions.
    11 changes: 7 additions & 4 deletions subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -71,13 +71,16 @@ for x in xrange(100):
    try:
    p.stdin.write(line)
    except IOError as e:
    # Silently fail on "Invalid pipe" or "Invalid argument".
    # Raise any other error.
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
    if e.errno = errno.EPIPE and e.errno = errno.EINVAL:
    # Stop loop on "Invalid pipe" or "Invalid argument".
    # No sense in continuing with broken pipe.
    break
    else:
    # Raise any other error.
    raise

    p.stdin.close()
    p.wait()

    print 'All done!'
    print 'All done!' # This should always be printed below any output written to less.
    ```
  4. Waylan Limberg revised this gist Apr 11, 2012. 1 changed file with 25 additions and 1 deletion.
    26 changes: 25 additions & 1 deletion subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -56,4 +56,28 @@ except IOError as e:
    raise
    ```

    Note that [`errno.EPIPE`](http://docs.python.org/library/errno.html?highlight=errno#errno.EPIPE) is "Broken pipe" and [`errno.EINVAL`](http://docs.python.org/library/errno.html?highlight=errno#errno.EINVAL) is "Invalid argument".
    Note that [`errno.EPIPE`](http://docs.python.org/library/errno.html?highlight=errno#errno.EPIPE) is "Broken pipe" and [`errno.EINVAL`](http://docs.python.org/library/errno.html?highlight=errno#errno.EINVAL) is "Invalid argument".

    So the final code looks like this:

    ``` python

    from subprocess import Popen, PIPE
    import errno

    p = Popen('less', stdin=PIPE)
    for x in xrange(100):
    line = 'Line number %d.\n' % x
    try:
    p.stdin.write(line)
    except IOError as e:
    # Silently fail on "Invalid pipe" or "Invalid argument".
    # Raise any other error.
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
    raise

    p.stdin.close()
    p.wait()

    print 'All done!'
    ```
  5. Waylan Limberg revised this gist Apr 11, 2012. 1 changed file with 27 additions and 1 deletion.
    28 changes: 27 additions & 1 deletion subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -30,4 +30,30 @@ for x in xrange(100):
    p.communicate('\n'.join(out))
    ```

    This works __great__! We only have one call to `communicate` and that calls `wait` properly. Unfortunately, we have to create the entire output in memory before writing any of it out. At least it works.
    This works. We only have one call to `communicate` and that calls `wait` properly. Unfortunately, we have to create the entire output in memory before writing any of it out. We can do better:

    ```python
    from subprocess import Popen, PIPE

    p = Popen('less', stdin=PIPE)
    for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
    p.stdin.close()
    p.wait()
    ```

    The key it to close stdin (flush and send EOF) before calling `wait`. This is actually what `communicate` does internally minus all the stdout and stderr stuff I don't need. If I wanted to force the buffer to remain empty, I suppose I could do `p.stdin.flush()` on each loop, but why? Note that there probably should be some error checking on write (like there is in the source of `communicate`. Perhaps something like:

    ``` python
    import errno

    ...

    try:
    p.stdin.write(input)
    except IOError as e:
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
    raise
    ```

    Note that [`errno.EPIPE`](http://docs.python.org/library/errno.html?highlight=errno#errno.EPIPE) is "Broken pipe" and [`errno.EINVAL`](http://docs.python.org/library/errno.html?highlight=errno#errno.EINVAL) is "Invalid argument".
  6. Waylan Limberg revised this gist Apr 10, 2012. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -26,7 +26,7 @@ from subprocess import Popen, PIPE
    out = []
    p = Popen('less', stdin=PIPE)
    for x in xrange(100):
    out.append('Line number %d.\n' % x)
    out.append('Line number %d.' % x)
    p.communicate('\n'.join(out))
    ```

  7. Waylan Limberg created this gist Apr 10, 2012.
    33 changes: 33 additions & 0 deletions subprocess_pipe.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,33 @@
    Here's a few things I tried to write output to a python subprocess pipe.

    ```python
    from subprocess import Popen, PIPE

    p = Popen('less', stdin=PIPE)
    for x in xrange(100):
    p.communicate('Line number %d.\n' % x)
    ```

    This seemed like the most obvious solution but it fails miserably. It seems that the first call to `communicate` also closes the pipe and the second loop raises an exception.

    ```python
    from subprocess import Popen, PIPE

    p = Popen('less', stdin=PIPE)
    for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
    ```

    This is expressly stated to be a bad idea in the docs, but it works - sort of. I get some weird behavior. There's no call to `p.wait()` (which `communicate` does by default) so anything after the loop runs before the subproccess (`less` in this case) is closed. Adding a call to `wait` after the loop causes even weirder behavior.

    ```python
    from subprocess import Popen, PIPE

    out = []
    p = Popen('less', stdin=PIPE)
    for x in xrange(100):
    out.append('Line number %d.\n' % x)
    p.communicate('\n'.join(out))
    ```

    This works __great__! We only have one call to `communicate` and that calls `wait` properly. Unfortunately, we have to create the entire output in memory before writing any of it out. At least it works.