Skip to content

Instantly share code, notes, and snippets.

@NormanEdance

NormanEdance/subprocess_filter.py

Forked from DGrady/subprocess_filter.py
Created February 12, 2018 05:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save NormanEdance/8b51e45b8abe449c7cee5bd94d1d03cd to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save NormanEdance/8b51e45b8abe449c7cee5bd94d1d03cd to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @DGrady DGrady revised this gist Jul 26, 2017. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion subprocess_filter.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,9 @@
    """
    Problem: provide two-way communication with a subprocess in Python.
    See also: https://kevinmccarthy.org/2016/07/25/streaming-subprocess-stdin-and-stdout-with-asyncio-in-python/
    See also:
    - https://kevinmccarthy.org/2016/07/25/streaming-subprocess-stdin-and-stdout-with-asyncio-in-python/
    - http://eli.thegreenplace.net/2017/interacting-with-a-long-running-child-process-in-python/
    """

    import asyncio
  2. @DGrady DGrady revised this gist Jun 27, 2017. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions subprocess_filter.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,7 @@
    """
    Problem: provide two-way communication with a subprocess in Python.
    See also: https://kevinmccarthy.org/2016/07/25/streaming-subprocess-stdin-and-stdout-with-asyncio-in-python/
    """

    import asyncio
  3. @DGrady DGrady revised this gist Jun 27, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion subprocess_filter.py
    View file
    Original file line number Diff line number Diff line change
    @@ -28,7 +28,7 @@ async def dequeue(stream: asyncio.StreamReader, callback: T.Callable[[bytes], No
    async def main():

    proc = await asyncio.subprocess.create_subprocess_exec(
    'sed', 's/o/o\n/g',
    'sed', r's/o/o\n/g',
    stdin=asyncio.subprocess.PIPE,
    stdout=asyncio.subprocess.PIPE,
    )
  4. @DGrady DGrady revised this gist Jun 27, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion subprocess_filter.py
    View file
    Original file line number Diff line number Diff line change
    @@ -28,7 +28,7 @@ async def dequeue(stream: asyncio.StreamReader, callback: T.Callable[[bytes], No
    async def main():

    proc = await asyncio.subprocess.create_subprocess_exec(
    ['sed', 's/o/o\n/g'],
    'sed', 's/o/o\n/g',
    stdin=asyncio.subprocess.PIPE,
    stdout=asyncio.subprocess.PIPE,
    )
  5. @DGrady DGrady created this gist Jun 27, 2017.
    48 changes: 48 additions & 0 deletions subprocess_filter.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,48 @@
    """
    Problem: provide two-way communication with a subprocess in Python.
    """

    import asyncio
    import sys
    import typing as T

    async def enqueue(values: T.Iterable[bytes], stream: asyncio.StreamWriter):

    for line in values:
    stream.write(line)
    # Yield to the asyncio loop
    await stream.drain()

    # Once we've exhausted values, we need to close the async stream to signal to
    # the subprocess that it can exit
    stream.close()

    async def dequeue(stream: asyncio.StreamReader, callback: T.Callable[[bytes], None]):

    while True:
    line = await stream.readline()
    if not line:
    break
    callback(line)

    async def main():

    proc = await asyncio.subprocess.create_subprocess_exec(
    ['sed', 's/o/o\n/g'],
    stdin=asyncio.subprocess.PIPE,
    stdout=asyncio.subprocess.PIPE,
    )

    await asyncio.wait([
    enqueue(sys.stdin.buffer, proc.stdin),
    dequeue(proc.stdout, sys.stdout.buffer.write),
    ])

    # I'm not completely sure the call to `communicate` is necessary
    (stdout_data, stderr_data) = await proc.communicate()
    await proc.wait()

    if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    rc = loop.run_until_complete(main())
    loop.close()