Skip to content

Instantly share code, notes, and snippets.

@caike
Last active October 17, 2022 00:27
Show Gist options
  • Select an option

  • Save caike/ebccc95bd46f5fa1404d to your computer and use it in GitHub Desktop.

Select an option

Save caike/ebccc95bd46f5fa1404d to your computer and use it in GitHub Desktop.

Revisions

  1. caike revised this gist Jul 17, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion streams-2-back-pressure.js
    Original file line number Diff line number Diff line change
    @@ -7,6 +7,6 @@ readStream.on('readable', function() {
    }
    })

    readStream.on('drain', function() {
    writeStream.on('drain', function() {
    canWrite = true;
    });
  2. caike revised this gist Jul 17, 2014. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions streams-2-back-pressure.js
    Original file line number Diff line number Diff line change
    @@ -1,10 +1,10 @@
    var canWrite = true;

    readStream.on('readable', function() {
    while(canWrite && (null !== (chunk = readStream.read()))){
    var bufferGood = writeStream.write(chunk)
    if (!bufferGood) canWrite = false;
    }
    while(canWrite && (null !== (chunk = readStream.read()))){
    var bufferGood = writeStream.write(chunk)
    if (!bufferGood) canWrite = false;
    }
    })

    readStream.on('drain', function() {
  3. caike revised this gist Jul 17, 2014. 2 changed files with 13 additions and 5 deletions.
    6 changes: 1 addition & 5 deletions about.md
    Original file line number Diff line number Diff line change
    @@ -6,13 +6,9 @@
    * Instead of `data` events spewing, call `read()` to pull data from source.
    * When there isn't any data to consume, then `read()` will return **undefined**.
    * Adding a `data` event listener will switch the Readable stream into "old mode", where data is emitted as soon as it is available rather than waiting for you to call `read()` to consume it. This requires you to handle backpressure problems manually.
    * The `pipe` method helps write less code and handles backpressure.
    * The `pipe` method helps write less code and handles back-pressure.
    * If you add an `end` listener and don't ever `read()` or `pipe()`, it'll never emit `end`.

    ## Question

    Is there still room for back-pressure issues in streams2 ?

    ## Resources

    [Isaac Z. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
    12 changes: 12 additions & 0 deletions streams-2-back-pressure.js
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,12 @@
    var canWrite = true;

    readStream.on('readable', function() {
    while(canWrite && (null !== (chunk = readStream.read()))){
    var bufferGood = writeStream.write(chunk)
    if (!bufferGood) canWrite = false;
    }
    })

    readStream.on('drain', function() {
    canWrite = true;
    });
  4. caike revised this gist Jul 9, 2014. 1 changed file with 1 addition and 14 deletions.
    15 changes: 1 addition & 14 deletions about.md
    Original file line number Diff line number Diff line change
    @@ -9,20 +9,7 @@
    * The `pipe` method helps write less code and handles backpressure.
    * If you add an `end` listener and don't ever `read()` or `pipe()`, it'll never emit `end`.

    From <http://nodejs.org/api/stream.html#stream_class_stream_readable>:

    The Readable stream interface is the abstraction for a source of data that you are reading from. In other words, data comes out of a Readable stream.

    A Readable stream will not start emitting data until you indicate that you are ready to receive it.

    Readable streams have two "modes":

    * flowing mode
    * non-flowing mode.

    When in flowing mode (old mode), data is read from the underlying system and provided to your program as fast as possible. In non-flowing mode (a.k.a. streams2), you must explicitly call `stream.read()` to get chunks of data out.

    ## Questions
    ## Question

    Is there still room for back-pressure issues in streams2 ?

  5. caike renamed this gist Jul 9, 2014. 1 changed file with 0 additions and 0 deletions.
    File renamed without changes.
  6. caike revised this gist Jul 9, 2014. 1 changed file with 27 additions and 0 deletions.
    27 changes: 27 additions & 0 deletions streams-2-file-upload.js
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,27 @@
    var http = require('http');
    var fs = require('fs');

    http.createServer(function(req, res) {

    var newFile = fs.createWriteStream('destination.jpg');
    var fileBytes = req.headers['content-length'];
    var uploadedBytes = 0;

    req.on('readable', function() {
    var chunk = null;
    while(null !== (chunk = req.read())){
    uploadedBytes += chunk.length;
    var progress = (uploadedBytes / fileBytes) * 100;
    res.write("Progress: " + parseInt(progress, 10) + "%\n");
    }
    });

    req.on('end', function() {
    res.end('uploaded!');
    });

    req.pipe(newFile);

    }).listen(8080);

    // run with `curl --upload-file <your-file.jpg> http://localhost:8080`
  7. caike revised this gist Jul 9, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion about.md
    Original file line number Diff line number Diff line change
    @@ -28,7 +28,7 @@ Is there still room for back-pressure issues in streams2 ?

    ## Resources

    [Izaac S. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
    [Isaac Z. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
    [Tim Caswell](http://howtonode.org/streams-explained)
    [Stream Compatibility](http://nodejs.org/api/stream.html#stream_compatibility_with_older_node_versions)
    [Getting Started With Streams2](http://maxantoni.de/blog/2013/06/getting-started-with-streams-2.html)
  8. caike revised this gist Jul 9, 2014. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions about.md
    Original file line number Diff line number Diff line change
    @@ -28,7 +28,7 @@ Is there still room for back-pressure issues in streams2 ?

    ## Resources

    [Izaac S. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
    [Tim Caswell](http://howtonode.org/streams-explained)
    [Stream Compatibility](http://nodejs.org/api/stream.html#stream_compatibility_with_older_node_versions)
    [Getting Started With Streams2](http://maxantoni.de/blog/2013/06/getting-started-with-streams-2.html)
    [Izaac S. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
    [Tim Caswell](http://howtonode.org/streams-explained)
    [Stream Compatibility](http://nodejs.org/api/stream.html#stream_compatibility_with_older_node_versions)
    [Getting Started With Streams2](http://maxantoni.de/blog/2013/06/getting-started-with-streams-2.html)
  9. caike revised this gist Jul 9, 2014. 1 changed file with 12 additions and 5 deletions.
    17 changes: 12 additions & 5 deletions about.md
    Original file line number Diff line number Diff line change
    @@ -4,11 +4,10 @@
    * Introduced in node v0.10.
    * "suck" streams instead of "spew" streams.
    * Instead of `data` events spewing, call `read()` to pull data from source.
    * When there isn't any data to consume, then `read()` will return null.
    * In order to tweak back pressure settings, there is a `highWaterMark` and `lowWaterMark` property. These are limits that lets us configure how different streams exert backpressure.
    * All Writables emit `finish` when ended and flushed.
    * New streams can switch into old-mode, where they spew data. Adding a `data` event listener will switch the Readable stream into "old mode", where data is emitted as soon as it is available rather than waiting for you to call `read()` to consume it.
    * If you add an `end` listener, but don't add a `data` listener, and don't ever `read()` or `pipe()`, it'll never emit `end`.
    * When there isn't any data to consume, then `read()` will return **undefined**.
    * Adding a `data` event listener will switch the Readable stream into "old mode", where data is emitted as soon as it is available rather than waiting for you to call `read()` to consume it. This requires you to handle backpressure problems manually.
    * The `pipe` method helps write less code and handles backpressure.
    * If you add an `end` listener and don't ever `read()` or `pipe()`, it'll never emit `end`.

    From <http://nodejs.org/api/stream.html#stream_class_stream_readable>:

    @@ -23,5 +22,13 @@ Readable streams have two "modes":

    When in flowing mode (old mode), data is read from the underlying system and provided to your program as fast as possible. In non-flowing mode (a.k.a. streams2), you must explicitly call `stream.read()` to get chunks of data out.

    ## Questions

    Is there still room for back-pressure issues in streams2 ?

    ## Resources

    [Izaac S. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
    [Tim Caswell](http://howtonode.org/streams-explained)
    [Stream Compatibility](http://nodejs.org/api/stream.html#stream_compatibility_with_older_node_versions)
    [Getting Started With Streams2](http://maxantoni.de/blog/2013/06/getting-started-with-streams-2.html)
  10. caike renamed this gist Jul 8, 2014. 1 changed file with 0 additions and 0 deletions.
    File renamed without changes.
  11. caike revised this gist Jul 8, 2014. 1 changed file with 27 additions and 0 deletions.
    27 changes: 27 additions & 0 deletions about.
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,27 @@
    # NodeJS Streams

    * Current implementation is known as **streams2**.
    * Introduced in node v0.10.
    * "suck" streams instead of "spew" streams.
    * Instead of `data` events spewing, call `read()` to pull data from source.
    * When there isn't any data to consume, then `read()` will return null.
    * In order to tweak back pressure settings, there is a `highWaterMark` and `lowWaterMark` property. These are limits that lets us configure how different streams exert backpressure.
    * All Writables emit `finish` when ended and flushed.
    * New streams can switch into old-mode, where they spew data. Adding a `data` event listener will switch the Readable stream into "old mode", where data is emitted as soon as it is available rather than waiting for you to call `read()` to consume it.
    * If you add an `end` listener, but don't add a `data` listener, and don't ever `read()` or `pipe()`, it'll never emit `end`.

    From <http://nodejs.org/api/stream.html#stream_class_stream_readable>:

    The Readable stream interface is the abstraction for a source of data that you are reading from. In other words, data comes out of a Readable stream.

    A Readable stream will not start emitting data until you indicate that you are ready to receive it.

    Readable streams have two "modes":

    * flowing mode
    * non-flowing mode.

    When in flowing mode (old mode), data is read from the underlying system and provided to your program as fast as possible. In non-flowing mode (a.k.a. streams2), you must explicitly call `stream.read()` to get chunks of data out.

    ## Resources
    [Izaac S. Presentation](https://dl.dropboxusercontent.com/u/3685/presentations/streams2/streams2-ko.pdf)
  12. caike created this gist Jul 8, 2014.
    21 changes: 21 additions & 0 deletions streams-1.js
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,21 @@
    // streams1
    // works in node v0.6.21
    // works in node v0.10.29 (latest stable)
    // Tested with `curl -d 'akshdakjhdakjhsdkajhdjkahdsjka' http://localhost:8080`
    var http = require('http');

    http.createServer(function(req, res) {

    console.log('Got Request');

    res.writeHead(200);

    req.on('data', function(chunk){
    console.log('got %d bytes of data', chunk.length);
    });

    req.on('end', function() {
    res.end('over');
    });

    }).listen(8080);
    23 changes: 23 additions & 0 deletions streams-2.js
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,23 @@
    // streams2
    // works in v0.10.29
    // does not work in v.0.6.21
    // Example taken from http://nodejs.org/api/stream.html#stream_class_stream_readable
    // Tested with `curl -d 'akshdakjhdakjhsdkajhdjkahdsjka' http://localhost:8080`
    var http = require('http');

    http.createServer(function(req, res) {

    console.log('Got Request');

    req.on('readable', function() {
    var chunk = null;
    while (null !== (chunk = req.read())) {
    console.log('got %d bytes of data', chunk.length);
    }
    });

    req.on('end', function() {
    res.end('Done');
    });

    }).listen(8080);