spion · November 2, 2019 12:27 · Nov 2, 2019 · Aug 19, 2018 · Sep 25, 2016 · May 5, 2016
diff --git a/01-fractal-weird-design.md b/01-fractal-weird-design.md
@@ -5,16 +5,16 @@
 This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If [promise-streams](https://github.com/spion/promise-streams) is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
 
 1. `enc` parameter - Why is encoding always passed together with the data? It should be a separate concern. It doesn't even make sense in objectMode
-2. eventemitter base - There is no reason to use an uncomposable grab-bag of stringy events as a base
-3. relying on nextTick etc for execution order - This is very unreliable and causes all sorts of unpredictable rules for implementers which are not documented anywhere.
-4. no error propagation
+2. eventemitter base - This encourages a lot of random events to be "attached" by other authors which doesn't work. Best to have an uniform (typed) interface so that everyone knows what to expect.
+3. relying on nextTick etc for execution order - This is very unreliable and causes all sorts of unpredictable rules for implementers which are not documented anywhere. When you attach listeners determines what will happen.
+4. no error propagation - we need the equivalent of promise error propagation
 5. attaching an error event listener changes behaviour of producer (producer now continues to emit events after error). This effectively couples the consumer with the producer and with all other consumers. The producer should not have to be aware of the listeners. More importantly, the consumer should not affect the consumption of other listeners: if they add an error listener they fundamentally change the way the stream behaves for everyone.
 6. buffering included - Perhaps a buffer should be a separate stream?
 7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding cocerns are implemented as separate decoder streams then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
-8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?
+8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled? Is this defined?
 9. Errors in the target stream cause it to unpipe from the source. The problem is that the [target stream will not end](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if multicast is extracted to be a separate stream.
-10. Combining readable and writable. I'm not sure writable streams should be a thing at all, they're a different API. But I'm definitely sure that combining the readable and writable API into a single object sucks. The worst thing it does is that it mixes both read and write error events into a single place which virtually guarantees that you can't forward errors.
-11. Write is just weird. It returns false, and then you have to listen to the drain event. Its all due to the silly "everything-is-an-event-on-eventemitter" approach. You can't even check if a stream needs to have a drain event!
+10. Combining readable and writable. I'm not sure writable streams should be a thing at all, they're a different API. But I'm definitely sure that combining the readable and writable API into a single object is bad. It mixes both read and write error events into a single place which virtually guarantees that you can't forward errors.
+11. Write is just weird. It returns false, and then you have to listen to the drain event. Its all due to the "everything-is-an-event-on-eventemitter" approach. You can't even check if a stream needs to have a drain event!
 
 
 Stuff that are good:

diff --git a/01-fractal-weird-design.md b/01-fractal-weird-design.md
@@ -14,6 +14,8 @@ This is a list of issues and confusions I've encountered with node streams, and
 8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?
 9. Errors in the target stream cause it to unpipe from the source. The problem is that the [target stream will not end](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if multicast is extracted to be a separate stream.
 10. Combining readable and writable. I'm not sure writable streams should be a thing at all, they're a different API. But I'm definitely sure that combining the readable and writable API into a single object sucks. The worst thing it does is that it mixes both read and write error events into a single place which virtually guarantees that you can't forward errors.
+11. Write is just weird. It returns false, and then you have to listen to the drain event. Its all due to the silly "everything-is-an-event-on-eventemitter" approach. You can't even check if a stream needs to have a drain event!
+
 
 Stuff that are good:
 

diff --git a/01-fractal-weird-design.md b/01-fractal-weird-design.md
@@ -8,7 +8,7 @@ This is a list of issues and confusions I've encountered with node streams, and
 2. eventemitter base - There is no reason to use an uncomposable grab-bag of stringy events as a base
 3. relying on nextTick etc for execution order - This is very unreliable and causes all sorts of unpredictable rules for implementers which are not documented anywhere.
 4. no error propagation
-5. attaching an error event listener changes behaviour of producer. This effectively couples the consumer with the producer and with all other consumers. The producer should not have to be aware of the listeners. More importantly, the consumer should not affect the consumption of other listeners: if they add an error listener they fundamentally change the way the stream behaves for everyone.
+5. attaching an error event listener changes behaviour of producer (producer now continues to emit events after error). This effectively couples the consumer with the producer and with all other consumers. The producer should not have to be aware of the listeners. More importantly, the consumer should not affect the consumption of other listeners: if they add an error listener they fundamentally change the way the stream behaves for everyone.
 6. buffering included - Perhaps a buffer should be a separate stream?
 7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding cocerns are implemented as separate decoder streams then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
 8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?

diff --git a/fractal-weird-design.md → 01-fractal-weird-design.md b/fractal-weird-design.md → 01-fractal-weird-design.md
@@ -60,13 +60,3 @@ The complex rules will only remain in MultiCast:
 * It will propagate errors forwards to all forks
 * It will only propagate unpipes (unsubscribes) to the source if all forks are unpiped (unsubscribed) 
 
-#### Error semantics
-
-Proposed error semantics:
-
-* Errors will propagate through piped streams until an error handling stream is encountered
-* An error handling stream can be created by using mapError, flatMapError or passing another parameter to ps.through which contains an error handler.
-* A stream is "consumed" if its converted to a promise by awaiting its end event, or if its otherwise subscribed to.
-* If an error propagates to a consumed stream, it will 
-  * reject that stream's promise
-  * cancel all subscriptions backwards, such that if a subscription count drops to zero that stream also unsubscribes backwards.
diff --git a/02-error-semantics.md b/02-error-semantics.md
@@ -0,0 +1,8 @@
+# Initial proposed error semantics
+
+* Errors will propagate through piped streams until an error handling stream is encountered
+* An error handling stream can be created by using mapError, flatMapError or passing another parameter to ps.through which contains an error handler.
+* A stream is "consumed" if its converted to a promise by awaiting its end event, or if its otherwise subscribed to.
+* If an error propagates to a consumed stream, it will 
+  * reject that stream's promise
+  * cancel all subscriptions backwards, such that if a subscription count drops to zero that stream also unsubscribes backwards.
diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -62,4 +62,11 @@ The complex rules will only remain in MultiCast:
 
 #### Error semantics
 
-TODO: error propagation, cancellation and finalization.
+Proposed error semantics:
+
+* Errors will propagate through piped streams until an error handling stream is encountered
+* An error handling stream can be created by using mapError, flatMapError or passing another parameter to ps.through which contains an error handler.
+* A stream is "consumed" if its converted to a promise by awaiting its end event, or if its otherwise subscribed to.
+* If an error propagates to a consumed stream, it will 
+  * reject that stream's promise
+  * cancel all subscriptions backwards, such that if a subscription count drops to zero that stream also unsubscribes backwards.
diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -58,4 +58,8 @@ The complex rules will only remain in MultiCast:
 
 * It will only pull the next packet from the source when all the forks are done pulling the current packet
 * It will propagate errors forwards to all forks
-* It will only propagate unpipes (unsubscribes) to the source if all forks are unpiped (unsubscribed) 
+* It will only propagate unpipes (unsubscribes) to the source if all forks are unpiped (unsubscribed) 
+
+#### Error semantics
+
+TODO: error propagation, cancellation and finalization.
diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -13,6 +13,7 @@ This is a list of issues and confusions I've encountered with node streams, and
 7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding cocerns are implemented as separate decoder streams then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
 8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?
 9. Errors in the target stream cause it to unpipe from the source. The problem is that the [target stream will not end](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if multicast is extracted to be a separate stream.
+10. Combining readable and writable. I'm not sure writable streams should be a thing at all, they're a different API. But I'm definitely sure that combining the readable and writable API into a single object sucks. The worst thing it does is that it mixes both read and write error events into a single place which virtually guarantees that you can't forward errors.
 
 Stuff that are good:
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -36,7 +36,7 @@ A stream buffer (decoder) will pull data until it fills up to highWatermark. Dep
 
 Then the buffer streams will start pulling again until they're full and so on.
 
-Extracting the buffers will eliminate all inconsistencies pertaining to encoding and object mode and will decouple them from the main stream logic.
+Extracting the buffers will result with all streams working in "objectMode" and eliminate all inconsistencies pertaining to encoding and buffering, which will be decoupled them from the main stream logic.
 
 #### Multicast stream
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -28,7 +28,7 @@ The resulting streams will be pull-only, single producer, single-consumer.
 
 #### Stream buffers (decoders)
 
-Stream buffers (decoders) will pull data until they fill up to highWatermark. Depending on the type of buffer it will then do different things when pulled from:
+A stream buffer (decoder) will pull data until it fills up to highWatermark. Depending on the type of buffer it will then do different things when pulled from:
 
 * `ByteBuffer` will return the entire combined buffer of the data thus far
 * `UTF8Decoder` will return as much data as possible that is a valid UTF-8 string, will keep the rest in the buffer

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -28,7 +28,7 @@ The resulting streams will be pull-only, single producer, single-consumer.
 
 #### Stream buffers (decoders)
 
-Stream buffers (decoders) will pull data until they fills up to highWatermark. Depending on the type of buffer it will then do different things when pulled from:
+Stream buffers (decoders) will pull data until they fill up to highWatermark. Depending on the type of buffer it will then do different things when pulled from:
 
 * `ByteBuffer` will return the entire combined buffer of the data thus far
 * `UTF8Decoder` will return as much data as possible that is a valid UTF-8 string, will keep the rest in the buffer

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -10,7 +10,7 @@ This is a list of issues and confusions I've encountered with node streams, and
 4. no error propagation
 5. attaching an error event listener changes behaviour of producer. This effectively couples the consumer with the producer and with all other consumers. The producer should not have to be aware of the listeners. More importantly, the consumer should not affect the consumption of other listeners: if they add an error listener they fundamentally change the way the stream behaves for everyone.
 6. buffering included - Perhaps a buffer should be a separate stream?
-7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding is left out as a concern of the stream base then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
+7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding cocerns are implemented as separate decoder streams then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
 8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?
 9. Errors in the target stream cause it to unpipe from the source. The problem is that the [target stream will not end](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if multicast is extracted to be a separate stream.
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -55,6 +55,6 @@ Extracting multicast will enable proper error propagation for regular transform
 
 The complex rules will only remain in MultiCast:
 
-* It will only pull the next packet when from the source when all the forks are done pulling the current packet
+* It will only pull the next packet from the source when all the forks are done pulling the current packet
 * It will propagate errors forwards to all forks
 * It will only propagate unpipes (unsubscribes) to the source if all forks are unpiped (unsubscribed) 
diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -12,7 +12,7 @@ This is a list of issues and confusions I've encountered with node streams, and
 6. buffering included - Perhaps a buffer should be a separate stream?
 7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding is left out as a concern of the stream base then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
 8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?
-9. Errors in the target stream cause it to unpipe from the source. The problem is that the target stream will not end. they [don't end the (transform) stream](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if the multicast
+9. Errors in the target stream cause it to unpipe from the source. The problem is that the [target stream will not end](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if multicast is extracted to be a separate stream.
 
 Stuff that are good:
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -42,11 +42,12 @@ Extracting the buffers will eliminate all inconsistencies pertaining to encoding
 
 To  enable multicast piping there will be a separate writable stream called multicast stream. It will be capable of generating multiple Readable streams. Example API:
 
+```js
 var multi = readable.pipe(new Multicast());
 
 multi.fork().pipe(target1);
 multi.fork().pipe(target2);
-
+```
 
 Extracting multicast will enable proper error propagation for regular transform streams. Since pipe will only be possible between two streams, the rules can be vastly simplified:
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -1,6 +1,6 @@
 # Node streams - a fractal of weird design
 
-*and a potential refactor that could fix that*
+> *and a potential refactor that could fix that*
 
 This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If [promise-streams](https://github.com/spion/promise-streams) is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -1,4 +1,7 @@
 # Node streams - a fractal of weird design
+
+*and a potential refactor that could fix that*
+
 This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If [promise-streams](https://github.com/spion/promise-streams) is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
 
 1. `enc` parameter - Why is encoding always passed together with the data? It should be a separate concern. It doesn't even make sense in objectMode

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -20,6 +20,8 @@ Stuff that are good:
 
 The solution here would be to extract the buffering/encoding logic out into separate transform streams and extract multicast into a separate MulticastStream. 
 
+The resulting streams will be pull-only, single producer, single-consumer.
+
 
 #### Stream buffers (decoders)
 

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -1,5 +1,5 @@
 # Node streams - a fractal of weird design
-This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If promise-streams is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
+This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If [promise-streams](https://github.com/spion/promise-streams) is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
 
 1. `enc` parameter - Why is encoding always passed together with the data? It should be a separate concern. It doesn't even make sense in objectMode
 2. eventemitter base - There is no reason to use an uncomposable grab-bag of stringy events as a base

diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -1,5 +1,4 @@
 # Node streams - a fractal of weird design
-
 This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If promise-streams is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
 
 1. `enc` parameter - Why is encoding always passed together with the data? It should be a separate concern. It doesn't even make sense in objectMode
@@ -38,12 +37,10 @@ Extracting the buffers will eliminate all inconsistencies pertaining to encoding
 
 To  enable multicast piping there will be a separate writable stream called multicast stream. It will be capable of generating multiple Readable streams. Example API:
 
-```
 var multi = readable.pipe(new Multicast());
 
 multi.fork().pipe(target1);
 multi.fork().pipe(target2);
-```
 
 
 Extracting multicast will enable proper error propagation for regular transform streams. Since pipe will only be possible between two streams, the rules can be vastly simplified:
@@ -52,7 +49,6 @@ Extracting multicast will enable proper error propagation for regular transform
 
 The complex rules will only remain in MultiCast:
 
+* It will only pull the next packet when from the source when all the forks are done pulling the current packet
 * It will propagate errors forwards to all forks
-* It will only propagate cancelations to the source if all forks are unpiped (unsubscribed)
-
-No error propagation or cancellation will happen through Duplex streams. A duplex stream will actually not exist: it will consist of a completely separate readable and writable streams.
+* It will only propagate unpipes (unsubscribes) to the source if all forks are unpiped (unsubscribed) 
diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -55,4 +55,4 @@ The complex rules will only remain in MultiCast:
 * It will propagate errors forwards to all forks
 * It will only propagate cancelations to the source if all forks are unpiped (unsubscribed)
 
-No error propagation or cancellation will happen through Duplex streams.
+No error propagation or cancellation will happen through Duplex streams. A duplex stream will actually not exist: it will consist of a completely separate readable and writable streams.
diff --git a/fractal-weird-design.md b/fractal-weird-design.md
@@ -1,3 +1,5 @@
+# Node streams - a fractal of weird design
+
 This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If promise-streams is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
 
 1. `enc` parameter - Why is encoding always passed together with the data? It should be a separate concern. It doesn't even make sense in objectMode

diff --git a/gistfile1.txt → fractal-weird-design.md b/gistfile1.txt → fractal-weird-design.md
diff --git a/gistfile1.txt b/gistfile1.txt
@@ -0,0 +1,56 @@
+This is a list of issues and confusions I've encountered with node streams, and things I wish were designed and implemented differently. If promise-streams is ever going to change to a design that doesn't build on top of node streams, this would be a list of mistakes to avoid
+
+1. `enc` parameter - Why is encoding always passed together with the data? It should be a separate concern. It doesn't even make sense in objectMode
+2. eventemitter base - There is no reason to use an uncomposable grab-bag of stringy events as a base
+3. relying on nextTick etc for execution order - This is very unreliable and causes all sorts of unpredictable rules for implementers which are not documented anywhere.
+4. no error propagation
+5. attaching an error event listener changes behaviour of producer. This effectively couples the consumer with the producer and with all other consumers. The producer should not have to be aware of the listeners. More importantly, the consumer should not affect the consumption of other listeners: if they add an error listener they fundamentally change the way the stream behaves for everyone.
+6. buffering included - Perhaps a buffer should be a separate stream?
+7. objectMode - There should be no difference between object and non-object mode. There are 2 reasons for objectMode: buffering and encoding. If the buffer is a separate stream and encoding is left out as a concern of the stream base then there will be no difference between objectMode and non-objectMode. Once you move out the buffer, you can do this because you don't have to handle encoding unless you're joining multiple data packets into one.
+8. semantics of errors - What is an error? Should it stop the stream? Should it be possible to continue the stream if an error is handled?
+9. Errors in the target stream cause it to unpipe from the source. The problem is that the target stream will not end. they [don't end the (transform) stream](http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm). Adding an automatic end event seems like a good idea. This should be possible if the multicast
+
+Stuff that are good:
+
+1. backpressure + multiple consumers - I think its rare to get this in a single package. Might be better to have a separate "multicaster" stream that lets you do this, rather than it being allowed all the time. As it is now, it only works if multiple consumers are being pushed the data (they use the `data` event), but not if they're pulling the data (they use the `readable` event and `read` method)
+
+### Potential solution
+
+
+The solution here would be to extract the buffering/encoding logic out into separate transform streams and extract multicast into a separate MulticastStream. 
+
+
+#### Stream buffers (decoders)
+
+Stream buffers (decoders) will pull data until they fills up to highWatermark. Depending on the type of buffer it will then do different things when pulled from:
+
+* `ByteBuffer` will return the entire combined buffer of the data thus far
+* `UTF8Decoder` will return as much data as possible that is a valid UTF-8 string, will keep the rest in the buffer
+* `ObjectBuffer` will return a single object of the data thus far.
+
+Then the buffer streams will start pulling again until they're full and so on.
+
+Extracting the buffers will eliminate all inconsistencies pertaining to encoding and object mode and will decouple them from the main stream logic.
+
+#### Multicast stream
+
+To  enable multicast piping there will be a separate writable stream called multicast stream. It will be capable of generating multiple Readable streams. Example API:
+
+```
+var multi = readable.pipe(new Multicast());
+
+multi.fork().pipe(target1);
+multi.fork().pipe(target2);
+```
+
+
+Extracting multicast will enable proper error propagation for regular transform streams. Since pipe will only be possible between two streams, the rules can be vastly simplified:
+
+* An error event propagates forwards, and will unsubscribe (cancel, unpipe) all transform streams backwards.
+
+The complex rules will only remain in MultiCast:
+
+* It will propagate errors forwards to all forks
+* It will only propagate cancelations to the source if all forks are unpiped (unsubscribed)
+
+No error propagation or cancellation will happen through Duplex streams.
No results found