Node streams — Readable and Writable

The state branch answers "what is true about this stream right now". Its four leaves are not independent flags — they are the coordinates of a position in a state machine, and the same method call produces different results depending on where you are standing.

The same three states, four ways. The machine view shows how you get between them; the matrix shows what is true inside each; the trace shows what the queue does over time; and the live view lets you drive it and watch individual chunks pile up.

Why null is not "false, but earlier"

A stream at null has never been asked for anything, so nothing is being pulled from the source and nothing can be lost. Attaching a 'data' handler flips it to true and starts the machine. The classic bug — attach the handler a tick late, lose the opening chunks — is therefore not really about timing. It is that observation is the trigger. At false the machine has already started and been told to hold, which is a different situation: bytes are arriving and piling up in memory.

The full state surface

The payoff: four different nulls

read() returns null for four unrelated reasons, and only the state branch can tell them apart. This is the concrete argument for state existing at all.

The methods branch has a split running through it that the tree does not show. Four of the six leaves are consumer verbs. push() is a producer verb, and calling it on a stream you did not implement is a category error.

Four more methods sit off the diagram but belong to the same branch:

The three consumption surfaces, in code

Same file, same buffer, same state machine. Only the arrival mechanism differs.

The 'readable' idiom deserves its own look, because it is the only surface where you control both when you read and how much. It is the shape every framing parser ends up in.

pipe() is backpressure wiring, not a shortcut

pipe() looks like a convenience for "send everything over there". What it actually does is subscribe to 'data', call write() on the destination, and — the whole point — pause() this stream whenever that write() returns false, resuming on the destination's 'drain'.

The events branch is inherited machinery — these are ordinary EventEmitter events. What makes them stream-specific is that two of them are not passive notifications at all: subscribing changes the object's behaviour.

Ordering is the thing worth memorising

The two consumption idioms produce genuinely different event sequences. Reading them side by side is the fastest way to see why mixing them is a bug rather than a style choice.

Note the third 'readable' on the lower lane. It fires at EOF too, and the read() that follows returns null — that null is the end-of-file signal in this idiom, which is why the loop condition and the termination condition are the same expression.

Errors arrive differently on each surface

The protocols branch is where Readable stops being a class with features and becomes an object that two independent contracts can both address. Neither contract was designed for streams.

The EventEmitter half

Because Readable extends Stream which extends EventEmitter, the entire emitter API is available and behaves exactly as it does anywhere else — including the ten-listener warning, which on a busy stream is a genuine leak signal rather than noise.

The AsyncIterable half

The contract is one method returning an object with next(). Readable implements it, which is the entire reason for await works. It is worth seeing that this really is a thin adapter and not a separate engine.

The inverse direction

Readable.from() goes the other way: hand it any iterable or async iterable and get a real stream back, with a buffer, a mark, and the full four-branch surface. This is the sharpest demonstration that the two things are not equivalent — the conversion adds machinery, and it only runs in one direction without loss.

Now the second diagram. Three prongs — buffer, events, async iterator — each with its own call, all converging on one consumer. The shape is right, and the thing it hides is that the prongs are not peers.

Read the legend colours again. The buffer is the storage. The events are notifications about that storage. The async iterator is a protocol implemented using both of the others — it subscribes to 'readable', and when woken it calls read(). Drawn flat, the three look like siblings. They are a substrate, a signal, and an adapter.

Cold and hot — where the buffer comes from

An async generator is cold. Its body does not run until next() is called, runs to the next yield, then stops dead. Nothing accumulates because nothing is produced unasked: consumption and production are the same act.

A Readable over a socket is hot. The kernel fills its receive queue on the network's schedule and libuv notifies the loop. The producing half runs whether or not a line of your code is waiting. Production and consumption are two independent processes — and every difference between a stream and an iterator is the machinery required to reconcile them.

Twelve pushes against three pulls. Those nine unconsumed chunks are the entire argument — they exist, they are somewhere, and "somewhere" is the buffer. A cold iterator has no such gap and therefore needs no queue, no fill level, no limit, and no way to tell a source to slow down.

What follows from the gap

State at rest. Between two next() calls an iterator exposes nothing — the protocol defines no surface beyond the promise you hold. Between two read() calls a Readable is a live object with a fill level, a flow mode and an end flag. You can ask a paused stream how much it is holding. There is no question you can ask a paused iterator.

Fan-out. Ten listeners on 'data' each receive every chunk. Two iterators over one stream split the chunks between them, because each next() pops the same queue. Observation and consumption are different relationships, and only one prong offers both.

Lifecycle visible to third parties. pipe() and pipeline() work by watching stream-level events on both ends. An iterator has no 'close' and no 'error' for an outside coordinator to subscribe to. You cannot build pipeline() from iterators without inventing an event layer — that is, without rebuilding the stream.

The iterator is a projection, not a peer

So the sentence to hold is not "a Readable has an iterator". It is: a Readable can be projected onto the iterator protocol, and the projection is lossy. You give up the fill level, the pause control, the fan-out and the third-party lifecycle. That is a fine trade for most consuming code, which is why for await is the right default. It is a bad trade the moment you need to know how full you are, hold a stream open without draining it, or hand the same bytes to two observers.

The state branch listed buffer as one leaf. It is the largest thing in the object, and it is a linked list of whole chunks rather than a byte array — a choice that explains nearly all of its observable behaviour.

Chunks queue in arrival order and leave in that order. Nothing is copied on the way in: the Buffer the producer handed over is the same object sitting in the queue. Copying happens only when your requested size does not line up with the chunk boundaries, at which point the queue slices or joins to give you exactly what you asked for.

The mark is advisory, and that word is load-bearing

highWaterMark does not cap the buffer. It changes what push() returns and whether the machine asks for more. A producer that ignores the false return can push a hundred megabytes into a 16 KiB stream and Node will queue every byte. A single chunk larger than the mark is likewise buffered whole — the queue never splits an arriving chunk to respect the limit.

read(n) and the boundary problem

Because the queue holds whole chunks, read(n) has to reconcile your requested size against the sizes that happened to arrive. Smaller than the head chunk, it slices; larger, it joins across several; and if the queue cannot satisfy n at all it returns null and keeps everything rather than handing you a short read.

That last case is the one to internalise. read(n) is all-or-nothing, so a framing parser can ask for exactly a header length and trust it gets the whole header or nothing — no partial-read bookkeeping. unshift() completes the picture: read a chunk, find you took too much, push the remainder back on the head.

Two accounting modes, one shape

Object mode changes what the queue counts, not how it behaves. Each push() queues one item, readableLength counts items, and read() ignores its size argument and returns exactly one. setEncoding() sits on the other side of the same problem: it decodes on the way out and holds partial multi-byte sequences across chunk boundaries, so a character is never split in half by an unlucky packet.

Production mechanics is the loop that keeps the buffer between empty and full while the producer and the consumer never speak to each other directly.

The decoupling is the point. A file descriptor knows nothing about your for await loop; your loop knows nothing about read syscalls. Between them the machine runs a demand loop — when the buffer has room it asks the producer for more, and when the buffer is full it stops asking. That is the whole mechanism, and backpressure is just the name for the second half of it.

Before the loop, the straight line. Four things happen between a byte existing at the source and your code holding it, and each is a separate step with its own timing.

Step four is the one that closes the line into a loop. Once the mark is involved, the chain stops being a pipeline and becomes a feedback system — which is what the next diagram draws.

Three verbs, three actors

The tree puts read() and push() side by side on the methods branch, which hides that they belong to opposite halves of the contract. A third verb never appears on the tree at all because you never call it — the machine does.

Three rules govern _read, and each is easy to violate:

• The machine will not call it again until you have pushed at least once. A body that pushes nothing stalls the stream forever, with no error and no warning.
• The size argument is a hint. Pushing a different amount is legal and completely normal.
• Pushing after push(null) raises an error. EOF is a one-way door.
• Resource setup belongs in _construct() and release in _destroy(), not in the constructor — the machine holds every read until setup calls back, and guarantees release runs exactly once.

Watch the loop run

Move the two rates apart and the buffer absorbs the difference. Producer faster than consumer: the fill climbs to the mark, push() starts answering false, and the demand loop goes quiet. Consumer faster: the buffer sits near empty and every consume is immediately followed by a fresh _read(). Switching the consumer surface changes the log and nothing else.

Writable divides the same four ways, and three of the four branches line up almost leaf for leaf with Readable. The fourth does not, and that mismatch turns out to be the most instructive thing on the whole diagram.

The empty slot is the thesis

Readable satisfies two contracts. Writable satisfies one. JavaScript standardised a pull protocol — AsyncIterable, driven by for await — and never standardised a push one. There is no language-level sink protocol, no Symbol.asyncSink, no syntax that means "feed this until it says stop".

The practical consequence is the single biggest asymmetry between the two halves. On the reading side the language hands you a loop that handles backpressure for free, because awaiting the next value is the flow control. On the writing side there is no such loop, so the flow control is a boolean return value you have to check by hand — and every unbounded-memory bug in Node streams traces back to somebody not checking it.

The same object, seen by role

Reverse every arrow from the Readable roles diagram and you get this one. Your code is now the supply, the sink is the consumer, and the core sits between them holding exactly the same kind of queue.

Hold this next to the Readable roles diagram. There, three surfaces fanned out of the core because a consumer can be offered a choice of how to take delivery. Here there is one outlet, because the stream is not offering the sink a choice — it is handing chunks over on a fixed schedule, one at a time, and waiting for each to be acknowledged. That serialisation is a guarantee Writable makes and Readable never has to.

You have been writing to these all along

Writable's state branch carries one distinction Readable has no need for: the difference between you have stopped writing and everything you wrote has actually landed. Those are two flags, and confusing them is the classic truncated-file bug.

ended is not finished

Calling end() flips writableEnded immediately and synchronously. At that instant the queue may still hold megabytes that have not been handed to the sink, the file descriptor may not have seen a single byte, and _final has not run. writableFinished is what turns true after all of that completes.

Why did write() return false?

Readable's read() returns null for four unrelated reasons. Writable's write() is narrower — false has exactly one meaning — but the surrounding state still answers a different question: whether writing is possible at all.

Six leaves, and the same split runs through them: four are yours to call, one belongs to whoever implements the sink, and one — end() — is the only method on either half of the stream API that both writes data and changes lifecycle state.

Three more belong to the branch without appearing on the diagram:

write() has three arguments and people use one

The per-write callback is the most under-used part of the interface. It fires when that specific chunk has been handed to the sink and acknowledged — which is how you attribute an error to the exact write that caused it, rather than learning about it later from a stream-level 'error'.

end() is two operations wearing one name

Five events, and the pair that carries almost all of the meaning is 'drain' and 'finish'. One says "there is room again", the other says "there is nothing left".

Ordering

The whole life of a Writable, drawn once. Note that 'finish' comes after _final, and 'close' comes after everything.

One contract instead of two. Everything on this branch is inherited from EventEmitter, and the interesting content is what fills the gap the missing protocol leaves.

Because there is no protocol, there is no syntax, and because there is no syntax the ecosystem grew helpers to paper over the gap. finished() turns the terminal events into a promise; pipeline() drives a whole chain and reports once; and on the web-streams side a WritableStream writer exposes a ready promise, which is the nearest thing to await write(chunk) that exists.

One boolean. Everything about writing safely to a stream is contained in what you do with the value write() hands back, and the reason it matters is that ignoring it is not an error — it works, right up until it exhausts memory.

The contract

write() queues the chunk and then reports whether the queue is now at or over the mark. The chunk is accepted either way. There is no version of write() that refuses. The return value is a request that you stop calling, addressed to code that is free to ignore it — exactly like push() on the reading side, and advisory for exactly the same reason.

The grey marker is the part people get wrong. Crossing back under the mark changes nothing observable — no event, no callback. The stream stays silent until the queue is completely empty, and only then, and only if it owes you one, does it emit 'drain'.

The loop that gets it right

Written out, the correct loop is awkward enough that it explains why pipe() and pipeline() exist at all. Both of them are this loop, written once, correctly, on your behalf — which is the real argument for using them rather than any claim about elegance.

Watch it run

Push the writer faster than the sink and the queue climbs to the mark, write() flips to false, and nothing clears until the queue hits zero. Turn on the cork and watch the queue hold everything until you release it.

The Readable buffer queues chunks. The Writable buffer queues write requests — and that difference is why cork(), _writev and the per-write callback can exist at all.

Each entry holds the chunk, the encoding it should be converted with, and the callback to invoke once it has been handed to the sink. A Readable's queue has nowhere to put a callback, because nobody is waiting on an individual chunk's fate.

cork and the batch

cork() tells the stream to stop forwarding and let the queue grow. uncork() releases it. The payoff arrives only if the sink implements _writev: instead of one call per chunk, the whole batch arrives as a single array, which for a socket means one syscall instead of twenty.

Encodings and object mode

Between write() returning and the bytes reaching the sink there is a chain with a guarantee attached: the sink sees chunks one at a time, in order, and never sees the next one until it has acknowledged the last.

The consequence worth carrying: write() returning true means queued, not delivered. Three synchronous writes all return before the sink has seen a single byte. The only way to know a specific chunk arrived is its own callback.

Implementing the sink

Four hooks, and only the first is mandatory. Forgetting to call the callback in _write is the writing side's version of a _read that never pushes: the stream stalls forever, silently, with a full queue and no error.

Readable's lifecycle has one terminal success state. Writable's has a two-stage ending, because stopping the input and finishing the output are separate events with real time between them.

Knowing it is really over

Four ways to wait, in ascending order of how much they actually cover.

Laid side by side, the two halves are nearly a reflection. Reading the correspondences is the fastest way to make either side stick — and the four places the reflection breaks are the places worth memorising.

Where the mirror holds

Where it breaks

Four asymmetries. Every one of them is a consequence of the same underlying fact: a reader can be offered a choice about when to take delivery, while a writer is handing over goods that must be accepted in order.

Connect the two and you get the thing every real pipeline actually is: two independent queues, each with its own mark, with one signal crossing between them.

That chain is the entire point of the abstraction. A slow disk at the far end reaches back through two independent queues and two independent marks to throttle a socket read, and no line of application code participates. What pipe() contributes is only the middle link — subscribe, write, pause on false, resume on drain — and what pipeline() adds on top is the error path that pipe() never had.

Both halves in one object

A Duplex is not a stream that does both things — it is two of these machines in one object, with two queues, two marks and two lifecycles that do not know about each other. A socket can be finished writing while still reading.

A Duplex is not a stream that reads and writes. It is two complete stream machines sharing one object identity — two queues, two marks, two demand loops, two lifecycles — joined by nothing except the fact that they answer to the same variable.

That is why the map cannot be a single four-branch tree. The branches double. Every leaf you learned on the reading side is present, every leaf from the writing side is present, and only the protocols branch is genuinely shared.

Two machines, drawn honestly

The word "duplex" suggests data flowing through. It does not. Unless you are looking at a Transform, nothing connects what you write to what you read — a socket's two directions are as unrelated as two separate streams that happen to share a file descriptor.

Every state getter you have met appears here, prefixed by which half it describes. The prefix is not decoration — it is the only thing telling you which of the two machines you are asking about.

Constructing one

The constructor takes the union of both option sets, and the per-half options exist precisely because the defaults are often wrong for one direction.

Two method sets, three implementation hooks, and one destroy() that does not respect the division.

Piping a Duplex

Because it is both, a Duplex sits in the middle of a chain — and can appear on both sides of one.

One emitter, two terminal events. 'end' belongs to the readable half and 'finish' to the writable half, they fire in whatever order the world produces them, and 'close' waits for both.

Half-open is the single behaviour that only exists because the halves are independent, and the one that surprises people most: a stream can be completely finished writing while still delivering data.

On a TCP socket it maps directly onto the protocol. Calling end() sends a FIN, which tells the peer I have nothing more to send. It says nothing about what the peer may still send you, and the readable half stays open until they send their own FIN.

allowHalfOpen

The allowHalfOpen option decides whether the two halves are allowed to end at different times. It defaults true for a raw Duplex and for net.Socket, and that default is why a socket you thought you closed can sit open for minutes.

Duplexes are the streams you touch most often without naming them. Every one of them is two machines sharing an object.

Duplex.from

Duplex.from — the constructor shortcut. Hand it a source and a sink and it wires both halves for you, which is the quickest way to put a Duplex in a pipeline without writing a class.

A Transform is a Duplex where the two halves are connected — and connected by exactly one thing: a function you write. Everything else is inherited unchanged.

That single addition changes what the object is for. A Duplex is two conversations that happen to share a handle. A Transform is one conversation with a stage in the middle, which is why it is the only stream shape you can meaningfully put in the middle of a pipeline.

What Transform adds, and nothing else

One function, three responsibilities: take a chunk, emit whatever it becomes, and say when you are done with it. The third is the one that gets forgotten.

push versus the callback's second argument

There are two ways to emit, and they are not equivalent. cb(null, value) is shorthand for exactly one push; push() can be called any number of times, including none.

Two queues, two marks, one hook between them. Backpressure through a Transform is the interesting case because the stall can originate at either end and travels the same way regardless.

The chain is worth stating precisely. A slow consumer fills the readable queue; push() starts returning false; a well-written hook stops pushing and waits; the callback is delayed; the writable queue fills because nothing is being dequeued; write() returns false to whoever is upstream. Five links, no code in the middle.

Watch it stall

Move the three rates apart. When the consumer is slowest the readable queue fills first; when the hook is slowest the writable queue fills first. In both cases the producer eventually gets the same answer.

A Transform has one more ending than the streams it inherits from, and it sits in a gap most people do not know exists: after the last chunk is transformed, but before the readable half is allowed to end.

That gap is what makes aggregating transforms possible at all. A hook that batches records pushes nothing during normal operation — it accumulates — and would have no way to emit the final partial batch if the readable side ended as soon as input stopped. _flush is that opportunity, and the stream will not emit 'end' until it calls back.

Where the leftovers go

Almost every real parser has state that outlives the last chunk: half a line, an unterminated record, a checksum still to append. Without a flush those are silently dropped, and the failure looks like data loss rather than a bug.

Errors from either hook

Nothing new. Every getter and every event comes from the Duplex above it — which is worth stating plainly, because the surprising part is how many people expect a Transform to have a single unified state.

Ordering, and what it costs

Because _transform is called one chunk at a time and gated by its callback, a Transform is strictly sequential. That is a guarantee, and it is also a ceiling: a slow hook is a slow pipeline, and no amount of concurrency elsewhere helps.

Where Transforms actually turn up, and the two shortcuts worth knowing.

PassThrough

PassThrough is a Transform whose hook pushes exactly what it received. It sounds pointless and is genuinely useful: it gives you a stream you can hand out before you know what will fill it, a tap you can attach listeners to in the middle of a pipeline, and a buffer with a mark you control.

compose

compose() welds several transforms welded into one, so a chain can be passed around and piped as a single stream.

The whole shape, once more