producer→ ❘❘❘ →consumer⟵ backpressure
Streams · the spine is one signal
contents

00 Explorable explanation · Streams from first principles

A fast producer, a slow consumer, and the one signal that flows backward to keep memory bounded.

Every stream you'll ever touch — a file read, a TCP socket, a gzip transform, a fetch() body — is a variation on a single situation: data is produced faster than it's consumed, and you refuse to hold it all in memory. The fix is backpressure: a signal that travels against the data, from the slow consumer back to the fast producer, saying slow down. Press play and watch the buffer fill, hit its high-water mark, fire the amber signal backward, pause the producer, drain, and resume — forever, in bounded memory. Hold this picture. Every later layer is this picture redrawn.

▶ producer · bounded buffer + watermarks · consumer · the backward signal
00 / 00
FRAME 00 · —

Press play to run one full backpressure cycle

Cyan is data flowing forward. Violet cells are the bounded buffer; the dashed line is the high-water mark. The amber arrow is backpressure — the only thing on screen that travels backward.

one hue per family · prose, code and diagrams agree backpressure — the backward signal node — runtime API & data moving forward os / kernel — buffers, syscalls, watermarks web spec — WHATWG streams footgun — lost backpressure
the one-line spine Understand backpressure → feel it for free with async iterators (pull) → see it physically in syscalls and kernel buffers → recognize classic Node streams as push-based machinery rebuilding it → watch Web Streams do the same thing cleanly and explicitly → trace it across a full chain. That sentence is the whole curriculum; everything below is one of its clauses, drawn out.

01 Phase 0 · the conceptual ground

Phase 0 — no code yet, just vocabulary

Two things run at different speeds, and the data won't fit in memory.

State the seed precisely, because everything grows from it. A producer emits data at one rate; a consumer absorbs it at another; the total volume is large or unbounded (a 10 GB file, an infinite socket). You cannot buffer it all — "read everything, then process" detonates your heap. So you stream: process in chunks with a bounded working set, and you need a way for the slow side to throttle the fast side. That throttle is backpressure, and it's the only idea in this whole document — seen at six heights.

Drag the slider below to feel why "read it all" is a non-starter. The same job — move a file to a slow sink — costs either a bounded few kilobytes of buffer or the entire file resident in RAM, depending only on whether you stream.

⊞ buffering vs streaming · peak resident memory for the same job file: 2 GB
file size 2.0 GB

The vocabulary you'll reuse at every layer. Tap each to open the detail — and notice that push vs pull is the axis that quietly decides whether backpressure is automatic or something you must build.

the question that never changes From here to io_uring, the question is always: when the consumer is slower than the producer, what tells the producer to wait, and where does the not-yet-consumed data sit while it does? Name the signal and name the buffer at each layer and you understand that layer.

02 Phase 0 · the single axis

Phase 0 — the design fork

Does the producer shove, or does the consumer ask?

One axis explains most of the design differences you'll meet. In a push model the producer calls the consumer the instant data exists — fast, but if the consumer is slow the surplus has to go somewhere, so push needs an explicit buffer and an explicit "stop" signal. In a pull model the consumer requests the next chunk only when it's ready for one — so the producer literally cannot run ahead, and backpressure is the default, not a feature. Toggle the two modes below and watch the queue. Same producer, same slow consumer; the only change is who's in charge of timing.

⊞ same speeds, two regimes · watch the in-flight queue
why this matters for everything below Async iterators (Phase 4) are pull → free backpressure. EventEmitter and classic Node streams (Phases 5–6) are push → they must rebuild backpressure with a buffer, a watermark, and a false/'drain' handshake. Web Streams (Phase 7) are push with explicit, first-class backpressure. Three points on this one axis.

03 Phase 1 · where backpressure is physically real

Phase 1 — the OS / kernel floor

Node's streams are a userspace mirror of mechanisms the kernel already runs.

"First principles" bottoms out here. Every stream reduces to two syscalls — read(2) and write(2) — operating on a file descriptor. Between them, data sits in kernel buffers: the page cache for files, the socket send/receive buffers for sockets, the fixed pipe buffer for pipes. And the kernel already has backpressure: on a non-blocking fd whose send buffer is full, write doesn't block — it returns EAGAIN. That refusal is the real backpressure signal. Node's writable.write() returning false is nothing more than the userspace echo of EAGAIN.

The diagram traces the whole chain for a TCP write: your write → the socket send buffer → the TCP sliding window → the peer's receive buffer → the peer app's read. A full receive window propagates back to a full send buffer propagates back to your write returning EAGAIN. Inspect it, then run the strace below to watch a real read/write/EAGAIN/epoll_wait interleaving.

▦ the backpressure chain · userspace → kernel → wire → kernel → userspace
00 / 00
FRAME 00 · —

Step the chain and watch which buffer fills first

Every element on this diagram is a door — click a buffer, a syscall or a numbered link to open the mechanism behind it.

Click any element of the diagram above — write(2), either kernel buffer, the wire, epoll_wait, read(2) or a numbered link — to open a live instrument for that one mechanism.

▶ strace -e read,write,epoll_wait · piping a file to a slow socket

Each clause of the chain is a card. The story climbs from a single fd up to io_uring; the punchline is always that the kernel already solved this, and Node is plumbing the signal up to JavaScript.

the echo you'll keep hearing Hold these three equivalences and Phase 6 writes itself: write() returns false ≈ the kernel returned EAGAIN; the 'drain' event ≈ epoll reports the fd writable again; highWaterMark ≈ SO_SNDBUF, the buffer size that decides when "full" happens.

04 Phase 2 · pull, for free, in pure language

Phase 2 — JavaScript's own flow control

Before any stream module: an async generator already has backpressure.

Build the cleanest possible stream out of language features alone, so that later you can see exactly what Node's machinery is rebuilding. An async generator produces values lazily: it runs to a yield, hands one value out, and suspends. A for await…of loop pulls the next value only when the body finishes. So if the body is slow, the generator simply waits at its yield — the consumer literally cannot pull the next chunk until it finishes the current one. Backpressure is automatic, and you wrote zero plumbing. Run it and watch the generator sit suspended while the slow body works.

▶ async generator (producer) ⟷ for-await body (slow consumer)
// the entire producer — lazy, suspends at every yield
async function* produce() {
  for (let i = 0; i < Infinity; i++) {
    yield makeChunk(i);     // ← hands ONE chunk out, then PAUSES here
  }
}                            //   no further chunk is built until asked

for await (const chunk of produce()) {
  await slowWork(chunk);    // ← consumer paces the producer, automatically
}                            //   the next() call only happens after this resolves
why this isn't the end of the story Async iterators are beautiful but limited: no native branching / tee (you can't fan one source to two consumers), no pluggable buffering policy (you get a queue of exactly one in-flight value), and awkward error & cleanup semantics (a thrown error mid-loop won't reliably run the generator's finally unless you're careful). Those three gaps are precisely what the richer stream types exist to fill.

05 Phase 3 · the push substrate

Phase 3 — EventEmitter

Classic streams are EventEmitters. You can't read their source without this.

An EventEmitter is a Map<eventName, Function[]> where emit is a synchronous for-loop over the array. Push-based I/O maps onto it naturally: a chunk arrives, emit('data', chunk) shoves it at every listener, synchronously, right now. That immediacy is the problem — emit has no buffer and no backpressure. If the listener is slow, emit doesn't care; it already moved on. Streams are built on this push substrate, and the buffering and flow control they add is exactly the thing the bare emitter lacks. Here are the stream-relevant events you'll meet properly in the next phase.

Every event name and every side below is annotated — click or tap one for what it actually announces, and when it does not fire.

eventsidewhat it announces
'data'readablea chunk is being pushed at you (flowing mode)
'readable'readabledata is available to pull with .read() (paused mode)
'end'readableno more data will ever come
'drain'writablethe buffer emptied below the mark — safe to write again
'finish'writableend() was called and all data is flushed
'error'bothsomething failed — unhandled, it throws
'close'boththe stream and its resources are torn down
the bridge to Phase 6 Every Readable emits 'data'/'end'; every Writable emits 'drain'/'finish'. The taxonomy below is just an EventEmitter with a buffer bolted on and a flow-control protocol expressed through these events.

06 Phase 4 · push machinery rebuilding Phase 2

Phase 4 — Node classic streams

Now the API stops feeling arbitrary: it's the drain dance, rebuilt for push.

Five types: Readable (source), Writable (sink), Duplex (both, independent), Transform (both, coupled — gzip, crypto), and PassThrough (a Transform that does nothing). Each holds an internal buffer governed by highWaterMark — which is a threshold, not a hard cap: cross it and write() still accepts your chunk and buffers it, but returns false to say "I'm over the line, please pause." When the buffer drains below the mark it emits 'drain'. That call-and-response is the drain dance, and it is the kernel's EAGAIN/epoll-writable cycle from Phase 3, rebuilt in userspace. Step through it.

▶ the drain dance · write() → false → wait 'drain' → resume
00 / 00
FRAME 00 · —

Press play to run the drain dance

Watch the boolean that write() returns, the buffer level against highWaterMark, and the 'drain' event that releases the loop.

Now make it live. Drag highWaterMark and consumer speed; the loop writes as fast as it can while respecting the false return. A correct loop never lets the buffer run away — that bounded ceiling is backpressure working.

⊞ live Writable · a write loop that respects backpressure
highWaterMark 16
consumer ms 160ms

pipe() vs pipeline()

pipe() wires a source to a destination and manages the drain dance for you — pause the source when the dest returns false, resume on 'drain'. Its flaw: on error it does not tear down the rest of the chain, leaking sockets and file descriptors and leaving half-open streams. pipeline() (and stream/promises' promise form) fixes exactly that — it propagates errors and destroys every stream in the chain, in order. The rule in real code: never bare pipe() across more than two stages; reach for pipeline().

the canonical bug Lost backpressure: a loop that calls write() and ignores the false. Flip the red toggle above and watch the buffer climb without limit — the exact unbounded-memory failure from Phase 1, reintroduced by hand. Every "my Node process OOMs streaming a big file" ticket is this.

07 Phase 5 · the clean redesign

Phase 5 — Web Streams (WHATWG)

The same idea, redrawn — with backpressure made explicit and first-class.

The WHATWG standard (now built into Node, and what fetch() bodies are) keeps the three roles — ReadableStream, WritableStream, TransformStream — but fixes classic streams' biggest wart: backpressure here is a number you can read. A controller exposes desiredSize = highWaterMark − queued. When it's positive the stream wants more and calls your pull(); when it goes to zero or negative, you stop enqueuing. No mode-switching, no implicit _read re-invocation guesswork — just a gauge. Drag the consumer speed and watch desiredSize swing, and pull() fire only when there's room.

⊞ ReadableStream controller · desiredSize drives pull()
highWaterMark 4
reader ms 220ms
classic ⟷ web, the same machine desiredSize > 0 ≈ classic's "buffer below highWaterMark, keep writing." pull() being called ≈ classic's _read() being re-invoked. A queuing strategy's size() ≈ object-mode vs byte-mode counting. Different ergonomics, identical physics — and both still echo EAGAIN.

08 Phase 6 · the layers, pulled together

Phase 6 — cross-cutting & real systems

One backpressure signal, propagating across four stages and a network.

The real test: file → gzip transform → socket → network. Each boundary is a place backpressure can stall, and a correct pipeline propagates one slow stage all the way back to a paused file read. In Node, req is a Readable and res is a Writable; an HTTP response streaming a gzipped file is exactly this chain. Watch the signal originate at a congested network and travel left, stage by stage, until the disk read itself pauses — and notice that no stage holds more than its own buffer.

▶ file → gzip → socket → network · trace the backward signal end to end
00 / 00
FRAME 00 · —

Press play to congest the network

Each stage has its own bounded buffer. The amber pulse is one backpressure signal moving left through the whole chain.

09 Phase 7 · synthesis & the wider family

Phase 7 — comparative view

Three points in one design space — and how other runtimes pick differently.

Pull, push-with-rebuilt-backpressure, push-with-explicit-backpressure: async iterators, classic Node streams, and Web Streams are not three unrelated APIs but three answers to the same question. Other ecosystems answer it too — some elegantly, one as a cautionary tale.

Every system and every model below is annotated — click one for how that ecosystem answers the same question, and what its answer costs.

systemmodelbackpressure
async iteratorspullautomatic — consumer paces producer
classic Node streamspushrebuilt — false + 'drain' + highWaterMark
Web Streamspushexplicit — desiredSize + queuing strategy
Go channelshybridbounded channel → send blocks when full
Rust Stream / tokiopullpoll_next — demand-driven, zero-cost
Reactive Streams / Java Flowhybrid"demand" requests — request(n)
RxJS observablespushnone native — instructive contrast
the modern footnote — io_uring Phase 1 said file reads aren't truly async on Linux, so libuv fakes it with a thread pool. io_uring changes that: a pair of shared ring buffers between userspace and kernel lets you submit real async file (and socket) operations without a thread per call. It doesn't change the concept — backpressure is still backpressure — but it reshapes the floor the whole tower stands on.

10 The whole thing, in one breath

A stream is a slow consumer's permission to slow a fast producer down — and every layer is that permission, expressed differently.

The kernel expresses it as EAGAIN. Async iterators express it by simply not calling next() yet. Classic Node streams express it as write() returning false and an eventual 'drain'. Web Streams express it as a desiredSize that drops to zero. Reactive Streams express it as request(n). They are dialects of one sentence. Learn to hear the sentence under the dialect and streams stop being API trivia and become — finally — obvious.

minimum viable path Short on time? Phase 0 (backpressure + push/pull) → Phase 2 ★ the async-generator build → Phase 1 ★ the strace exercise → Phase 4 (Readable/Writable + the ★ drain dance + pipeline) → Phase 5 (controllers, desiredSize, queuing strategies). That's the 20% that gives 80% of the real understanding.
Back to JIT-Atlas