Mission 00 · Node.js Internals Campaign · start here

Descriptors, ports and sockets — the five things behind .listen(3000)

The groundwork. The rest of this series uses four words constantly and defines none of them — "socket" appears nearly three hundred times across Parts 8 and 9. This part says what they are: what a file descriptor actually indexes, why a port is not an object, why a listening socket and a connected socket are different things wearing one name, and which of five layers each API call brings into existence.

14 sections18 figuresfive layersfd · port · socket · handle
§ 0 · THE SHORT ANSWER FIRST

Five things, and only one of them is a socket

The rest of this series uses four words constantly and defines none of them. "Socket" appears nearly three hundred times across Parts 8 and 9. This part is the one that says what they are.

When you write net.createServer().listen(3000), five distinct things come into existence, in four different places, and people call at least three of them "the socket".

JAVASCRIPTONE LINE, FIVE OBJECTS
const server = net.createServer();
server.listen(3000);

// server            a JS object in V8's heap
// server._handle    a C++ TCPWrap
// (inside it)       a libuv uv_tcp_t
// (inside that)     fd 17, an integer in your process
// (behind that)     a socket, in the kernel

They are not synonyms and they do not have the same lifetime. The JS object can exist with no socket behind it; the socket can outlive the process that created it; and the integer 17 means nothing outside your process.

!Lock this in

If you only take one thing: the file descriptor is a NUMBER, not a thing. It is an index into a table your process owns. Everything confusing about fds, from dup() to passing sockets between processes, follows from that one fact.

§ 1 · ONE CHAIN, FOUR BOUNDARIES

Which of these is the socket?

Each layer holds a reference to the one below it and adds something the layer below does not have. Click through them and note where each one lives, because the boundaries are where the confusion is.

Four objects nested in your process, and one that is not
§1 · what contains what
net.Servera JS object in V8's heapTCPWrapserver._handle, a C++ objectuv_tcp_ta libuv handlefd 17an integer in your fd tablerefers tothe socketin the kernel· buffers· a state machine· a four-tuple· a queueoutlives every layer left of ityour processthe kernel
Four of the five live inside each other in your process, each holding the next. Only the innermost — an integer — reaches across the boundary. Closing it drops ONE reference: if another process holds a descriptor for the same socket, the socket carries on serving.
buffers
a receive buffer and a send buffer
kernel
Bytes that arrived and are waiting for your read(), and bytes you wrote that are waiting for the network. Both are kernel memory charged to the socket — §9 walks them.
a state machine
ESTABLISHED, TIME_WAIT and the rest
kernel
Advanced by segments arriving and by your calls — which is why closing a connection is a negotiation rather than an event. Part 8 walks the full diagram.
a four-tuple
source IP · source port · dest IP · dest port
kernel
The connection's identity in the kernel's lookup table. Change any one of the four fields and it is a different connection — §6 is built on this.
a queue
two of them, on listening sockets
kernel
The SYN queue for half-open handshakes and the accept queue for completed ones, sized by the backlog argument. §9 places them; Part 8 is largely about them.
the arrow crosses the only boundary that matters — everything left of it is yours
Five layers, four boundaries
§1 · the identity chain
the boundaries are where each layer stops meaning anything

The two boundaries worth naming now: the fd is meaningless outside your process, and the socket is meaningless outside the kernel. Everything above the fd is Node's bookkeeping; everything below it is the operating system's.

§ 2 · THREE LEVELS, NOT ONE

A number, a description, and the thing itself

Almost every confusing thing about descriptors comes from collapsing three levels into one. There are three, and they are shared differently.

From the integer to the thing
§2 · three levels of indirection
one process's viewthe same integer in another process refers to something else entirelyfd 3a small integeran index into YOUR tableper processindexesintothe open file descriptionoffset · mode · flagsONE object, shared bydup() and fork()points atthe inode / the socketthe thing itselfreference-counted — dieswith the last referencetwo arrows, three lifetimes — and close() only touches the first arrow
Three levels of indirection behind one integer. The number is an index into your process's table; the entry points at a shared kernel object holding the offset; that object points at the thing itself, which lives until its last reference goes.
the file descriptor
a small integer · per process
main
An index into your process's descriptor table. 0, 1 and 2 are stdin, stdout and stderr by convention; the kernel hands out the lowest free number. The same integer in another process refers to something completely different.
the open file description
a kernel object · shared by dup and fork
kernel
What the entry points at. It holds the FILE OFFSET, the access mode and the status flags. Two descriptors that share one of these share their read position — which is why dup() behaves the way it does.
the inode or socket
the thing itself
kernel
The actual file on disk, or the actual socket with its buffers and its state machine. Many open file descriptions can point at one of these; the object stays alive until the last reference goes.
the table is per process · the description is shared · the object is shared harder
CTHE DIFFERENCE THAT MATTERS
int a = open("f", O_RDONLY);
int b = dup(a);          // same open file description: SHARED offset
int c = open("f", O_RDONLY);   // new description: INDEPENDENT offset

read(a, buf, 10);        // offset now 10 for BOTH a and b
                         // c is still at 0
!Lock this in

A descriptor is a reference, and the object behind it is reference-counted. close() does not close anything — it drops one reference. The socket goes away when the last reference does, wherever that reference happens to live.

§ 3 · DUP, FORK, AND THE POST

How two descriptors end up pointing at one thing

There are exactly three ways a second descriptor comes to refer to something that already exists, and Part 11's socket passing is the third of them.

Descriptors point at a shared object, not at the file
§3 · two arrows deep
process Afd 17process Afd 23process Bfd 5open file descriptionone shared objectholds the file OFFSETand the access modethe inodeor the socketthe thing itselfdup() · fork() · SCM_RIGHTSthree ways to arrive at the middle box —and all three SHARE the offset that lives thereseek in one, and the others move toocloses only when the lastreference is gone
Three descriptors, in two processes, all pointing at ONE open file description — which is why a seek through one is visible through the others, and why closing a descriptor is a decrement rather than a shutdown. Two arrows deep, not one.
the middle box is why dup, fork and SCM_RIGHTS all behave the same way
How a second reference is created
§3 · the three doors
dup() / dup2()
within one process
main
A second entry in the same table pointing at the same open file description. This is how shell redirection works: dup2(fd, 1) makes descriptor 1 refer to your file, so everything that writes to stdout writes there.
fork()
to a child process
kernel
The child gets a COPY of the descriptor table, so the same numbers refer to the same descriptions. Parent and child share file offsets — which surprises people the first time two processes interleave writes to one log.
sendmsg() with SCM_RIGHTS
to any process
kernel
Ancillary data on a unix domain socket. The integer does not travel; the kernel installs a reference to the same open file description in the receiver, under whatever number is free THERE. This is the mechanism Part 11 is built on.
in all three, one object gains a reference — nothing is copied

Note what is absent from that list: there is no way to name someone else's descriptor. You cannot ask for "fd 17 in process 4821". The number is only meaningful inside the table that contains it, which is why passing one requires the kernel's cooperation rather than just sending an integer.

§ 4 · THE TABLE HAS A SIZE

EMFILE, and why servers hit it

The descriptor table is finite and the limit is per process. Every connection your server accepts consumes one entry, and so does every file, every pipe and every watcher.

A finite table, and what full looks like
§4 · the ceiling
your process's descriptor tableone slot per descriptor — every file, socket, pipe and watcher0123456789100 1 2 · stdin stdout stderrconnections · files · pipesfreeulimit -n = 1024the whole table, connections includedthe accept queuecompleted handshakes, stillarriving while the table is fullaccept()EMFILEno free slot — the client seesa hang, not a refusal
Thirteen slots drawn; a common default is 1024. The queue keeps taking completed handshakes even when the table is full — accept() fails with EMFILE, nothing is refused, and the client waits on a connection your process can never pick up.
an idle connection costs a slot and some kernel memory, not CPU
BASHTHE NUMBER THAT DECIDES HOW MANY CONNECTIONS YOU CAN HOLD
$ ulimit -n
1024                      # a common default — 1024 CONNECTIONS, not 1024 requests

$ ls /proc/self/fd        # your process's table, as files
0  1  2  3  17  18  19

When the table is full, accept() fails with EMFILE — and the connection stays in the accept queue rather than being refused, so the client sees a hang rather than an error. A server that leaks descriptors degrades slowly and then stops accepting entirely, with no crash to point at.

!Lock this in

An idle connection costs a descriptor and some kernel memory, not CPU. That is why an event-loop server holds tens of thousands of them happily — and why the descriptor limit, not the CPU, is usually what caps it.

§ 5 · SIXTEEN BITS IN A HEADER

Not an object, not owned, not a resource the OS allocates

A port is a number in a range: 0 to 65535, because the field in the TCP header is sixteen bits wide. That is the whole definition. It is not a device, not a file, and not something the kernel creates.

A port is a field, not a thing
§5 · where the number lives
one TCP segment, first four bytesthe only place a port physically existsbit 0151631source port54321destination port3000sixteen bits each · 0 to 65535 · that is the whole definitionlooked up on arrivalthe kernel's lookup table3000 → the LISTENING socket8080 → (no entry · RST back)bind(3000)= INSERT INTO the tableEADDRINUSE = duplicate key
Two sixteen-bit fields in every segment, and one table in the kernel. A port is never created or destroyed — binding inserts a row mapping the number to your socket, and an arriving segment's destination field is the key the kernel looks up.
16 bits in the TCP header
two of them, source and destination
hw
Every segment carries both. That is the only place a port physically exists — in the bytes on the wire.
the kernel keeps a lookup table
from port to socket
kernel
"Binding" a port means inserting an entry into that table. EADDRINUSE is a duplicate-key error, nothing more exotic.
below 1024 is privileged
a policy, not a mechanism
kernel
A convention enforced by the kernel: you need root or CAP_NET_BIND_SERVICE to bind one. Nothing about port 80 is technically different from port 8080.
binding is an insert · EADDRINUSE is a duplicate key

Which raises the question the next panel answers: if a port is just a number in a table, how can one of them serve ten thousand simultaneous connections without ambiguity?

§ 6 · THE FOUR-TUPLE

Not the port — the pair of pairs

A connection is identified by four values together: source address, source port, destination address, destination port. The kernel demultiplexes incoming segments by looking up that tuple, and any two connections that differ in any one of the four are distinct.

Three connections, and what separates them
§6 · four fields, one identity
source IPsource portdest IPdest portconnection A203.0.113.754321198.51.100.93000connection B203.0.113.754322198.51.100.93000connection C198.51.100.454321198.51.100.93000each row is ONE connection — the four together are its identityone field differs · therefore a different connection
Three connections to the SAME server port. A and B differ only in the source port; A and C only in the source address. Change any one of the four and it is a different connection — which is why one listening port can serve thousands at once, and why the ephemeral range is the field that runs out.
change any one of the four and it is a different connection
What makes two connections different
§6 · the tuple
change any one of the four and it is a different connection

That is why your server needs exactly one listening socket on port 3000 to serve ten thousand clients. Every one of those connections has destination port 3000, and they are told apart entirely by the client's address and port.

!Lock this in

A listening socket is identified by a PAIR — local address and local port. A connected socket is identified by all FOUR. That difference is the reason the two are different kinds of object, which is §9.

§ 7 · THE CLIENT SIDE OF THE NUMBER

Where the other port comes from, and how you run out

The client never chooses its port. The kernel picks one from the ephemeral range when you connect(), and on Linux that range is about twenty-eight thousand numbers wide.

Which numbers the kernel will actually pick
§7 · the port space, to scale
28,232ephemeral portsper destination · 32768 to 60999ports 0–1023privilegedports 1024–32767registeredports 32768–60999THE EPHEMERAL RANGEports 61000–65535above the rangeTIME_WAITholds a tuple for ~60safter close, so you reachthe ceiling sooner thanthe arithmetic suggests
The whole port space, drawn to scale. The thick band is the only part the kernel draws from when it needs a source port — about 28,000 of 65,536 — and it is per DESTINATION, not per machine. Connections to two different servers do not compete for it.
the thick band is the only part in play · and it is per destination, not per machine
BASHTHE RANGE, AND WHAT IT LIMITS
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768   60999            # 28,232 ports

# so ONE client can hold ~28k simultaneous connections
# to ONE (destination ip, destination port) — and no more

The limit is per destination, not global, because the tuple includes the destination. Twenty-eight thousand connections to one upstream service exhausts it; the same client can open twenty-eight thousand more to a different one.

And this is where Part 11's TIME_WAIT comes back. A closed connection holds its tuple for roughly sixty seconds, so a client that opens and closes connections fast enough can exhaust the range with connections that are already finished. That is the shape of most "cannot assign requested address" incidents.

§ 8 · THE OVERLOAD THAT CAUSES THE CONFUSION

A listening socket is not a connection

"Socket" names two different objects with different fields, different lifetimes and different state machines. Almost every muddle in this area is these two being treated as the same thing.

What accept() actually produces
§8 · one of these, many of those
the LISTENING socket*:3000never carries dataholds two queuesone per process, foreverONE of theseconnected socket203.0.113.7:54321its own buffers · its own stateconnected socket203.0.113.7:54322its own buffers · its own stateconnected socket198.51.100.4:54321its own buffers · its own stateconnected socket… thousands moreits own buffers · its own stateMANY of theseaccept()
One word, two objects. The listening socket exists once and never carries a byte; every accept() produces a NEW connected socket with its own buffers, its own state machine and its own four-tuple — all of them sharing the same local port, because the other three fields differ.
the listening socket never carries a byte · every accept makes a new object
Listening vs connected
§8 · not the same object
a listening socket
identified by (local ip, local port)
kernel
Owns the SYN queue and the accept queue that Part 8 takes up. Has no peer, no send buffer, no receive buffer, and never carries a single byte of your data. Its only job is to manufacture connected sockets.
a connected socket
identified by all four tuple fields
kernel
Owns a send buffer and a receive buffer, has a peer, and runs the TCP state machine Part 8 walks — ESTABLISHED, CLOSE_WAIT, TIME_WAIT and the rest. This is the one that carries your bytes.
accept() is the factory
one in, one out
node
Called on the listening socket; returns a NEW descriptor for a NEW connected socket. The listening socket is unchanged and keeps listening — accept does not consume it.
one has queues · the other has buffers · they are never the same object
!Lock this in

When someone says "the socket closed", ask which one. A connected socket closing ends one conversation; the listening socket closing means your server stops accepting, and that is a completely different outage.

§ 9 · THE MEMORY BEHIND THE DESCRIPTOR

Buffers, queues, and who is allowed to touch them

Behind the descriptor is a kernel structure with real memory attached, and knowing which memory belongs to which object explains several things the rest of the series takes for granted.

What actually sits behind the fd
§9 · the memory
your processthe networkthe socketkernel memory, charged to the socketthe receive bufferbytes that arrived, not yet readsegmentsarriveread()drains itthe send bufferbytes you wrote, not yet sentwrite()fills itdrained asthe peer allowsthe state machineconnected sockets onlySYN + accept queueslistening sockets only
Two buffers, one state machine, and — on listening sockets — two queues. All of it is kernel memory charged to the socket, none of it in V8's heap: read() and write() are the only doors your process has into it.
the receive buffer
bytes that arrived, not yet read
kernel
Filled by the kernel from the network, drained by your read(). When it is empty, a non-blocking read gives you EAGAIN — Part 9's whole §6.
the send buffer
bytes you wrote, not yet sent
kernel
Filled by your write(), drained by the kernel as the peer's window and congestion control allow. When it is full, write() returns short — which is where Part 9's backpressure comes from.
the SYN and accept queues
listening sockets only
kernel
Half-open and completed handshakes waiting for accept(). Sized by the backlog argument. Part 8 is largely about these two.
the state machine
connected sockets only
kernel
ESTABLISHED, FIN_WAIT, TIME_WAIT and the rest, advanced by segments arriving and by your calls. Part 8's §21 walks it.
every buffer here is KERNEL memory, not your heap

None of this memory is in V8's heap, which is why a Node process holding ten thousand idle connections shows modest RSS and large kernel memory. The buffers are charged to the socket, not to your JavaScript.

§ 10 · WHICH CALL CREATES WHICH LAYER

Step the lines and watch the five layers appear

Now the two halves meet. Here is what each line of the opening example actually does, in terms of the five layers from §1 — which object comes into existence, which syscall fires, and what the kernel knows afterwards.

Which of them cross into the kernel
§10 · two lines, six calls
your JavaScriptNode C++libuvthe kernelcreateServer()one JS object · 0 syscallslisten(3000)now everything happensuv_tcp_bindstill no syscallsocket()the fd appears herebind()the row in the lookup tablelisten()the queues are allocatedthree of these six lines are syscalls — and all three land on the last lane
Read it downward: two lines of your code produce six calls, and only the ones reaching the right-hand lane cost a transition into the kernel. createServer creates ONE layer and fires nothing; listen fires three syscalls, and every layer below the JS object appears at once.
lanes are layers · every arrow reaching the right-hand lane is a syscall
One line at a time, five layers deep
§10 · the chain
three of these five lines are syscalls · the other two are bookkeeping

The shape to notice is how much of this is bookkeeping. createServer touches nothing outside V8. listen is where three syscalls fire in a row and every layer below the JS object comes into existence at once.

§ 11 · THE FACTORY, ONE LEVEL DOWN

accept() gives you a descriptor; Node gives you a Socket

A connection arriving does not create a JS object by itself. Something has to call accept(), take the descriptor it returns, and build the layers back up — and that is the same five-layer chain in reverse.

From a ready fd to a net.Socket
§11 · building a connection
the five layers, rebuiltfor THIS connection only — fd 17 stays as it isand goes straight back to listeningthe LISTENING socket*:3000 · fd 17accept queue: one completedhandshake waitingaccept(17)the connected socketits own buffers · state · four-tuplefd 23a new integer in your tableuv_tcp_tregistered with epoll for readinessTCPWrapa C++ wrap around the handlenet.Sockethanded to your connection handler
The chain from §1, built in reverse for one connection. The listening socket never becomes any of this — it manufactures the connected socket, accept() hands you the descriptor, and Node rebuilds the layers on top until your handler is called with a net.Socket.
the loop wakes: fd 17 is readable
a completed handshake is waiting
kernel
Part 8's accept queue is non-empty, so the kernel marks the LISTENING socket readable. "Readable" on a listening socket means "a connection is waiting", not "there is data".
accept(17) → fd 23
a new descriptor for a new socket
kernel
A new connected socket with its own four-tuple, its own buffers and its own state machine. fd 17 is untouched and still listening.
libuv wraps fd 23 in a uv_tcp_t
a handle for the loop
node
Registered with epoll so the loop learns when this connection has data.
C++ builds a TCPWrap, JS builds a net.Socket
the layers, rebuilt
main
Now the same five-layer chain exists for this one connection, and your connection handler is called with the top of it.
the listening socket makes sockets · it never becomes one
!Lock this in

Every connected socket in your process has its own descriptor, its own buffers and its own entry in the kernel's tuple table. Ten thousand connections is ten thousand of each — which is why §4's descriptor limit is usually the ceiling.

§ 12 · THE MODELS WORTH CORRECTING

Ten claims about the primitives

These are the beliefs that make the rest of the series harder than it needs to be.

Which of these are true?
§12 · mental models
sort before you check — a guess you have committed to is worth more than a correction you read
!Lock this in

Nearly every wrong take here treats a NAME as a THING. A descriptor is a name for a description; a port is a name in a lookup table; "socket" is one name for two different objects.

§ 13 · THE FIVE SENTENCES WORTH KEEPING

If you remember nothing else

Everything the rest of this series says about I/O sits on top of these.

What to carry away
§13 · the five sentences
a descriptor is an index, not an object
§2
main
Per process, meaningless elsewhere, and pointing at a reference-counted kernel object through one more level of indirection you can share.
a port is a number in a lookup table
§5
kernel
Sixteen bits in a header. Binding inserts a key; EADDRINUSE is a duplicate.
the four-tuple identifies a connection
§6
kernel
Which is how one listening port serves ten thousand of them, and why the client side runs out at about twenty-eight thousand per destination.
listening and connected sockets are different objects
§8
kernel
One has queues and no peer; the other has buffers and a state machine. accept() turns one into the other without consuming it.
createServer creates nothing outside V8
§10
main
The five layers below the JS object all come into existence at listen(), in three syscalls.
five sentences, one chain

Part 1 picks up from here with module resolution. Part 5 then follows a single fs.readFile down through these same layers to the disk and back.

Back to JIT-Atlas