SECTION 01The C abstract machine
Open the C standard and you'll find a description of an imagined computer. Bytes are uniform. Memory is a flat array. Instructions execute in the order you wrote them. Every operation takes roughly the same time. The standard never quite says these things outright, but it builds a world where reasoning about them otherwise feels unnatural.
This world is convenient. It is also false in ways that span six orders of magnitude.
The real machine is asymmetric to a degree that's hard to believe until you measure it. A register access is essentially free. A main-memory access can stall the entire pipeline for hundreds of cycles. A predictable branch is invisible. A mispredicted branch flushes 15+ stages of speculative work. A linear scan of an array can run 50× faster than a linked list with the same big-O.
None of this is in the C standard. None of it is visible in your source. But all of it is the difference between a program that uses 5% of the machine and one that uses 95%.
The two models
/* Looks like: read a, read b, add, store result. */ /* Each step seems equal-cost. Order seems fixed. */ int sum(const int *a, const int *b) { return *a + *b; }
/* load *a → may hit L1 (~4c), miss to L3 (~40c), or miss to DRAM (~250c) */ /* load *b → can run in parallel with *a when there's no dependency */ /* the ADD is ~1 cycle — completely dominated by either load */ /* meanwhile the CPU is speculating ~500 µops ahead */ /* the "return" is just the retirement of a result that already finished */
The same six lines of C can run with wildly different performance depending on what the machine is doing underneath. Mechanical sympathy is the practice of holding both models in your head at once, and writing code where they line up.
SECTION 02Cache lines are the actual unit
When your code reads one byte, the CPU does not fetch one byte. It fetches 64 bytes: an aligned chunk called a cache line, which becomes the smallest unit of memory traffic from there to the bottom of the hierarchy. Every load, every store, every coherence message — they all operate on cache lines.
This single fact has enormous consequences. Reading arr[0] and then arr[1] is essentially free for the second access — it's already there. Reading arr[0] and then arr[1000000] is two full misses. Linear traversal isn't just slightly faster than random access; it can be 50–100× faster, with the same big-O.
Latency the way the silicon sees it
| Layer | Latency | Bandwidth | Capacity | Shared? |
|---|---|---|---|---|
| Register | ~1c | ∞ | 16 GPR | per-core |
| L1d | ~4c | ~1 TB/s | 32–64 KB | per-core |
| L2 | ~12c | ~500 GB/s | 0.5–2 MB | per-core |
| L3 | ~40c | ~200 GB/s | 10–64 MB | shared |
| DRAM | ~250c | ~50 GB/s | GBs | shared |
| SSD | ~100,000c | ~5 GB/s | TBs | device |
Array of structs vs. struct of arrays
Try this — toggle between two layouts of the same logical data. The cache-line view shows what the CPU actually fetches when your hot loop touches a single field across many entities.
The same code, two layouts
typedef struct { float x, y, z; /* 0..11 */ float health; /* 12..15 */ char name[32]; /* 16..47 */ int id; /* 48..51 */ double weight; /* 52..55 pad, 56..63 → 64 B total */ } Entity; Entity es[10000]; for (int i = 0; i < 10000; i++) es[i].x += dt; /* 1 useful float per cache line */
typedef struct { float x[10000], y[10000], z[10000]; float health[10000]; char name[10000][32]; int id[10000]; } Entities; Entities es; for (int i = 0; i < 10000; i++) es.x[i] += dt; /* 16 useful floats per cache line */
The first version wastes 60 of every 64 bytes the CPU fetches. The second uses every byte. On a memory-bound loop, the SoA version is commonly 5–10× faster — no algorithmic change, no parallelism, just code shaped around how memory actually moves.
Pointer chasing is the enemy
A linked list of a million nodes scattered across the heap is a performance disaster, even though traversal is O(n). The reason is geometric: each node = node->next is almost certainly a cache miss, and worse, it's a dependent load — the CPU cannot prefetch the next access because it doesn't know the next address until this load completes. You've serialized yourself against memory latency at full ~250-cycle cost.
An array traversal of the same logical data has none of this pathology. The prefetcher sees the stride, pulls lines ahead of demand, and the CPU mostly sees throughput, not latency. Same algorithm, same big-O, two completely different programs.
SECTION 03Inside the dataflow engine
The C standard talks about "the abstract machine" executing one statement after another. A real CPU like Intel's Golden Cove or Apple's Firestorm does nothing of the sort. Underneath, it is closer to a small superscalar dataflow processor that pretends to be sequential because the architects committed to that illusion forty years ago.
What's actually in there
- A frontend that fetches instruction bytes, decodes them into the CPU's internal RISC-like micro-ops (a single
add [rax+8], rbxbecomes 2–3 µops), and queues them up. - A reorder buffer (ROB) holding ~500+ in-flight µops simultaneously. This is the window of instructions the CPU is actively considering at any moment.
- Register renaming that maps the architectural registers you see in assembly (
rax,rbx, …) onto a pool of hundreds of physical registers, eliminating false dependencies. - A backend with 8–12 specialized execution ports — some integer, some FP, some load, some store, some branch — picking ready µops and running several per cycle, out of program order.
- A retirement unit that commits completed µops to architectural state in program order. This is what creates the illusion you started with.
fetch → decode → rename → [ROB · ~500 in flight] → dispatch ↓ execution ports (8–12 in parallel) ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ int int fp fp ld ld st br ↓ retirement (in program order) ↓ architectural state
The headline number is IPC — instructions per cycle. A well-tuned numerical kernel can sustain 4–6 IPC; it is using the wide machine. A pointer-chasing loop sits at 0.3 IPC because the dataflow engine is stalled waiting for memory. Same hardware. The code decides which one you get.
Speculation: the CPU runs the future
When the CPU hits a branch, it does not wait for the condition to resolve. The branch predictor guesses, the frontend keeps feeding µops down the predicted path, and the backend executes them speculatively. If the prediction was right (and it usually is, for predictable branches), the speculative work retires and you get its results for free. If it was wrong, the entire pipeline flushes, the speculative state is thrown away, and you pay 15–20 cycles to restart from the correct address.
This is also true for memory: the CPU speculates that a load won't alias with a recent store, executes the load early, and rolls back if it turns out the store mattered. It speculates that a function will return to where it was called from. It speculates that an indirect branch goes where it last went. The entire machine is forecasting your program a few hundred µops in advance, and the only thing that gives you the illusion of sequential execution is that mistakes get cleaned up before retirement.
SECTION 04The two numbers
Latency is the number of cycles between issuing an instruction and being able to use its result. Throughput is how often you can issue another one regardless of who's waiting on the result. The same instruction can have a small latency and a tiny reciprocal throughput at the same time — they are not in conflict, they measure different things.
| Instruction | Latency | Rec. throughput | What it means |
|---|---|---|---|
add r, r | 1c | 0.25 | 4 per cycle if independent |
imul r, r | 3c | 1 | 1 per cycle, 3-cycle wait for result |
vfmadd | 4c | 0.5 | 2 per cycle, 4-cycle dep distance |
div r | 12–40c | 4–17 | slow, non-pipelined |
load L1 | 4–5c | 0.5 | 2 loads/cycle, but 4c to value |
Dependency chains serialize the machine
Consider the most innocent loop in C:
double sum = 0; for (int i = 0; i < n; i++) sum += a[i]; /* sum depends on prior sum */
Each sum += a[i] depends on the previous one. The chain is n adds long, each with 4-cycle latency. The total time is bound to ~4n cycles regardless of how wide the machine is, because the dataflow engine cannot execute the next add until the previous one's result is ready. You are using a tiny fraction of the FP unit's throughput.
Now break the chain into four independent ones:
double s0=0, s1=0, s2=0, s3=0; for (int i = 0; i < n; i += 4) { s0 += a[i]; s1 += a[i+1]; s2 += a[i+2]; s3 += a[i+3]; } double sum = (s0 + s1) + (s2 + s3);
Four independent chains, each n/4 long. The CPU interleaves them across its FP ports, hiding the 4-cycle add latency behind unrelated work. Throughput rises from one add every four cycles to roughly one add per cycle, so the loop finishes in ~n cycles — about 4× faster. (Fully saturating two FP ports would take even more accumulators; four is already enough to win big.)
The compiler will sometimes do this for integer types automatically. For floating-point it cannot, because reordering FP additions changes the result and the standard forbids it without explicit permission (-ffast-math, -fassociative-math). This is why people who care write code that explicitly exposes parallelism the compiler is forbidden from creating.
Dependency chains hide everywhere
Once you have this frame, you start seeing them in code you'd never have suspected:
- A linked list is a chain of dependent memory loads at full DRAM latency.
- A pointer-chasing tree traversal is the same, branching.
i++with a complex termination check can chain the loop counter with the body.- Building a
minviaifchains compares; replacing with(x < y ? x : y)may break it depending on whether the compiler emits acmovor a branch. - Updating a hash table's probe index by chained modular arithmetic serializes lookups.
SECTION 05The famous demo
One of the most-cited Stack Overflow answers of all time is about the following loop running ~6× faster on a sorted array than on a random one, with no other changes:
long sum = 0; for (int i = 0; i < N; i++) if (data[i] >= 128) /* branch depends on the data */ sum += data[i];
The instructions are identical. The data is identical. What changes is whether the branch predictor can guess correctly. On a sorted array the branch is taken consistently (no, no, no, … yes, yes, yes); the predictor learns this in two iterations. On a random array there is no pattern; the predictor is essentially flipping a coin and getting it wrong half the time. Every mispredict flushes ~15–20 cycles of speculative work. Over a few million iterations, this is the entire difference between fast and slow.
Try it
Feed the predictor a stream. Watch the hit/miss rate evolve over time. The "sorted" stream is what your branch sees when the data is ordered; "random" is the coin-flip case.
Branchless code, and when it helps
If a branch is unpredictable and small, sometimes the best fix is to eliminate it. Replacing a conditional store with a conditional move (cmov on x86) turns a possibly-mispredicted branch into a guaranteed-cheap data dependency. The compiler does this automatically in many cases:
/* branchy — speed hinges on the predictor guessing right */ int max_branch(int a, int b) { if (a > b) return a; return b; } /* ternary — the compiler often lowers this to a cmov */ int max_cmov(int a, int b) { return a > b ? a : b; } /* explicit bit trick — branchless and portable */ int max_bits(int a, int b) { int mask = -(a > b); /* 0 or all-ones (two's complement) */ return (a & mask) | (b & ~mask); }
The tradeoff: a branch you can predict is essentially free; you "execute" it for one cycle and the rest is speculative. A cmov always costs that cycle and creates a data dependency. So branchless wins only when the branch is hard to predict — typically when the condition depends on input data with no structure.
SECTION 06MESI in one paragraph
Every cache line in every core's cache carries a state: Modified, Exclusive, Shared, or Invalid. To write a line, a core must first own it Modified or Exclusive — which means messaging every other core that has a copy and making them invalidate theirs. To read a line another core has Modified, the owning core must first write it back. These messages travel a coherence interconnect that crosses the entire die.
So an atomic increment of a contended counter is not really one instruction. It is a cache-line ownership transfer: invalidate every other copy, fetch the latest data, modify, possibly write back. Uncontended, this takes ~20 cycles. Contended across 16 cores, it can take thousands. This is why "atomic" feels free in benchmarks and catastrophic in production.
False sharing
The most painful version of this happens when two threads write to different variables that happen to live on the same cache line. From your C-level view, the variables are independent. From the hardware view, they are the same coherence unit. Every write by one thread invalidates the other's copy. The cache line ping-pongs between cores on every write.
What this looks like in C
struct Counters { long a; /* thread A increments this — only this */ long b; /* thread B increments this — only this */ }; /* 16 bytes total · same 64 B cache line */ /* Looks parallel. Runs ~100x slower than the padded version below. */
#include <stdalign.h> struct Counters { alignas(64) long a; /* a starts on its own cache line */ alignas(64) long b; /* b forced onto the next line */ }; /* sizeof == 128: one line each */ /* Each counter owns a cache line — no ping-pong. alignas already inserts the gap; no manual padding needed. */
Nothing in C tells you this layout matters. Nothing in your algorithm changes. But the geometry of the cache-coherence protocol does not care about your intent — only about which 64-byte chunks two cores want at the same time.
Atomics, locks, and the same underlying truth
Every lock-free data structure you've admired is fighting this geometry. A well-tuned MPMC queue carefully separates its head and tail onto different cache lines so producers and consumers don't fight. A sharded counter gives each thread its own line and sums on read. A spinlock with backoff is trying to limit how often it touches a hot line. All of these are designs against the coherence protocol's tendencies.
SECTION 07Undefined behaviour as license
The single thing most C programmers underrate about their compiler is this: when the standard says some behaviour is undefined, the compiler is not granted "permission to do something weird." It is granted permission to assume the behaviour never happens, and optimize accordingly.
Signed integer overflow is the cleanest example. The C standard says it's UB. So when the compiler sees:
for (int i = 0; i <= n; i++) { /* … */ }
…it knows i cannot wrap. So it can prove the loop terminates, vectorize aggressively, widen the index to 64-bit pointer arithmetic without checks. Change int to unsigned, and wrapping becomes defined — those optimizations now have to assume the loop might wrap, and some of them disappear. UB is not a curse word; it's a contract you make with the compiler about what it doesn't have to prove.
Aliasing, restrict, alignment
Most of the compiler's optimization power depends on what it can prove about your pointers. Specifically: do these two pointers refer to the same memory? If they might, the compiler must conservatively assume that writing through one invalidates reads through the other — which kills load reordering, vectorization, and dependency-breaking. If it can prove they don't, the gloves come off.
/* compiler must assume a, b, c may overlap → it cannot vectorize or reorder the loads/stores */ void add(const float *a, const float *b, float *c, int n) { for (int i = 0; i < n; i++) c[i] = a[i] + b[i]; } /* you promise no overlap → it can vectorize freely */ void add_restrict(const float *restrict a, const float *restrict b, float *restrict c, int n) { for (int i = 0; i < n; i++) c[i] = a[i] + b[i]; }
Same story with alignment. SIMD loads on aligned data are faster than unaligned ones; __builtin_assume_aligned(ptr, 64) tells the compiler to emit the fast form. With strict aliasing rules, type information lets the compiler reorder accesses to different types. The whole optimization pipeline is a chain of proofs, and you are either supplying the premises or denying them.
Read the assembly
Everything in this guide collapses to one habit: look at what the compiler produced. Godbolt makes it trivial. You write a function, see the asm at -O2, and if it isn't what you expected, you ask why. Usually the answer is a missing restrict, a function call that couldn't inline, a memory access the compiler couldn't prove was safe to reorder, or an FP reduction the standard forbade.
Once you do this for a while, you stop guessing. You stop saying "the compiler is smart, it'll figure it out." You start writing C with a mental model of the asm it produces. That is the closing move of mechanical sympathy.
restrict and alignment hints tell it what you've already proved. Read the asm — it's the only way to close the loop.CLOSINGThe meta-frame: think in bottlenecks
After all of this, the unifying mental move is this: at any moment, your code is bottlenecked on exactly one resource. Identifying which one is the entire game.
The candidates
| Bottleneck | Symptom | How to find it | What helps |
|---|---|---|---|
| Memory bandwidth | High cache miss, low IPC | perf stat cache-misses | better layout, fewer bytes/op |
| Memory latency | Backend stalls, deps | perf stat stalled-cycles-backend | prefetch, break deps, locality |
| FP / int throughput | High IPC at port limit | perf stat uops_dispatched | SIMD, more parallelism |
| Branch mispredict | High mispredict count | perf stat branch-misses | sort data, branchless |
| Coherence | Scaling tanks past 1 core | perf c2c for false-share | pad, shard, batch writes |
| Frontend | Decode/uop-cache pressure | perf stat idq.* | smaller hot code, align loops |
The roofline picture
The classic visualization is the roofline model: on the x-axis, your operational intensity (FLOPs performed per byte of memory loaded). On the y-axis, the performance you actually achieve. The ceiling is two lines: a sloped line representing memory bandwidth, and a flat line representing peak compute. Where they meet is the only place your code can sit.
If you are below the sloped line, you are memory-bound — more FP units would not help, but better locality would. If you are below the flat line, you are compute-bound — more bandwidth would not help, but vectorization would. The reshaping of code that mechanical sympathy enables is, almost always, an attempt to move your operating point from one regime into another.
perf
^
| ───────────────── peak compute (flat)
| ╱
| ╱
| ╱ ← memory bandwidth (sloped)
| ╱
└─────────────────────────────────────────→ ops per byte loaded
memory-bound | compute-boundThe deepest level
If everything in this guide compressed to a single sentence, it would be this:
Identify the gap between the model your code assumes and the machine your code runs on, then close it where it costs you.
Sometimes that means a different algorithm. Often it means the same algorithm with a different memory layout. Occasionally it means writing assembly. Always it means measuring rather than guessing — because intuition is wrong often enough that the only durable habit is to look at what the silicon actually did.
The mechanically sympathetic programmer has not memorized a list of tricks. They have stopped believing that the source code is the program. The program is what the silicon does. The source code is a proposal.