proofstone
★ Star on GitHub Edit View source

Distributed Systems Engineer Roadmap

The pieces of this field are excellent. Nobody has stitched them together, and nobody covers the middle.

This is the map that does both. Every node is a milestone you can prove you passed — a test suite that goes green, a workload that survives a partition, a fork your toy chain resolves — not a keyword you can nod at. When you finish a node you have an artifact, not a feeling.

It is built for a backend or infrastructure engineer who ships services, is comfortable with concurrency and one language, and wants to stop treating consensus, linearizability and split-brain as words other people understand.

This is not a blockchain roadmap. Byzantine consensus gets one compact section, because that is the honest weight: in MIT's canonical course, Bitcoin is 1 lecture out of 22. If the blockchain career is what you want, roadmap.sh/blockchain is a different audience and not this map's fight.

Why this exists

The components of this field already exist, and they are excellent:

What nobody has done is stitch them together, and nobody covers the middle. 6.5840 is a graduate course, Go-only, with weeks of front-load; Maelstrom is a set of challenges plus a tutorial that stops where it stops. They are complementary and nobody has sequenced them — what to do first, where the forks are, how it connects to a job. And the middle — failure detection (SWIM), DHTs, storage engines, pointing Jepsen at your own code, model-checking with TLA+ — is taught by no one as a checkable milestone.

This map does three things:

  1. Sequences what already exists — what to do first, what to skip, where the forks are.
  2. Closes the gaps nobody teaches, each with a verifiable artifact.
  3. Gets a working engineer to consensus fast — you do not need all of a graduate course to stop being afraid of the word.

This is the honest demand signal, and it is a learning signal, not a job-board one: there are 4,438 public repositories of solutions to the 6.824 labs (1,068 for the current 6.5840) — thousands of engineers already grinding the artifact path — and "how do I actually learn distributed systems?" has been a recurring top question on Hacker News every year since 2011. The audience is already doing the milestones. What it does not have is a map.

A note on honesty, because this field will check: we did not invent any of the pieces, and we do not pretend the ground was empty. roadmap.sh has a good, actively-maintained System Design roadmap — but across its 147 nodes there is not one on Raft, Paxos, consensus, gossip, CRDTs or linearizability. That is the gap, and it is one of format and depth, not freshness.

How to use this

  1. Every node is a milestone: a You're done when … line that ends in something which either passes or fails. Do the milestone, not "the topic." No node here can be completed by reading.
  2. Pick your track once, at the start (below). Both tracks end in a consensus implementation verified by real tests.
  3. Where no good hands-on resource exists, the milestone is to build the artifact yourself — that is marked and explained, not hidden. One such gap, failure detection (§4), ships as a companion lab, swim-lab.
  4. The map is a spine, not a queue. §3 (consistency without consensus) can run alongside §2; §4–§6 are independent and you pick by what your job needs. See Order and dependencies.

Pick your track — the one fork you choose up front

Both tracks reach a consensus artifact verified by real tests. Choose by language and appetite.

Track A — Go Track B — any language
Engine MIT 6.5840 labs Maelstrom (Jepsen-based)
Get it git clone git://g.csail.mit.edu/6.5840-golabs-2026 — public, anonymous, no MIT affiliation needed jepsen-io/maelstrom releases
Verification go test -race — 28 tests for Raft alone 10 workloads, fault injection + consistency checking
Consensus path Labs 3A → 3D → 4 → 5 doc/06-raft → lin-kv workload
Best if you know or want Go; you want the deepest Raft test suite in existence you want your own language; you want gossip / CRDT / log coverage too

Track A has the harder Raft tests. Track B has the wider coverage. Many people do A for consensus and B for everything else. Each node below is tagged with the track that covers it.

The map

The path from working engineer to distributed systems engineer

Rendered deterministically by scripts/render_map.py — edit the data there, not the SVG.

Contents

Legend: ⭐ = a public artifact worth putting on your résumé (there are exactly five). 🔨 = a build-it-yourself milestone with no ready-made lab — the point, not a shortfall.

§1 — Foundations: why distributed is different

Kept deliberately short. You do not need a semester of theory to start — you need enough to know why the network is out to get you.

M1.1 — Your first node · Track B

You're done when your node passes maelstrom test -w echo --node-count 1 --time-limit 10.

Before consensus, before CRDTs, you need one node that speaks the protocol and a harness that grades it. Echo is trivial on purpose: it gets Maelstrom installed, your message loop working, and the feedback cycle — write, test, read the log — into your fingers.

M1.2 — The network is the enemy · Track B

You're done when your unique-ID node passes maelstrom test -w unique-ids --time-limit 30 --availability total --nemesis partition — i.e. it keeps serving during a network partition.

This is where CAP stops being a triangle on a slide and becomes a thing your process is doing right now. Generating globally-unique IDs with no coordination — while the network is cut in half — is your first real taste of the trade every later section refines.

M1.3 — Partial failure, RPC and concurrency · Track A

You're done when make mr passes: your MapReduce coordinator finishes the job correctly while workers are being killed.

MapReduce is the gentlest possible introduction to the real problem — a coordinator handing work to unreliable machines — and the 6.5840 harness kills workers mid-job to prove your recovery actually works. This is also how you get the Go labs cloned and building.

M1.4 — Time, clocks and causality · 🔨 build-it-yourself

You're done when your vector-clock implementation correctly labels every pair of events in a generated history as concurrent or happens-before — and disagrees with wall-clock ordering on at least one pair you can explain.

"Which event happened first?" has no answer across machines, and the fix — logical and vector clocks — underlies everything from causal consistency to conflict resolution. No off-the-shelf lab grades this, so building the clock and the checker is the milestone: a few hundred lines that make happens-before concrete.

§2 — Consensus: the fear you came here to lose

The core of this map. Everything before it is on-ramp; everything after it assumes it. Both tracks reach a consensus artifact — Go labs with the deepest Raft test suite in existence, or Maelstrom's language-agnostic Raft tutorial ending in a linearizable store.

Track B note — where the fail-condition lives. Maelstrom's Raft tutorial guides you through M2.1 (election) and M2.2 (replication) but does not grade each step on its own; on Track B those are guided build steps, and the gradeable gate is M2.6, where maelstrom test -w lin-kv --nemesis partition either passes or fails. That is the node that proves the ones before it — because on this map nothing counts until something passes or fails.

M2.1 — Leader election · Track A / B

You're done when make RUN="-run 3A" raft1 passes all three 3A tests (TestInitialElection3A, TestReElection3A, TestManyElections3A) — or, on Track B, your nodes elect a single leader per term under the Maelstrom Raft harness.

Election is where Raft begins and where most subtle bugs are born: split votes, terms that go backwards, two leaders who both think they won. Get it right and every later part builds on solid ground.

M2.2 — Log replication · Track A / B

You're done when make RUN="-run 3B" raft1 passes all ten 3B tests (TestBasicAgree3B … TestCount3B, incl. TestBackup3B) — or your Track B leader replicates entries and repairs divergent follower logs.

Replication is the heart of the algorithm: the leader appends, followers agree, and conflicting tails get overwritten to converge. TestBackup3B deliberately forces followers apart and checks that your leader drags them back into line.

M2.3 — Persistence, and the case that breaks everyone · Track A

You're done when make RUN="-run 3C" raft1 passes all eight 3C tests — including TestFigure8Unreliable3C — and you can explain out loud what Figure 8 of the Raft paper is protecting against.

If you remember one milestone from this map, make it this one. Figure 8 is the scenario where naïvely committing an entry from a previous term corrupts the log — and it is where "I understand Raft" turns out to be false for most people. Passing it under an unreliable network is the moment Raft stops being a diagram.

M2.4 — Snapshots and log compaction · Track A

You're done when make RUN="-run 3D" raft1 passes all seven 3D tests (TestSnapshotBasic3D … TestSnapshotInit3D).

A log that only grows eventually eats the disk. Snapshotting lets a node discard applied entries and bring a lagging follower up to date with a state transfer instead of a replay — the last piece before your Raft is production-shaped.

M2.5 — A Raft that actually holds · ⭐ Track A

You're done when make raft1 passes all 28 tests with -race, repeatedly — run it ten times and get ten passes, not one lucky green.

This is the headline artifact of the whole map. A Raft that passes the full 6.5840 suite under the race detector, reliably, is a thing very few engineers have built and can point to. Publish the repo; it is the single strongest line on your résumé this roadmap produces.

M2.6 — A linearizable service on top of your consensus · ⭐ Track A / B

You're done when make kvraft1 passes (Track A) or your store survives maelstrom test -w lin-kv --nemesis partition (Track B).

Consensus is a means; a service is the end. Putting a key/value store on top of your Raft — and proving it stays linearizable under partition — is the artifact that shows you can turn an algorithm into something a caller can actually trust.

M2.7 — Paxos: why Raft had to be invented · 🔨 build-it-yourself

You're done when your single-decree Paxos passes a proposer-crash test suite, and you can name one thing "Paxos Made Simple" leaves unspecified that Raft nails down.

You do not truly understand why Raft is shaped the way it is until you have felt Paxos leave the awkward parts as an exercise. 6.5840 teaches Paxos in a lecture but gives no lab, and no artifact-verified Paxos exercise exists — so the milestone is to build single-decree Paxos and a crash test yourself.

M2.8 — Shard it, then reconfigure it without stopping · ⭐ Track A

You're done when make shardkv passes: your sharded store serves reads and writes correctly while shards are migrating between replica groups.

One Raft group does not scale; real systems shard across many and move shards around live. Serving correctly during a reconfiguration — no dropped writes, no double-serve — is the hardest lab in 6.5840 and a direct proxy for the reconfiguration work infra teams actually do.

§3 — Consistency and replication without consensus

Consensus is expensive. Half of the real world buys something weaker on purpose. This section is where Track B shines — and where you learn to reason about the whole spectrum, from linearizable at one end to eventually-consistent at the other, and the quorum trade-offs (R + W > N, read-repair, hinted handoff) that live in between. It runs in parallel with §2.

M3.1 — Linearizability: what you have been paying for · Track A / B

You're done when you produce a history that a checker rejects as non-linearizable from a deliberately broken version of your own KV — and the same checker passes your fixed one.

Linearizability is the strongest single-object guarantee and the most expensive; you cannot decide when to give it up until you have seen a checker catch you violating it. Break your own store on purpose, watch the checker reject the history, then fix it and watch it pass.

M3.2 — Eventual consistency and CRDTs · Track B

You're done when your grow-only counter passes maelstrom test -w g-counter --nemesis partition with total availability; then extend to pn-counter and g-set.

CRDTs buy total availability by making concurrent updates merge instead of conflict. Building a g-counter that stays correct while partitioned — then extending to increment/decrement and sets — teaches the one idea behind Riak, Redis CRDTs and collaborative editors.

M3.3 — Gossip and broadcast · Track B

You're done when your broadcast passes maelstrom test -w broadcast under partition and meets the efficiency bar (messages-per-operation and latency budget) — correctness alone is not the milestone here.

Broadcast is easy to make correct and hard to make efficient: flood the network and you pass correctness but drown in messages. The Fly.io challenge grades both, so this is where you learn to trade latency against message load — the core tension of every gossip protocol.

M3.4 — A replicated log · Track B

You're done when your Kafka-style log passes maelstrom test -w kafka.

The replicated log is the backbone of modern data infrastructure — Kafka, event sourcing, write-ahead logs all share its shape. Implementing append, poll and offset-commit semantics that hold under Maelstrom's checks is a compact way to internalize it.

M3.5 — Distributed transactions · Track B

You're done when your store passes maelstrom test -w txn-rw-register; stretch goal: txn-list-append.

Transactions across keys and nodes are where isolation levels stop being trivia and start being the difference between correct and corrupt. This challenge makes you implement multi-key operations that stay available and consistent under fault injection.

§4 — Membership and failure detection: the uncovered middle

The clearest gap in the field, and the one this roadmap fills with its own code. Consensus assumes you know who is in the cluster — but deciding who is alive is a hard problem taught by no one as a checkable exercise.

M4.1 — Who is even alive? SWIM · ⭐ 🔨 companion lab: swim-lab

You're done when your SWIM-based membership converges on a 20-node cluster, detects a real death within the window, and produces zero false positives under 10% packet loss — i.e. go test ./... is green in the swim-lab repo.

SWIM libraries are plentiful — Corrosion, ScaleCube, hashicorp/memberlist all implement it — but a teaching lab with a test suite did not exist anywhere. That is exactly why it is here. This roadmap ships that lab as a companion repo, swim-lab: a SWIM membership test suite (in the style of the 6.5840 labs) plus a reference implementation, so you build your own detector and prove it converges and stays quiet under loss. It also ships the naïve heartbeat detector that fails under loss, so you can watch exactly why indirect probing and suspicion exist.

§5 — Partitioning and placement

Once you have more data than one machine, you have to decide where each key lives — and how little you can disturb when the cluster grows or shrinks.

M5.1 — Consistent hashing · 🔨 build-it-yourself

You're done when a property test shows your ring moves no more than ~K/n keys when a node joins or leaves, across 1,000 random configurations.

Consistent hashing is the answer to "add a node without reshuffling everything," and its correctness is a property, not an example — which is why a property test over a thousand random rings is the right proof. It is small, it has no ready-made lab, and it appears in nearly every distributed store, so building it yourself is high-value-per-line.

§5.2 — Reconfiguration without downtime (not a separate milestone)

Covered by §2.8 (make shardkv) — moving shards between groups while serving is the reconfiguration milestone. Cross-referenced here; no duplicate node.

§6 — DHT and peer-to-peer

Distributed hash tables push placement all the way to its logical end: no coordinator at all, every node finding any key in a logarithmic number of hops.

M6.1 — Kademlia / Chord · 🔨 build-it-yourself

You're done when your node joins a 50-node simulated network and resolves any key in O(log n) hops — and lookups still succeed after 30% of nodes vanish.

A DHT is consistent hashing plus routing plus churn tolerance, and building one is the clearest way to feel how BitTorrent, IPFS and Ethereum's discovery layer actually find things. No turnkey teaching harness exists, so the simulation and the hop-count check are yours to build; a BitTorrent client is the friendliest on-ramp to the same machinery.

§7 — Storage: what the data actually sits on

Every consensus log and every KV store eventually writes bytes to a disk that can lose power mid-write. This section is the layer underneath everything above.

M7.1 — Write-ahead log · 🔨 build-it-yourself

You're done when your WAL recovers with zero torn or lost committed records after kill -9 at 100 random points mid-write.

The write-ahead log is how a database survives a crash: append the intent, then apply it, so recovery can replay. Proving yours loses nothing across a hundred randomly-timed kills is a crash-consistency test you build yourself — and the same discipline your Raft persistence (M2.3) already leaned on.

M7.2 — LSM tree / B+Tree

You're done when your storage engine passes its own test suite for crash recovery, range scans, and compaction under concurrent writes.

The two dominant on-disk structures — B+Trees (read-optimized) and LSM trees (write-optimized) — are the choice every database makes first. Building one, with a test suite that survives crashes and concurrent compaction, turns "why is my database slow on this workload?" into something you can reason about from the disk up.

§8 — Testing and verification: the part that gets you hired

The part nobody teaches, and the part that separates an engineer a lab will pay from someone who "read about Raft." Everyone reads Jepsen reports about other people's databases; almost nobody has run one against their own.

M8.1 — Point Jepsen at your own code · ⭐ 🔨 build-it-yourself

You're done when Jepsen finds a real linearizability violation in a deliberately broken build of your KV — and reports clean on the fixed one.

This is the asymmetry that gets you hired: turning the tool everyone admires from the outside onto your own store. Maelstrom is the gentle on-ramp; a full Jepsen harness against your KV, catching a bug you planted and clearing the fix, is an artifact almost no candidate has.

M8.2 — Deterministic simulation testing · 🔨 build-it-yourself

You're done when your harness reproduces a specific interleaving bug from a stored seed, byte-for-byte, on demand.

Deterministic simulation is how FoundationDB and TigerBeetle find bugs that only appear one time in a billion interleavings — inject all non-determinism (clock, RNG, network, disk) behind a seed, and any failure replays exactly. Building even a small version changes how you think about testing concurrency forever.

M8.3 — TLA+: model-check the thing you built

You're done when TLC finds the violation in a deliberately broken spec of your Raft — and proves the invariant holds on the fixed spec.

Tests sample the state space; a model checker searches it. Writing a TLA+ spec of your Raft and watching TLC find a safety violation you missed — then proving the fixed spec holds — is the closest thing to a correctness proof most engineers will ever hold in their hands.

§9 — Byzantine faults and blockchain: one compact section

One section, because that is the honest weight. Everything so far assumed nodes fail by stopping; here they fail by lying. You need to know what that changes — and no more than that, unless the blockchain career is your actual goal.

M9.1 — When nodes lie · Track A

You're done when you can demonstrate, against your own Raft, a scenario where a single malicious leader corrupts the log — and state precisely what PBFT changes to stop it.

Raft assumes honesty; one lying leader breaks it. Actually staging that attack on the Raft you built in §2 — and naming the specific mechanism (signed messages, quorum intersection, view changes) that Byzantine fault tolerance adds — is how you learn where the crash-fault world ends.

M9.2 — Nakamoto consensus as a special case

You're done when your toy blockchain accepts blocks, resolves a fork by picking the longest valid chain — and you can state the trade Bitcoin makes against Raft: probabilistic vs deterministic finality, open vs fixed membership, and what each costs.

Bitcoin is not magic; it is consensus with a different set of trades, and the fastest way to see that is to build a few dozen lines that mine blocks and resolve forks by chain length. Once your toy chain reorgs correctly, "probabilistic finality" and "open membership" stop being buzzwords and become knobs you have turned.

§10 — Where this leads

Two milestones to convert everything above into a career direction and an interview you can win.

M10.1 — Read a real one

You're done when you can trace a single write through the consensus path of etcd's source and point at the exact function where the entry is marked committed.

Reading production Raft closes the loop between the toy you built and the systems that run the internet. etcd is the right one to read: it is the most widely-deployed Raft in Go, its raft module is self-contained, and the commit point is a specific, findable function — not a maze. Trace one write from proposal to commit and the whole codebase opens up.

M10.2 — The job · (articulation milestone — no artifact, and we say so)

You're done when you can answer, with your own repo open on screen: "tell me about a time you debugged a correctness bug under partition."

This is the one node on the map with no artifact to pass or fail — it is career, not code, and pretending otherwise would be dishonest. The point is that by here you have the artifacts (a Raft that holds, a linearizable store, a Jepsen run, a SWIM detector) that make the answer real. Open the repo, tell the story, and let the work speak.

Order and dependencies 

§1 Foundations ──► §2 Consensus (core) ──► §2.6 linearizable service ──► §2.8 sharding
     │                    │
     │                    └──► §8 Testing & verification   (do M8.1 as soon as you have §2.6)
     │
     └──► §3 Consistency without consensus   (parallel to §2 — Track B can run alongside)

§4 Membership ─┐
§5 Partitioning├──► independent of §2; pick by what your job needs
§6 DHT ────────┘
§7 Storage ──► pairs naturally with §2.6 (your KV needs to persist)
§9 Byzantine ──► after §2 (know what Raft assumes before you learn what it can't take)
§10 ──► last

Forks:

Contributing

This roadmap lives or dies by one rule: every node is a checkable milestone, not a keyword. A milestone must state an artifact (You're done when …), and any resource must be open, live, and best-in-class — three excellent resources beat ten mediocre ones, and no resource appears twice. See CONTRIBUTING.md for the quality gate. Suggestions welcome via the issue templates.

License

Content is licensed CC BY-SA 4.0; the code in scripts/ is MIT (see scripts/LICENSE). Share and adapt freely with attribution.