Leaders, locks and the interview answer

Part of the free System Design course on LogicWiz, in the chapter: Consistency and coordination. The interactive whiteboard exercises and simulations run in the app.

Week 8, Day 4: Two riders, one order

"Both workers were sure they owned the lock. The rider got assigned to two orders and was nowhere near either."

An SRE at SwiftKart

Why you want one leader

Many tasks go wrong if two machines do them at once: the nightly settlement job, assigning a rider to an order, deciding which database copy accepts writes. If two nodes both think they are in charge, you get duplicated payments and conflicting data: the split brain of Week 5.

The simple fix is a leader: out of a group of nodes, one is chosen to do the job and the others wait, ready to take over. A single writer also gives you one agreed order of changes, which is what keeps followers' copies identical. The hard part is choosing the leader when machines can crash and the network can partition, which is the leader election problem.

A lock that is not enough: the double-booking

Often the leader idea appears as a distributed lock: before assigning rider Ajay, a worker takes a lock on "rider-Ajay". It comes with a lease, a time limit (say 10 seconds), so a crashed holder cannot block everyone forever. Here is how it still fails:

  1. Worker A takes the lock and begins assigning Ajay to order 101.
  2. A pauses for 15 seconds. The cause might be a long garbage-collection pause (the runtime stops the program to free memory) or a stalled VM. It does not know it paused.
  3. The lease expires. Worker B takes the lock and assigns Ajay to order 102.
  4. A wakes up, still believing it holds the lock, and writes "Ajay is on order 101".

Both writes succeed. Ajay is double-booked. The lock was fine; A simply acted on an old belief.

Meera had assumed that a lock with a timeout made the worker safe. It does not. The holder cannot be sure its lease is still valid at the moment its write lands.

The fix is a fencing token: a number handed out with each lock grant, which only grows. A gets token 33; B, later, gets token 34. Every write to the rider table carries its token, and the table rejects any write with a token lower than the highest it has seen. When A wakes up and writes with 33, the table has already seen 34 and refuses. The resource, not the client, makes the final call.

⚠️ Warning: A lock service alone cannot make a protected action safe. If the thing being protected cannot check a fencing token, treat the lock as an optimisation to reduce duplicate work, not as a guarantee.

Raft, at the level of intuition

Writing leader election and agreement yourself is a common source of bugs. In practice, teams use a small coordination service, or a database built on a consensus algorithm such as Raft. Consensus makes a group of machines agree on one sequence of decisions. You only need the idea:

  • Time is cut into numbered terms. Each term has at most one leader.
  • The leader sends frequent small messages called heartbeats to say "I'm alive". Followers wait a random time; if no heartbeat arrives, a follower becomes a candidate, adds one to the term, and asks for votes.
  • A candidate wins with votes from a majority. With 5 nodes that is 3. Two candidates cannot both get a majority, because any two majorities share at least one node, which votes once per term.
  • A change counts as committed only when a majority has stored it. Each node keeps a log: its ordered list of changes. A node votes only for a candidate whose log is at least as up to date as its own. So any new leader already has every committed change.
  • If the leader is cut off in a minority group, it cannot commit anything. The majority side elects a new leader with a higher term. When the old leader reconnects, it sees the higher term and steps down.

Notice the pattern from last lesson: majority overlap again. And this is why consensus groups have 3 or 5 nodes: 5 nodes survive 2 failures; 3 survive 1.

flowchart LR
  L["Leader (term 7)"] -- "heartbeat" --> F1["Follower 1"]
  L -- "heartbeat" --> F2["Follower 2"]
  L -- "replicate write" --> F1
  L -- "replicate write" --> F2

"Strong or eventual?" in an interview

The answer to give is: "It depends on the feature, so let me choose per feature." Then show how.

Ask what a stale or conflicting read costs, versus what refusing or slowing the request costs. A wallet debit or a payment must be strongly consistent: a stale balance means money lost, and a short refusal is acceptable. The last item of stock during a sale is similar. A like-count, a view-count or a recommendation list can be eventually consistent: showing 4,812 instead of 4,815 hurts nobody, and being always available and fast matters more.

A strong answer names the partition behaviour too: "For payments I'd choose consistency; during a partition that side refuses writes. For likes I'd stay available and reconcile later."

📌 Summary: Expect two nodes to believe they lead, so the resource must check a fencing token. Raft uses terms and majority votes. In interviews, choose per feature: strong for money, eventual for counts.

What Meera learns next

You now have the building blocks: sizing, caching, databases, queues, consistency. The last step is practice. In the guided practice she designs complete systems end to end, with Kabir asking the questions you will hear in a real interview.

Check your understanding

  1. Worker A holds the "rider-Ajay" lock with a 10-second lease, then freezes for 15 seconds. Worker B takes the lock and assigns Ajay. What stops A's late write when it wakes?
  2. A 5-node Raft group is split into 2 nodes and 3 nodes. The old leader is on the 2-node side. What happens?
  3. Why do consensus groups usually run 3 or 5 nodes rather than 4?
  4. An interviewer asks: "Should SwiftKart's data be strongly or eventually consistent?" Which answer is strongest?