Sample post - replace with your own writing.
Flight search is a strange workload. The data is enormous but perishable — a price is stale in minutes. The queries are combinatorially cruel — "Edinburgh to anywhere, sometime in October, cheapest month" is a real query shape, not an edge case. And the user tolerance is brutal: if results take more than a second or two to start appearing, travellers simply leave.
I spent three years inside that machine. Here is what stuck.
Cheap answers first, right answers second
The single most important architectural idea I absorbed: serve an approximate answer immediately and refine it live. A search kicks off dozens of partner queries that take seconds to resolve, but the user sees cached prices within ~150 ms, clearly marked, updating as live quotes land.
This shaped how I think about every system since. The question is never "how fast can we compute the answer" — it's "what's the best answer we can give at every point on the latency curve?" A robot's obstacle avoidance works the same way: a conservative answer now beats an optimal answer 200 ms late.
The cache is the product
Naive mental model: cache as optimisation layer in front of the real system. Actual situation: the cache was the system. Live partner quotes existed mostly to keep the cache honest.
Getting there meant treating cached data as a first-class citizen with its own semantics:
- Every price carried its age, and downstream consumers made policy decisions on it — display it, background-refresh it, or refuse it.
- Invalidation was probabilistic. Routes with volatile prices got refreshed aggressively; a mid-week domestic hop in March could safely sleep.
- We measured staleness cost in actual user harm (price changed between search and booking), not in abstract hit rates.
p99 is where the truth lives
Averages at that scale are a bedtime story. The interesting graph was always the 99th percentile, and p99 problems were almost never in the code path — they were in the shape of the traffic. One partner slowing from 800 ms to 8 s wouldn't move the median a pixel while quietly pinning a thread pool.
The antidote was aggressive isolation:
// Every partner gets a bulkhead; nobody sinks the ship.
var result = bulkheads.forPartner(partnerId)
.withTimeout(quoteBudget.remaining())
.call(() -> partnerClient.quote(query))
.recover(PartnerTimeout.class, () -> Quote.absent(partnerId));
Timeouts, bulkheads, load-shedding — none of it is glamorous, and all of it matters more than whatever clever thing the request handler does. A search that returns 34 airlines instead of 35 is a fine search. A search that waits for the 35th is an abandoned one.
Backpressure beats buffering
Every queue in the pricing pipeline started life as "a bit of slack" and grew into a lie about capacity. Buffers convert overload from an error you can see into latency you can't. The systems that behaved best under stress were the ones that said no earliest — shed at the edge, degrade the query (fewer dates, fewer partners), and keep the core loop tight.
Robotics re-taught me this within a month. A telemetry pipeline that buffers when the radio link degrades delivers a beautiful, complete history of the crash. You want the freshest frame, not all the frames — drop early, drop deliberately.
What didn't transfer
Web-scale habits assume you can retry. A stateless request that fails is a rounding error; a thruster command that fails is a dent. Moving to robotics meant unlearning the reflex that idempotent-retry solves everything, and re-learning respect for state you cannot re-fetch — the physical world is a database with no read replicas and a very unforgiving write path.
The short list
- Serve something useful at every point on the latency curve.
- Make staleness explicit and let consumers set policy.
- Watch p99, and hunt traffic shape before code.
- Prefer refusal to buffering; degrade loudly, early, and by design.
- Know which of your operations can be retried, and treat the rest as precious.
Hundreds of millions of searches a month is a wonderful teacher, mostly because it removes the option of pretending your system is simpler than it is.