How to build systems on top of models that are wrong sometimes by design. Written for the engineer who has a model behind an API and a product that has to work on Monday — not for researchers, and not for people collecting prompt tricks.
Nothing is written yet. Here is the whole map anyway.
Publishing the outline before the chapters is the honest version of “growing”: you can see the real territory now and decide whether it is worth coming back for, instead of discovering the shape one post at a time. Every chapter link below is a permanent address from today, so anything that references a chapter keeps working when that chapter lands. Two rules the book is held to — no figure written from memory, and every technique carries its failure mode and the eval that catches it.
Why this is separate from the Cost Handbook.
The Cost Handbook owns what it costs. This book owns whether it works. Neither repeats the other; each links across at the seam. If you are here to work out what something costs, that book is the one you want — it re-derives its numbers from vendor pricing feeds and carries the provenance. This one assumes you have decided to build the thing and now have to make it actually work.
67 chapters across 11 parts · 0 written · first chapters in progress
An AI engineer is not an ML researcher who ships, and not a backend engineer who calls an API. The job is building reliable products on an unreliable component. Everything else in this book follows from taking that sentence literally.
The role against ML engineer, researcher, and data scientist — and what you are accountable for when it breaks at 2am.
Every layer from token to product, who owns each, and which layer your bug is actually in.
Why the habits that make deterministic software reliable — unit tests, exact assertions, retries — mislead you here, and what replaces them.
Regex, a lookup table, and a form still win a surprising share of the time. The cheapest correct answer is the one to beat.
You cannot debug what you do not understand one level down. This part is the minimum mechanical model of an LLM needed to explain a bad output, not enough to train one.
Next-token prediction, attention, and the handful of mechanics that explain most production surprises.
BPE, why non-English and code cost more per unit of meaning, and the off-by-one bugs that come from counting characters.
Advertised length against effective length, position effects, and why the middle of a long prompt gets ignored.
Why the same prompt gives different answers, which knobs actually change that, and why temperature 0 is not determinism.
Similarity is not relevance. Dimensionality, normalisation, and the cases where cosine distance confidently lies.
Why public benchmarks do not transfer to your task, and how to run the only comparison that counts.
What extended thinking buys, where it hurts latency and quality, and deciding per call site rather than per app.
How non-text inputs are processed, where they silently degrade, and why a PDF is the hardest one.
A prompt is source code that happens to be written in English. It gets versioned, reviewed, and tested, or it rots. Context engineering — deciding what enters the window and in what order — is the larger half of the discipline and the less discussed one.
Version control, review, and regression tests for text. The workflow that separates a prompt from a lucky guess.
Specificity, ordering, negation failure, and why most 'the model ignored me' bugs are ambiguity bugs.
The tasks where three examples beat three paragraphs, and how badly chosen examples poison a distribution.
What it measurably improves, what it only appears to improve, and the tasks it makes worse.
Schemas, constrained decoding, validation, and repair loops — turning free text into something a type system can hold.
Assembly, ordering, compaction, and eviction. The discipline of deciding what the model sees on this turn.
Retrieval is how a model touches facts it was not trained on and cannot be trusted to remember. RAG is not a product feature, it is a search problem with a language model attached — and it fails for search reasons far more often than for model reasons.
Grounding, freshness, and provenance — plus the failure classes that retrieval cannot touch.
The quiet decision that sets a ceiling on everything downstream, and how to choose it from the query shape rather than a default.
What the index actually does, the operational cost of one more datastore, and when a Postgres extension is the correct answer.
Why pure vector search underperforms on names, codes, and exact phrases, and how lexical and dense retrieval get combined.
The cheapest large improvement available in most RAG systems, and how to measure that it worked.
Letting the model search, refine, and search again — against fetching once and hoping.
Missed retrieval, distracting retrieval, correct-but-ignored context, and stale ground truth. Diagnosis before treatment.
Short-term against long-term, what genuinely needs to persist between sessions, and why most memory features are a retrieval problem wearing a costume.
An agent is a loop that calls tools until it decides to stop. Almost every hard problem in this part is a consequence of those two words: loop, and decides.
A definition tight enough to build against, and the line between an agent and a workflow with good marketing.
How a model calls code, what the schema actually does, and where the round trip breaks.
Tool descriptions are prompts. Granularity, naming, error messages, and the interfaces that cause silent misuse.
Termination conditions, step budgets, progress detection, and stopping a loop that believes it is nearly finished.
ReAct, plan-and-execute, reflection, and router patterns — what each is good at and how each fails.
When a model can fix its own mistake, when retrying makes it worse, and how to tell the two apart at runtime.
Why writing code beats a hand-built tool for open-ended work, and the sandbox that makes it survivable.
Standard interfaces between models and tools, what they standardise, and the trust boundary they introduce.
The most consequential architectural decision in this book is how much autonomy to hand over. Most systems shipped as agents should have been workflows, and most of the rest needed a human in one specific place.
The decision that determines your reliability ceiling before you write a line of it.
Breaking work into steps a model can finish, and replanning when step three invalidates the plan.
Supervisor, pipeline, and parallel fan-out — and the honest case that one good agent usually beats five.
Approvals, interrupts, and resumability — placing the gate where it catches damage without destroying the point.
Long-running agents that survive a deploy, a crash, and a rate limit, without starting over.
Every technique in every other part is unfalsifiable without this one. Teams that cannot measure quality do not improve it; they change it and hope. This part comes seventh but should be started first.
Why nothing else here works without it, and the smallest version that is still real.
Twenty examples you wrote yourself beat a public benchmark. Where they come from and how to grow them.
Making a model grade a model without fooling yourself — calibration against human labels, and the biases that survive it.
Two different questions, two different instruments, and the gap between them that catches teams out.
CI for a component that legitimately returns different text each run, without a suite that flakes.
Spans, replay, and the recorded state you need to explain a decision a week later.
Capture, triage, and labelling — the loop that makes the system improve rather than drift.
Everything that separates a working demo from a service other people depend on. Nothing here is specific to AI except the ways it is: latency you cannot bound, capacity you do not own, and failure that returns 200 OK.
API, managed endpoint, or your own serving stack — and what each choice forces on everything above.
Streaming, time to first token, speculative UI, and the difference between fast and feeling fast.
Timeouts, fallback models, degraded modes, and designing for the provider incident rather than around it.
Backoff, queueing, and admission control when the constraint is on a vendor's side of the wire.
Exact, semantic, and prefix caching — what each is safe for, and the staleness each one hides.
Version pinning, canaries, and rolling back a prompt — because a model deprecation is a breaking change you did not schedule.
Agents introduce a vulnerability class that has no clean fix: instructions and data arrive on the same channel. Anyone shipping an agent that reads untrusted content and holds real permissions is operating a security system, whether or not they know it.
The defining vulnerability of tool-using systems, why input sanitisation does not solve it, and what actually reduces the blast radius.
Private data, untrusted content, and an exfiltration channel. Any two are survivable; all three is an incident waiting for a trigger.
Capability scoping, irreversible-action gates, and giving a probabilistic system credentials responsibly.
Schema checks, groundedness checks, and refusing to render or execute output you have not verified.
What leaves your boundary, what a vendor keeps, training opt-outs, and reading the terms that govern it.
Logs that answer 'why did it do that' months later, and the record an auditor or a regulator will ask for.
The last resort, not the first. Almost everything attributed to a model limitation is a context, retrieval, or eval problem. This part is for the minority of cases where it genuinely is not — and for doing it properly when it is.
The specific conditions under which training beats prompting, and the far more common ones where it does not.
Collection, cleaning, labelling, and holdouts. The nine tenths of a fine-tune that is not training.
The techniques, what each changes, and what each costs to serve afterwards.
What reinforcement learning from feedback is actually for, and why it is rarely the tool an application team needs.
Proving it improved the target without silently regressing everything else — the suite that keeps a trained model honest.
Worked architectures for the shapes that recur, each with the failure mode that kills it and the eval that catches that failure early.
In-product help that has to be right often enough to be trusted, and what to do on the turns it is not.
Extraction, classification, and enrichment at volume — where accuracy is measurable and the economics are real.
The most mature agent category, why it works there first, and which of its lessons generalise.
The designs that reliably fail, collected in one place so they can be recognised before they are built.