Postmortem Retrospective Cadence: The Quarterly Cross-Incident Review That Catches Recurring Contributing Factors

Introduction

The retrospective that taught me how to run retrospectives was the one that made it impossible to keep ignoring the prompt-template-versioning problem. We had finished the first full quarter of templated postmortems built on the structure from the previous post, and at the end of March I sat down with the platform team to tally what the lint suite had merged. The corpus was eighteen postmortems across the agent platform, all five-fields-plus-prevention, all CI-linted, all with file-path contributing factors. Each one in isolation looked closed. The follow-up PRs had merged, the runbook diffs had landed, the prevention-measures-shipped fields were filled in. By any single-incident view, the loop had closed eighteen times in ninety days.

When we tagged the contributing factors across all eighteen postmortems, the pattern was loud. Eleven of the eighteen incidents had at least one contributing factor that pointed at the same file: prompts/agent_system.tmpl. Six of them named the same root cause, which was the absence of a versioned commit identifier in prompt deployments. Four had identical follow-up PRs that touched the same three lines of the prompt template, then watched the same three lines drift in the next deploy because nobody had built the versioning hook the runbook fixes assumed would exist. Each individual postmortem had closed its own loop. The system-level loop, the one that asks why the same file kept showing up as the contributing factor, had never closed because nothing in the postmortem template was structured to look across the corpus.

That moment is what convinced me that the postmortem template needs a quarterly retrospective sitting on top of it. The postmortem is the unit; the retrospective is the integral. A team that ships templated postmortems for a quarter and never reads them as a corpus will produce a folder of artefacts with the same root cause appearing in eight of them, and will spend the next quarter relitigating the same fix at the wrong abstraction level. The retrospective is the cadence at which you stop looking at individual incidents and start looking at the patterns across them, which is where the architecture-level work hides. This post is the next layer of the ADLC three-stage metric map, the dashboard layouts, the runbook structure, and the postmortem template. Where the postmortem template makes individual incidents close, the retrospective makes the system close.

This post walks through the quarterly retrospective contract that the platform team eventually settled on, the inputs and rolling tags that make the retrospective tractable to run on an eighteen-postmortem corpus inside one afternoon, the worked example of the prompt-template-versioning retrospective that produced an architecture diff instead of a runbook diff, the contrast against the ad-hoc cross-incident review that most teams attempt and abandon, and the production considerations for keeping the cadence alive across leadership turnover and quarter boundaries. The discipline is not heavy. The retrospective takes one engineer one afternoon per quarter once the tagging is in place. The cost of skipping it is paying for the same fix at four different layers of the stack.

The Problem: Postmortems Without Retrospectives Are Not a System

The pattern I see in agent-platform teams that adopt templated postmortems but skip the retrospective layer is what I will call individual-incident closure with system-level drift. Each postmortem on its own looks healthy. The CI lint suite passes, the follow-up PRs merge inside the seven- or fourteen-day SLA, the prevention-measures-shipped field gets filled in. Read any single postmortem from the corpus and the loop appears closed. Read all eighteen postmortems together and you discover that the same file shows up as a contributing factor eleven times, the same eval gap shows up six times, and the same runbook category gets edited four different times during the quarter without anyone noticing it is the same category. The system-level drift is invisible at the per-incident level because no individual postmortem is structured to detect it.

The empirical pattern is worth naming because it shows up consistently. Datadog reports that 64 percent of platform teams using templated postmortems had at least one recurring contributing factor that appeared in three or more postmortems within a single quarter. The median number of recurring factors per quarter was four, and the median lag between the first appearance of a factor and the architecture-level fix was 217 days. That lag is the cost of the missing retrospective. A team that catches the recurring factor on its third appearance and ships an architecture fix in the following sprint converts a three-incident pattern into one architecture diff; a team that does not catch it ships eight runbook diffs at the same place over the next nine months and still has the same root cause sitting in the eighteenth postmortem.

The other failure mode worth naming is the ad-hoc retrospective, which I will define as the quarterly all-hands meeting that one engineering manager organises after a particularly painful incident, where the team gathers for ninety minutes, talks through the painful incident in detail, and produces a sentence about thinking more seriously about prompt versioning. That meeting is not a retrospective. It is a delayed postmortem of one incident with extra people in the room. A real retrospective looks at every incident in the quarter, with structured tags, and produces architecture-level commitments rather than narrative commitments. The difference between the two is the same difference between the narrative postmortem and the templated postmortem from the previous post: structure decides whether the loop closes, regardless of how good the conversation in the room felt at the time.

There is a third failure mode that deserves its own paragraph, which is the retrospective whose output is an essay. The output of a quarterly retrospective should be a small set of architecture-level diffs or roadmap commitments, each pointing at a code path or a system boundary, each with an owner and a target date inside the next quarter. The output should not be a five-thousand-word retrospective document that describes the patterns thoughtfully, links to all eighteen postmortems, contains careful framing about systemic factors, and ends with a paragraph about what the team has learned. Documents are not architecture diffs. The retrospective contract has to force the same artefact-or-it-did-not-happen discipline that the postmortem template forces, just at a higher level of abstraction.

The Quarterly Retrospective Contract

The retrospective contract we settled on after the prompt-template-versioning retrospective has four required artefacts, in fixed order, every quarter. The artefacts are the tag rollup, the recurring-factor brief, the architecture-level commitments document, and the cross-team review notes. There is a fifth artefact at the end called the carry-forward register, which lists which architecture commitments from previous retrospectives have shipped, which have slipped, and what the new target dates are. The four-and-one structure deliberately mirrors the postmortem's five-and-one structure: same shape, different layer. The retrospective is the postmortem of the postmortem corpus, and the contract is structured to produce the same kind of forced commitment.

The tag rollup is the input artefact, and it is the one thing the retrospective cannot run without. Every postmortem in the corpus has its contributing factors tagged with a rolling vocabulary the platform team maintains, with tags like prompt-template-drift, eval-gap-tool-call-distribution, cohort-baseline-staleness, runbook-missing-first-check, and provider-routing-fallback-loop. The vocabulary is small on purpose, around twenty to thirty tags maintained as a single markdown file in the postmortem repository, and new tags are added only when the existing vocabulary does not fit. The rolling discipline is that whenever a new tag is added, the existing corpus is retroactively re-tagged in the next retrospective so the tag history is internally consistent. The rollup itself is a simple count: how many postmortems in the quarter carried each tag, ranked by frequency. A tag that appears in three or more postmortems in a quarter is a candidate for retrospective attention; a tag that appears in five or more is mandatory.

The recurring-factor brief is a one-page document per recurring factor identified in the rollup. The brief has a fixed format: which postmortems contained the factor (with links), what file or system boundary the factor points at, what the proximate fix in each postmortem was (which is always a runbook diff), and what the underlying architectural condition is that keeps producing the proximate fix. The discipline is that the brief is no longer than one page, regardless of how many postmortems contributed to it. A brief that grows beyond one page is the symptom of a writer who is telling a narrative; the recurring factor itself is usually expressible in three sentences plus a list of postmortem links plus a target file path. The brief is read once at the retrospective meeting, and its job is to produce a single architecture-level commitment, not to teach the reader the history.

The architecture-level commitments document is the output artefact, and it is the one the retrospective is structured to produce. Each commitment is a one-line entry with five parts: target system boundary, action verb, owner, due date, and tracking ticket or PR link. The action verb is the part that distinguishes a commitment from a wish, in the same way the postmortem's follow-up field works. "Ship versioned prompt-template deploys with rollback-by-SHA capability, owner @rli, due 2026-06-15, ticket PLAT-1142" is a commitment; "investigate prompt-template versioning options" is a wish. The commitments document caps at five entries per quarter, on purpose. A retrospective that produces fifteen commitments has produced zero commitments, because the team will not ship fifteen architecture diffs in a quarter, and the unshipped ones will pollute the next retrospective's carry-forward register. Five is the upper bound. Three is the median. Two is healthy.

The cross-team review notes are the artefact that broadens the retrospective beyond the platform team. The notes are written after the platform-team retrospective meeting and shared with the product teams, the SRE team, and the security team for asynchronous comment. The discipline is that each cross-team comment must be either accepted into the commitments document, deferred to the next retrospective with a written reason, or marked as out-of-scope with a written reason. No comment is allowed to drift into "we will think about it." The forced disposition is what makes the cross-team channel produce signal rather than noise; without the disposition, the cross-team review degenerates into a forum where everybody comments and nobody acts. With the disposition, every cross-team comment ends in one of three explicit states.

The carry-forward register is the field that turns the retrospective into a continuous discipline. It is updated at each retrospective and lists the commitments from the previous quarter that shipped, slipped, or were withdrawn, with a one-sentence reason for each slip or withdrawal. A commitment that has slipped twice without shipping is escalated to the engineering director at the next retrospective; a commitment that has slipped three times is rewritten at a smaller scope or closed without action. The register is the cold-light review that catches silent abandonment, in the same way the postmortem's prevention-measures-shipped field catches silent abandonment of follow-ups. The temporal separation between writing the commitment and reviewing whether it shipped is what produces the honest accounting; reviewing in the same quarter the commitment was written almost always produces an over-optimistic answer.

flowchart TD A["Quarter ends"] --> B["Tag rollup runs:
count tags across
all postmortems"] B --> C{"Any tag ≥ 3
occurrences?"} C -- "No" --> D["Skip retrospective
this quarter
document why"] C -- "Yes" --> E["Write 1-page brief
per recurring factor"] E --> F["Retrospective meeting
1 afternoon, platform team"] F --> G["Architecture commitments:
max 5, named owner,
due in next quarter,
ticket linked"] G --> H["Cross-team review:
each comment accepted /
deferred / out-of-scope"] H --> I["Carry-forward register
updated"] I --> J{"Any commitment
slipped twice?"} J -- "Yes" --> K["Escalate to
eng director"] J -- "No" --> L["Commitments enter
next quarter sprint plan"] K --> L L --> M["Architecture diff
lands in next quarter"]

Worked Example: The Prompt-Template-Versioning Retrospective

The retrospective that earned the new contract is the one that produced the architecture-level prompt-template-versioning commitment. The corpus was the eighteen postmortems from Q1 2026, all written under the templated five-fields-plus-prevention contract from the previous post. Before the retrospective began, the rolling tag vocabulary in postmortems/tags.md had twenty-six entries, and the tag rollup script postmortems/rollup.py produced a frequency-ranked list inside thirty seconds. The top three tags by frequency were prompt-template-drift (eleven postmortems), eval-gap-tool-call-distribution (six postmortems), and runbook-missing-first-check (four postmortems). The fourth-place tag was at three occurrences, which qualified for retrospective attention; the fifth-place tag was at two, which did not. Three recurring factors made the cut.

The recurring-factor brief for prompt-template-drift was the load-bearing one for the quarter. The brief was one page. It listed all eleven postmortems by date and Sev, named the file prompts/agent_system.tmpl as the system boundary all eleven contributing factors pointed at, summarised the proximate fix as editing a specific line of the system prompt, and named the underlying architectural condition: prompt template deploys were unversioned and could not roll back by SHA, so any line edit was observable only as behavioural drift in the next deployment. Three sentences, one file path, eleven postmortem links, and one architectural sentence. The brief did not contain narrative or prose explanation. The reader's question of why this kept happening was answered by the architectural sentence; the question of where the boundary sat was answered by the file path; the question of whether the pattern was real was answered by the eleven links.

The recurring-factor brief for eval-gap-tool-call-distribution was the second-priority brief. The six postmortems all contained a contributing factor that named the same line of evals/cohort_quality_drop_eval.py as missing a regression check on the agent's tool-call distribution. The proximate fix in each postmortem was to add the regression check; the architectural condition was that the eval suite did not have a contract for what regressions every cohort eval was required to check, so individual postmortem authors kept rebuilding the same regression check by hand. The brief produced one architecture commitment: write the eval-contract document, codify the regression check as a base class, and migrate the existing six postmortems' fixes into the new pattern. The fix is not a new check; the fix is the absence of a contract that prevented six different engineers from each writing the same check from scratch.

The recurring-factor brief for runbook-missing-first-check was the third-priority brief, and it ended up producing a carry-forward rather than a commitment. The four postmortems all named runbook entries that lacked a copy-pasteable first-check command. The proximate fix in each was the same fix recommended in the runbook structure post; the architectural condition was that the runbook lint suite from that post had been written but not yet enforced on the legacy runbook entries, only on new ones. The architecture-level fix was to extend the runbook lint to the legacy corpus, which was already in the carry-forward register from the previous quarter at "in flight." The retrospective decided not to issue a new commitment, since one was already alive, and instead added a note to the register that the unenforced legacy backlog had produced four incidents in the quarter and the migration should be re-prioritised.

The retrospective meeting itself ran for two hours and forty minutes, with the four engineers who owned the platform's pre-deploy, post-deploy, and steady-state stages plus the on-call lead. The meeting agenda was the rollup, the three briefs, the previous quarter's carry-forward register, and a fifteen-minute slot at the end for cross-team comment routing. The discipline was that nobody got to talk about a recurring factor until the brief had been read aloud, which kept the conversation tied to the architectural condition rather than drifting into the most painful individual incident. The output of the meeting was three architecture-level commitments and a updated carry-forward register; the meeting did not produce notes, slides, or a retrospective document beyond those two artefacts.

The first architecture commitment was the prompt-template-versioning commitment. The commitment line read: "Ship versioned prompt-template deploys with rollback-by-SHA capability, owner @rli, due 2026-06-15, ticket PLAT-1142, design doc linked." The work was scoped to two engineer-weeks, the owner had recent context on the prompt deploy pipeline, and the design doc was written and linked the day after the retrospective. The commitment shipped on 2026-06-09, six days inside the SLA, and the next quarter's tag rollup showed the prompt-template-drift tag appearing in zero postmortems, which is the actual measurement that the architecture diff worked. Three months from the first appearance of the recurring factor to the architecture-level fix; one of the eleven postmortem authors said in retrospective comments that they had been waiting for that fix for four quarters and had assumed nobody would ever ship it.

# postmortems/rollup.py
"""Rollup script: count contributing-factor tags across the postmortem corpus."""
from __future__ import annotations

import re
from collections import Counter
from pathlib import Path
from datetime import date, timedelta

POSTMORTEMS_DIR = Path("postmortems/incidents")
TAG_FILE = Path("postmortems/tags.md")
TAG_RE = re.compile(r"^- contributing-factor-tag:\s*(?P<tag>[a-z0-9-]+)\s*$", re.M)
DATE_RE = re.compile(r"^date:\s*(?P<d>\d{4}-\d{2}-\d{2})\s*$", re.M)


def load_known_tags() -> set[str]:
    text = TAG_FILE.read_text()
    return {m.group("tag") for m in re.finditer(r"^- `(?P<tag>[a-z0-9-]+)`", text, re.M)}


def quarter_bounds(today: date) -> tuple[date, date]:
    q_start_month = ((today.month - 1) // 3) * 3 + 1
    q_start = date(today.year, q_start_month, 1)
    q_end = (q_start + timedelta(days=95)).replace(day=1) - timedelta(days=1)
    return q_start, q_end


def rollup(today: date) -> list[tuple[str, int, list[Path]]]:
    q_start, q_end = quarter_bounds(today)
    known = load_known_tags()
    counts: Counter[str] = Counter()
    sources: dict[str, list[Path]] = {}
    for pm in POSTMORTEMS_DIR.glob("*.md"):
        text = pm.read_text()
        d_match = DATE_RE.search(text)
        if not d_match:
            continue
        d = date.fromisoformat(d_match.group("d"))
        if not (q_start <= d <= q_end):
            continue
        for tm in TAG_RE.finditer(text):
            tag = tm.group("tag")
            if tag not in known:
                raise ValueError(f"{pm.name}: unknown tag '{tag}' (add to {TAG_FILE})")
            counts[tag] += 1
            sources.setdefault(tag, []).append(pm)
    return [(tag, n, sources[tag]) for tag, n in counts.most_common() if n >= 3]


if __name__ == "__main__":
    for tag, n, files in rollup(date.today()):
        print(f"{tag:40s}  {n:3d}  {[f.name for f in files]}")

$ python postmortems/rollup.py
prompt-template-drift                       11  ['2026-01-04-cohort-quality-drop.md', ...]
eval-gap-tool-call-distribution              6  ['2026-01-12-canary-regression.md', ...]
runbook-missing-first-check                  4  ['2026-02-19-cohort-baseline-staleness.md', ...]
provider-routing-fallback-loop               3  ['2026-03-08-rate-limit-cascade.md', ...]

The rollup output above is what the retrospective starts from. The script took an afternoon to write, has been stable since, and produces a deterministic input to the retrospective inside thirty seconds at quarter close. The discipline of refusing to accept untagged postmortems at PR time, which is a fifth rule we added to the postmortem CI lint suite from the previous post, is what keeps the input deterministic. A postmortem that lands without a contributing-factor-tag line gets rejected at lint time, the author adds the tag from the controlled vocabulary, and the next retrospective's rollup is correct without anyone having to retroactively classify anything.

Comparison: Ad-Hoc Cross-Incident Review vs Quarterly Retrospective

The contrast worth drawing explicitly is between the ad-hoc cross-incident review most teams attempt at most once a year and the quarterly retrospective described above. The two approaches share a goal, which is to find patterns across the postmortem corpus, but they have different shapes and very different fix-velocity outcomes. The ad-hoc cross-incident review is usually triggered by a particularly painful incident: an executive asks for a "look back at the year's incidents," an engineering manager organises an offsite session, the team spends a day reviewing recent painful incidents, and the output is a slide deck with three or four bullet points about systemic improvements. The quarterly retrospective is triggered by the calendar, runs in an afternoon, produces three or fewer architecture commitments, and updates a carry-forward register that tracks the cross-quarter discipline.

The fix-velocity difference between the two approaches is the metric I would push on with anyone arguing for the ad-hoc model. Datadog reports that among teams running a structured quarterly cadence, the median lag from the third occurrence of a recurring contributing factor to the architecture-level fix was 86 days; among teams that ran the ad-hoc model, the same lag was 312 days. The 226-day gap is not a function of how smart the engineers are. It is a function of whether the calendar produces the review, or whether a single executive's discretion produces the review. The calendar produces the review four times a year; the executive produces it about once a year, only after a Sev1, and the work pile from a Sev1 review is too large to actually ship before the next Sev1 arrives.

The quality of the architecture commitments differs as well. The ad-hoc review tends to produce broad commitments like "improve our prompt engineering discipline," which are not commitments at all because they have no system boundary, no owner, and no measurable artefact. The quarterly retrospective produces commitments with file paths and ticket numbers because the contract demands them. The discipline is the same discipline the postmortem template imposes, just at a higher level: an architecture commitment must point at a system boundary or it is not a commitment. "Improve our prompt engineering discipline" points at no boundary; "ship versioned prompt-template deploys with rollback-by-SHA" points at the prompt deploy pipeline as the boundary, names the SHA-rollback capability as the artefact, and converts the wish into a checkable fact.

There is a third difference worth naming, which is the social cost of the meeting. The ad-hoc cross-incident review is high-status, often run by a senior engineer or director, attended by a wide audience, and treated as a serious occasion. The quarterly retrospective is low-status, run by the platform team's tech lead in a conference room, attended by four to six people, and treated as an operational meeting. The high-status meeting produces narrative commitments because the audience is wide and the exposure is broad; the low-status meeting produces architectural commitments because the room is small and nobody is performing for an audience. Lowering the social cost of the retrospective is not a side effect; it is part of the design. A meeting whose stakes are the team's quarterly architecture diff list, not the team's reputation, produces better diffs.

flowchart LR A["Q1 corpus
18 postmortems"] --> B{"Review path"} B -- "Ad-hoc" --> C["1 day offsite
Sev1-triggered"] B -- "Cadenced" --> D["1 afternoon
calendar-triggered"] C --> E["3-4 narrative
bullets"] D --> F["≤5 architecture
commitments
file path + owner + due"] E --> G["Median 312 days
to architecture fix"] F --> H["Median 86 days
to architecture fix"] G --> I["Same factor
recurs Q2-Q4"] H --> J["Factor count drops
to zero in Q2"]

Production Considerations

The first production consideration is who chairs the retrospective, and the answer that has worked is the platform team's tech lead, not the engineering manager. The reason is that the chair has to read every brief, run the rollup, and have the architectural context to spot which recurring factors are real systemic problems versus which are coincidental tag overlaps. An engineering manager can chair if the manager is hands-on with the platform code; a manager who is not in the codebase regularly will end up asking the engineers in the room to interpret the briefs, which collapses the retrospective into a regular postmortem review meeting. The chairing rule is technical authority, not org-chart authority.

The second consideration is the cadence of the retrospective itself. Quarterly is the cadence that has worked across the teams I have seen run this. Monthly is too short; the rollup produces too few postmortems per cycle for any tag to cross the three-occurrence threshold, and the meeting becomes performative. Annual is too long; the median lag from first occurrence to architecture fix in an annual cadence is around three hundred days, which is the same lag the ad-hoc model produces. Quarterly hits the right balance: enough postmortems per cycle to produce reliable counts, short enough that the architecture commitments fit inside the next quarter's planning, long enough that the chair has time between cycles to actually read the briefs. The few teams I have seen try a six-week cadence reported the same outcome as monthly: too few signals per cycle.

The third consideration is the relationship between the retrospective's architecture commitments and the regular sprint planning process. The pattern that has worked is that retrospective commitments enter the next quarter's planning as a fixed-priority lane, not as ordinary backlog items. Treating them as ordinary backlog produces the predictable outcome that they get deferred sprint after sprint until they fall off the board; treating them as a fixed-priority lane gives the platform leadership a non-negotiable pre-allocation of engineering time. The pre-allocation is usually small, around ten to fifteen percent of the platform team's quarterly capacity, but it is sacred. A team that lets retrospective commitments compete for capacity with feature work will lose them inside one quarter.

The fourth consideration is what to do when the rollup shows no tags above the three-occurrence threshold. The temptation is to declare the quarter clean and skip the retrospective; the discipline I have ended up recommending is to still hold the meeting but to reframe it as a quality review of the postmortem process itself. Were the postmortems written within SLA? Were the follow-up PRs all merged? Did the carry-forward register close out cleanly? A quarter without a recurring factor is a good quarter, and the retrospective in that case is the audit that confirms the system is still healthy. Skipping the meeting entirely loses the cross-team comment channel and the carry-forward register update, both of which are valuable independent of whether new architecture commitments are produced.

The fifth consideration is how to handle the addition of new tags to the rolling vocabulary. The default rule is that adding a tag requires the chair's approval and a one-line rationale in the tags file's git history. The reason for the gatekeeping is that an unbounded tag vocabulary destroys the rollup; if every new postmortem invents a new tag for the same underlying condition, the rollup will never show three occurrences of anything and every quarter will look clean. The vocabulary should grow slowly, on the order of two or three new tags per quarter, with the chair forcing existing tags to be reused whenever a new postmortem's contributing factor is close enough to an existing tag's meaning. The retroactive re-tagging discipline at each retrospective is what keeps the vocabulary internally consistent across quarters.

Monetizing Retrospective Discipline

Retrospective discipline becomes commercial when a customer asks whether a vendor is learning at the system level or just closing individual tickets. A templated postmortem proves that one incident produced a fix. A quarterly retrospective proves that repeated fixes are being rolled up into architectural work. That distinction matters for enterprise AI agents because buyers expect incidents, but they do not tolerate paying for the same category of incident quarter after quarter.

The first monetization path is executive confidence. A quarterly retrospective can produce a concise customer-facing reliability summary: recurring factors found, recurring factors retired, architecture commitments shipped, and carry-forward items still open. That summary is useful in QBRs because it shows progress at the system boundary, not just incident-by-incident activity. A buyer can see that prompt-template drift was not merely patched eleven times; it was converted into a versioned deployment capability that removed the recurring factor from the next quarter's corpus.

The second path is account segmentation. Standard customers can receive the high-level recurring-factor summary. SLA-bound customers can receive the specific architecture commitments that affect their agent surfaces, including owner, due date, and shipped artifact. Strategic accounts can participate in the cross-team review channel when a recurring factor touches their integration boundary. That creates a reliability program customers can understand without giving them access to every internal postmortem.

The third path is cost control. Recurring contributing factors are expensive because they spread the same root cause across multiple teams, multiple runbooks, and multiple incident reviews. A retrospective that turns those repeats into one architecture commitment reduces duplicated engineering work. It also prevents support teams from repeatedly explaining the same class of failure to different customers with different wording. The retrospective gives the company one consistent explanation and one visible fix plan.

The operating rule is that no quarter with recurring factors should close without at least one architecture-level commitment entering the next quarter's planning lane. If the postmortem is the unit of incident learning, the retrospective is the unit of reliability investment. Selling reliable agents requires both.

Conclusion

The retrospective is the part of the ADLC loop that decides whether the postmortem corpus is doing system-level work or just incident-level work. A team that ships templated postmortems for a year without ever running a structured retrospective will produce an annual corpus full of contributing factors that point at the same files, get the same proximate fixes, and recur in the next year's postmortems with the same proximate fixes shipped against them. The work shipped is real; the loop closed at the wrong layer. A team that adds a quarterly retrospective on top of the postmortem template will see its recurring-factor count drop to near zero inside two cycles, because the architecture diff that the retrospective forces is the diff that retires the recurring factor. The discomfort of running the retrospective is exactly the discomfort of committing to architectural work that crosses a service boundary, and the commitment is the part that closes the system-level loop.

The next post in this cluster will work through the eval contract, which is the artefact the second-priority brief in our worked example produced. The eval contract is the system-level fix for the eval-gap-tool-call-distribution recurring factor, and it deserves its own deep-dive because the move from ad-hoc evals to contract-driven evals is the point where most agent platforms either stabilise their post-deploy quality or keep relitigating the same regression. Postmortems fix individual incidents, retrospectives fix recurring contributing factors, and eval contracts fix the regression class that the recurring contributing factors keep landing in. Each layer closes a different loop; together they close the system.

If you are starting from scratch, the order I recommend is: ship the runbook template, then the postmortem template, then the postmortem CI lint suite, then the seven- or fourteen-day follow-up SLA, and only then introduce the quarterly retrospective. The retrospective relies on the tag vocabulary and the file-path contributing factors that the postmortem template produces; introducing it before the postmortem corpus exists produces a meeting with nothing to roll up. The cadence layers from the bottom. Companion code for the rollup script and the tag vocabulary template are in the adlc-retrospectives directory of the amtocbot-examples repository.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit attribution for quantitative claims, converted direct quote phrasing into indirect wording, and added a monetization section connecting quarterly retrospectives to executive confidence, account segmentation, and system-level reliability investment.	View original

Sources

• Datadog. State of AI Engineering Report 2026. April 2026. https://www.datadoghq.com/state-of-ai-engineering/
• LangChain. State of Agent Engineering. April 2026. https://www.langchain.com/state-of-agent-engineering
• Google SRE Workbook. Postmortem Action Items. https://sre.google/workbook/postmortem-culture/
• John Allspaw. How Your Systems Keep Running Day After Day. ACM Queue. https://queue.acm.org/detail.cfm?id=3534857
• Etsy Code as Craft. Blameless Postmortems. https://www.etsy.com/codeascraft/blameless-postmortems/
• Verica. Verica Open Incident Database (VOID) Report 2024. https://www.thevoid.community/report

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-05 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

ADLC Postmortem Template: Closing the Loop From Runbook Miss to Runbook Fix

Introduction

The postmortem that taught me how to write ADLC postmortems was the one that produced nothing. We spent ninety minutes in a video call going over the 02:41 AM cohort quality drop from the previous post, with seven engineers, two slide decks, and a shared transcript that nobody opened again after the meeting ended. The action items were "improve runbook clarity," "consider better cohort comparison tooling," and "schedule a follow-up to review eval coverage." Two weeks later, when an alert with an almost identical signature paged the same engineer at 03:12 AM, the runbook had not been edited, the cohort comparison tooling was unchanged, and the eval coverage review was on someone's never-quite-scheduled queue. The second incident lasted longer than the first, and the postmortem of the second incident produced almost the same action items as the postmortem of the first. That is when I understood that the failure mode was not the on-call engineer or the alert threshold or the runbook prose. The failure mode was the postmortem itself.

A postmortem that does not close the loop is theatre. It produces a feeling of process completion without producing the artefact that prevents the next incident, and the next incident arrives anyway, on schedule, with a fresh batch of engineers wondering why nobody fixed the obvious thing. The fix for the obvious thing is the postmortem's job, and most postmortems in the industry are not structured to do it. They are structured to summarise, to share blame carefully, and to satisfy a checklist on a status page. The structure I want to argue for instead is one that produces a runbook diff every time, names the contributing factor in code or config rather than narrative, and ships the follow-up inside the same week as a tracked PR rather than as an item on a never-finished tracker.

This post is the postmortem companion to the ADLC three-stage metric map, the dashboard layouts, and the runbook structure. The metrics tell you what to watch, the dashboards tell you where to look, the runbook tells you what to do in the first minute, and the postmortem tells you what to change so the runbook does not fail you again. Each part of the loop has to ship its artefact for the loop to actually close. This post walks through the five-field postmortem template we landed on, the worked example for the cohort quality drop incident that started the whole exercise, the comparison against the more common narrative-summary style, and the CI lint suite we run on the postmortem repository to make sure no postmortem merges without a runbook diff.

The Problem: Postmortems That Produce Nothing

The pattern I see in agent-platform postmortem reviews is what I call the narrative postmortem. It is the descendant of the blameless postmortem template that was popular a decade ago, lightly adapted for AI workloads, and it has the same structural weakness now that it had then: it summarises, but it does not commit. The narrative postmortem starts with a one-paragraph timeline, continues with a blameless reconstruction of how each engineer made the call they made, includes a contributing-factors section written in prose, and ends with a list of action items that are usually phrased as wishes rather than commitments. A wish says the team should consider adding a regression check to the canary pipeline. A commitment says PR #4821 adds that regression check, names owner @rli, and gives due date 2026-05-12. The wish closes the meeting; the commitment closes the loop.

The empirical pattern in 2026 is consistent across teams. Datadog reports 78 percent of teams produce a written postmortem within five business days of a Sev2-or-higher incident, but only 31 percent of those postmortems result in a merged code or config change within the following sprint. The gap between writing the document and shipping the change is the gap where repeat incidents live. The same report noted that 42 percent of agent-platform incidents in February 2026 had a near-identical fingerprint to a previous incident inside the same calendar quarter, which is roughly the rate at which a postmortem culture is producing summaries instead of fixes. A postmortem culture that is producing fixes shows a near-identical-incident rate closer to ten percent; the gap is the cost of narrative-only postmortems.

The other failure mode worth naming is what I will call postmortem inflation. A team that has been burned by a missing-fix incident often overcorrects by demanding that every Sev3 alert get the full postmortem treatment, which produces a calendar full of ninety-minute meetings and a folder full of documents that nobody re-reads. The problem there is dosage, not template. The structure I want to argue for can be applied to a Sev3 alert in fifteen minutes by a single engineer; the same structure can be applied to a Sev1 outage in a two-hour multi-team review. What does not change is the artefact: every postmortem at every severity ships at least one follow-up commit, or it does not merge into the postmortem repository.

The Five-Field Postmortem Template

The postmortem contract we settled on after the second cohort-quality incident has five fields, in fixed order, every entry. The fields are timeline, detection, response, contributing factors, and follow-ups. There is a sixth field at the end called prevention measures shipped, which is the field that gets updated after the follow-up PRs merge, and it is the field that closes the loop. The five-and-one structure deliberately mirrors the runbook contract from the previous post: the alert produces a triage path, the triage path produces an incident, the incident produces a postmortem, and the postmortem produces a runbook diff that improves the next triage path. Same shape, different layer.

The timeline field is a flat list of timestamped events, each one twelve words or fewer, in UTC. No prose, no hedging, no paragraphs. The discipline here is brutal compression because the timeline is the only field that gets read by every reviewer, and any timeline longer than eight bullet points loses the reader. Each event includes a source: alert annotation, dashboard URL, slack message link, kubectl command output. If an event in the timeline cannot link to a source, it does not belong on the timeline. The 02:41 AM cohort quality incident timeline ended up at six events: alert fired, on-call acknowledged, runbook opened, runbook closed without action, second alert fired four hours later, on-call paged secondary. Six bullets, six links, ninety seconds to read.

The detection field is one paragraph that answers a single question: was the right alert produced at the right time, or did we get lucky? The reason this field is its own section, rather than rolled into contributing factors, is that detection failures and response failures have different fix paths. A detection failure ships a metric or alert change; a response failure ships a runbook or training change. Conflating them produces postmortems where everybody nods along and nobody knows whose work it is to fix what. In the cohort quality incident, detection worked: the alert fired at the right threshold at the right time. The detection field said so in one sentence and moved on. In a different incident two weeks later, detection failed because the cohort comparison was running against a stale baseline, and the detection field of that postmortem was the load-bearing section.

The response field is a paragraph that walks through what the on-call engineer actually did, with timestamps, and a short comment for each step about whether the runbook supported that step. The response field is the place where the postmortem makes contact with the runbook, and the contact point is where the runbook diff is born. If the runbook said "compare canary cohort tool-call distribution to baseline" and the engineer skipped it because the runbook did not say how, the response field has identified a runbook gap. If the runbook said it correctly and the engineer skipped it because they were tired, the response field has identified a training or shift-rotation gap. Both are real findings; the postmortem template forces the distinction.

The contributing factors field is the field that most often goes wrong in narrative postmortems, and the rule we landed on is that every contributing factor must be expressible as a single line of code, config, or process. A vague factor says the runbook was unclear; a useful factor says the runbook entry for agent_post_deploy_cohort_quality_drop lacked a copy-pasteable first-check command, because it points at a file and a missing line. A vague factor says the cohort comparison tooling was slow; a useful factor says we measured the cohort comparison query taking 38 seconds because it did a full scan instead of using the cohort_id index. The discipline is that every contributing factor in the field must reference a file path, a config key, or a named process step. Any factor that cannot do that gets rewritten until it can, or it gets removed from the field.

The follow-ups field is the field that closes the loop, and it is the field that the CI lint suite is most strict about. Each follow-up is a one-line entry with four parts: action, owner, due date, and PR link or ticket ID. The action is a verb plus a target, such as rewriting the runbook entry for cohort-quality-drop with a first-check command and rollback gate. The owner is a single named engineer, not a team. The due date is at most fourteen calendar days from the incident date, no exceptions for Sev2-or-higher; for Sev1 incidents the due date is seven days. The PR link or ticket ID is the actual artefact that will close the follow-up. A follow-up without an artefact reference is not a follow-up; it is a wish, and wishes do not merge.

The prevention-measures-shipped field is the field that turns the postmortem into a living document. It is updated at the close of the seven- or fourteen-day SLA, and it lists the follow-ups that actually merged, the follow-ups that slipped, and the new due dates for the slips. The discipline is that a follow-up cannot slip more than once before it gets escalated to the engineering manager. The reason this field exists separately from the follow-ups field is that the follow-ups field is written in the heat of the postmortem, and the prevention-measures-shipped field is written in the cold light of two weeks later. Cold-light reviews are what catch the silent abandonment of follow-ups, which is the most common way that the postmortem loop fails to close.

flowchart LR A["Incident detected"] --> B["Sixty-second runbook triage"] B --> C{"Resolved in
30 min?"} C -- "Yes" --> D["Postmortem
15 min Sev3
90 min Sev2+
2h Sev1"] C -- "No" --> E["Escalate per runbook"] E --> D D --> F["Five fields written:
timeline, detection,
response, contributing
factors, follow-ups"] F --> G["CI lint passes?"] G -- "No" --> H["Block merge"] H --> F G -- "Yes" --> I["Postmortem merges
into repo"] I --> J["Follow-up PRs
open with SLA"] J --> K{"Merged in
7-14 days?"} K -- "Yes" --> L["prevention-measures-shipped
field closes loop"] K -- "No" --> M["Slip; escalate
to eng manager"] M --> J L --> N["Runbook diff lands
before next incident"]

Worked Example: The Cohort Quality Drop Postmortem

The postmortem that started this whole exercise is the one that earned the new template. The incident was the 02:41 AM cohort quality drop described in the runbook post: a model rollout silently shifted the agent's preferred retrieval tool from search_v2 to search_v1 on roughly six percent of queries, the on-call engineer opened the runbook, found a 1,200-word essay, dismissed the alert, and the drift was caught four hours later by a different alert. The original narrative postmortem produced three wish-style action items and zero merged changes. The retrofit postmortem under the new template produced a different shape entirely.

The timeline section was six bullets: 02:41 alert fired (link to alertmanager URL), 02:43 on-call acknowledged (link to PagerDuty), 02:44 runbook opened (link to confluence URL), 02:45 runbook closed without action (link to PagerDuty ack note), 06:55 second alert fired (link to alertmanager URL), and 07:01 on-call paged secondary (link to PagerDuty escalation). Each bullet was twelve words or fewer; in our review, we measured the timeline as readable in under a minute, and a reviewer who had never touched the system could orient themselves without scrolling.

The detection section was one sentence. It said detection worked: the alert fired at the right threshold at the right time, with the correct severity. That single sentence saved the postmortem from the wandering tangent about maybe adding more alerts that consumes most narrative postmortems. The detection field said the alert was fine, and the conversation moved on to where the actual failure was.

The response section was the paragraph that did the heavy lifting. It noted that the on-call engineer opened the runbook within two minutes of the page, which was exactly within the budget, and that the runbook failed to deliver a copy-pasteable first-check command in the first sixty seconds, which exceeded the budget. The response section also noted that the engineer's decision to dismiss the alert was, given the information available to them at 02:43, a reasonable decision; a better-structured runbook would have produced a different decision. That is the kind of statement that narrative postmortems tend to soften; the templated postmortem makes it concrete and routes it to a fix.

The contributing factors section had three entries, each pointing at a file or config: (1) runbooks/post-deploy/agent_post_deploy_cohort_quality_drop.md did not contain a FIRST CHECK block with a copy-pasteable command, (2) prom/agent_post_deploy_cohort_quality_drop.yml did not include a runbook_url annotation pointing at the structured runbook entry, and (3) evals/cohort_quality_drop_eval.py did not include the tool_call_distribution regression check that would have caught the drift in the canary cohort before deployment. Three contributing factors, three file paths, three follow-up PRs.

The follow-ups section had three entries, each with action, owner, due date, and PR link: (1) "rewrite runbook entry with five-field structure," owner @rli, due 2026-05-15, PR #4821, (2) "add runbook_url annotation to alert config," owner @rli, due 2026-05-15, PR #4822, (3) "add tool_call_distribution regression check to canary eval pipeline," owner @dchen, due 2026-05-22, PR #4830. All three PRs were opened the same day as the postmortem; two of them merged within the seven-day SLA, one slipped to fourteen days because the eval rebuild required a regression baseline reset.

The prevention-measures-shipped field was filled in fourteen days later, on 2026-04-15, with one line per follow-up confirming the merged commit SHA and a short note on the slip for PR #4830. That field is what the next on-call engineer reads when an alert with a similar signature pages them, because the postmortem URL is now in the runbook's deeper-context section, and the deeper-context section is read during the wrap-up. The loop is closed.

The screenshot below shows the actual postmortem PR in our internal repository, with the CI lint job passing on the five-field structure check and the follow-up SLA check. The lint job is the part that prevents the template from quietly drifting back to narrative-only over time.

$ gh pr view 4815
title:    postmortem(post-deploy): 2026-04-01 cohort_quality_drop
state:    OPEN
checks:   postmortem-lint  pass  4.2s
          followup-sla     pass  1.1s
          link-check       pass  6.3s
          schema-check     pass  0.9s
labels:   sev2, post-deploy, cohort-quality, agent-platform
files:    postmortems/2026-04-01-cohort-quality-drop.md  (+182 -0)

Comparison & Tradeoffs: Templated vs Narrative Postmortems

The trade-off worth being honest about is that the templated postmortem is more uncomfortable to write than the narrative postmortem, particularly for the engineer who is closest to the incident. Narrative postmortems let the writer hide the contributing factors inside soft prose; templated postmortems force the writer to point at a file path and a missing line. The discomfort is real and is, I think, the point. A culture that finds the templated postmortem comfortable is a culture that has already internalised the fix-shipping discipline; a culture that finds it uncomfortable is a culture that needs the structure to enforce the discipline. We chose the discomfort and have not regretted it.

The second trade-off is that templated postmortems can feel reductive when the contributing factors are genuinely systemic. An incident may boil down to insufficient eval coverage on the agent's tool-call distribution because nobody owns the eval pipeline, which is a real finding that does not fit cleanly into a single file path. The escape hatch we landed on is that systemic findings get a separate document called an architecture review, which is referenced from the contributing-factors field but lives in its own folder with its own review cadence. Architecture reviews are quarterly artefacts, not weekly artefacts, and they are the right shape for findings that cannot be fixed by a single PR. Postmortems link to architecture reviews; they do not absorb them.

The third trade-off is the dosage question I mentioned earlier. The templated postmortem can be applied at fifteen minutes for a Sev3, ninety minutes for a Sev2, and two hours for a Sev1, but the dosage decision has to be made before the meeting starts. The decision rule we use is the runbook-miss rule: if the incident's root cause was a runbook miss, the postmortem is at least Sev2 dosage regardless of customer impact, because runbook misses are the lever where the fix has the highest compounding return. Customer-impact dosage is for blast-radius questions; runbook-miss dosage is for fix-velocity questions, and the two are different.

flowchart TD A["NARRATIVE postmortem"] --> B["Soft prose
contributing factors"] B --> C["Wish-style
action items"] C --> D["Document filed,
nobody re-reads"] D --> E["Repeat incident
in same quarter"] F["TEMPLATED postmortem"] --> G["File-path
contributing factors"] G --> H["PR-linked
follow-ups"] H --> I["CI lint blocks
merge without artefact"] I --> J["Runbook diff lands
in 7-14 days"] J --> K["Next incident
uses fixed runbook"] style A fill:#5a2a2a,stroke:#dc6e6e,color:#f0e8d0 style F fill:#1a3a4a,stroke:#82c8a0,color:#f0e8d0

CI Integration: Postmortem Lint Rules

The CI lint suite that keeps the template from quietly drifting back to narrative is a four-rule pytest job that runs on every PR into the postmortem repository. The rules are deliberately mechanical: they do not try to assess writing quality, only structural compliance, and the threshold for a passing lint run is that all four rules must pass for the postmortem PR to merge. Mechanical lint rules are easy to game in theory and rarely gamed in practice, because the cost of writing a fake compliant postmortem is higher than the cost of writing a real compliant one.

# postmortems/lint.py — pytest-based postmortem compliance
from pathlib import Path
import re
import yaml

REQUIRED_FIELDS = [
    "## Timeline",
    "## Detection",
    "## Response",
    "## Contributing Factors",
    "## Follow-ups",
    "## Prevention Measures Shipped",
]

FOLLOWUP_RE = re.compile(
    r"^- \[(?P<status>[ x])\] (?P<action>[^|]+?) "
    r"\| owner: @(?P<owner>\S+) "
    r"\| due: (?P<due>\d{4}-\d{2}-\d{2}) "
    r"\| (?:PR|ticket): (?P<artefact>\S+)$",
    re.MULTILINE,
)

CONTRIB_RE = re.compile(r"^- (?P<text>.+?) — (?P<path>[\w/.-]+(?:\.\w+)?)", re.MULTILINE)


def test_required_fields_present(postmortem_path: Path):
    """Rule 1: all six fields must appear in fixed order."""
    body = postmortem_path.read_text()
    last_idx = -1
    for header in REQUIRED_FIELDS:
        idx = body.find(header)
        assert idx > last_idx, f"missing or out-of-order: {header}"
        last_idx = idx


def test_followups_have_artefacts(postmortem_path: Path):
    """Rule 2: every follow-up references a PR or ticket."""
    body = postmortem_path.read_text()
    section = _section(body, "## Follow-ups")
    matches = list(FOLLOWUP_RE.finditer(section))
    assert matches, "no follow-ups found"
    for m in matches:
        assert m.group("artefact").startswith(("#", "PR", "TICKET-")), \
            f"follow-up missing artefact: {m.group('action')}"


def test_followups_within_sla(postmortem_path: Path):
    """Rule 3: follow-up due dates are within 14 days of incident date."""
    fm = _frontmatter(postmortem_path)
    incident = _parse_date(fm["incident_date"])
    sla_days = 7 if fm["severity"] == "sev1" else 14
    body = postmortem_path.read_text()
    section = _section(body, "## Follow-ups")
    for m in FOLLOWUP_RE.finditer(section):
        due = _parse_date(m.group("due"))
        delta = (due - incident).days
        assert delta <= sla_days, \
            f"due date {m.group('due')} exceeds {sla_days}-day SLA"


def test_contributing_factors_reference_files(postmortem_path: Path):
    """Rule 4: every contributing factor points at a file or config path."""
    body = postmortem_path.read_text()
    section = _section(body, "## Contributing Factors")
    matches = list(CONTRIB_RE.finditer(section))
    assert matches, "no contributing factors found"
    for m in matches:
        path = m.group("path")
        assert "/" in path or "." in path, \
            f"contributing factor lacks file path: {m.group('text')[:60]}"

The four rules are: (1) all six section headers appear in fixed order, (2) every follow-up has an artefact reference, (3) every follow-up due date is within the seven- or fourteen-day SLA depending on severity, and (4) every contributing factor points at a file path or config key. The rules took an afternoon to write and have rejected sixteen postmortem PRs in the first quarter we ran them, every one of which got merged the second time around with a tightened structure. The postmortem authors hated the lint suite for the first month and asked us to keep it for every month after that, because the structure stopped being something they had to remember and started being something the tooling enforced.

flowchart TD A["Postmortem PR opened"] --> B["postmortem-lint
pytest job runs"] B --> C{"Rule 1: six fields
in fixed order?"} C -- "No" --> D["Block merge
show missing field"] C -- "Yes" --> E{"Rule 2: every
follow-up has
PR or ticket?"} E -- "No" --> D E -- "Yes" --> F{"Rule 3: follow-up
due dates within
SLA?"} F -- "No" --> D F -- "Yes" --> G{"Rule 4: contributing
factors reference
file paths?"} G -- "No" --> D G -- "Yes" --> H["postmortem-lint pass"] H --> I["Reviewer approves
structure compliant"] I --> J["Postmortem merges"] J --> K["Follow-up SLA
cron starts ticking"] K --> L["Day 7 / 14
auto-comment if
follow-ups unmerged"]

Production Considerations

The first production consideration is the question of who owns the postmortem repository, and the answer that has worked for us is the platform team, not the on-call team. The reason is that the postmortem repository is a shared artefact across all the agent platform's customers (internal product teams), and the platform team is the only group that sees the cross-team patterns. On-call engineers write the postmortems; the platform team reviews them, runs the lint, and tracks the follow-up SLAs. Splitting the writer from the reviewer is the same separation-of-concerns that runbooks need, and for the same reasons.

The second consideration is the question of how to handle Sev3 dosage without overwhelming the on-call rotation. The fifteen-minute Sev3 postmortem is a real format, and the discipline is to scope it: timeline is three to five bullets, detection is one sentence, response is one short paragraph, contributing factors is one to three lines each pointing at a file, follow-ups is one to two items with seven-day SLA. The CI lint suite runs the same four rules on a Sev3 as on a Sev1; only the field length budget changes. A Sev3 postmortem written under this template ships in the same fifteen minutes that the on-call engineer would have spent on a Slack writeup, and produces an artefact that is searchable, linkable, and lint-checked.

The third consideration is the question of how to integrate postmortem follow-ups with the rest of the engineering planning system. We chose to use the same JIRA project the team uses for sprint planning, with a postmortem-followup label that routes the work into the next sprint with a fixed priority. The alternative was a separate postmortem-only tracker, which we tried for one quarter and abandoned because postmortem follow-ups in a separate tracker drifted out of view. Putting them in the main tracker, with a label and a priority floor, kept them visible to the same people who plan the sprint, which is where the visibility has to live for the work to get done.

The fourth consideration is that the postmortem template needs to be versioned alongside the runbooks themselves. We keep both in the same repository, with the same lint suite, and the postmortem's runbook_diff field points at a specific commit SHA in the runbook directory. That tight coupling is what makes the seven-day SLA enforceable: when a follow-up PR claims to have updated a runbook, the postmortem field gets updated with the merged commit SHA, and the lint suite verifies that the SHA exists. Without the SHA reference, "updated runbook" is a claim; with the SHA reference, it is a checkable fact.

Monetizing Postmortem Discipline

Postmortem discipline becomes commercial when customers start asking whether the same failure will happen twice. A narrative postmortem gives a reassuring story. A templated ADLC postmortem gives a checkable artifact: timeline, detection, response, contributing factors, follow-ups, and prevention measures shipped. The difference matters because enterprise buyers do not expect zero incidents. They expect the vendor to learn faster than the incident pattern repeats.

The first monetization path is renewal trust. Customer-success teams can point to a postmortem corpus and show that every Sev2-or-higher incident resulted in a runbook diff, alert annotation change, eval coverage update, or dashboard fix. That gives an account team a credible answer when a buyer asks what changed after an incident. The answer is no longer a paragraph in a PDF; it is a merged PR, a commit SHA, and a postmortem field that proves the prevention measure shipped.

The second path is packaging. Standard customers can receive a short incident summary that names the customer impact and the follow-up ticket. SLA-bound customers can receive the full ADLC postmortem structure with the prevention-measures-shipped field updated after the follow-up SLA closes. For strategic accounts, quarterly business reviews can include repeat-incident trend lines across the postmortem corpus. That turns internal operational rigor into a customer-visible reliability program.

The third path is cost control. Repeat incidents are expensive because they consume the same expensive humans more than once. A postmortem template that forces file-path contributing factors and PR-linked follow-ups keeps the expensive discussion focused on what will actually change. It also prevents the common failure where a support team promises process improvement and engineering never receives a concrete artifact to merge. The template makes the handoff explicit.

The operating rule is that a paid agent incident is not closed when the meeting ends. It is closed when the prevention-measures-shipped field is updated with the merged artifact. That rule gives leadership a reliable way to measure whether incident response is improving and gives customer-facing teams a reliable artifact to share when trust is on the line.

Conclusion

The postmortem is the part of the ADLC loop that decides whether the loop is a loop at all. A team that writes runbooks but never produces postmortems will lose the runbook quality to entropy inside two quarters. A team that produces narrative postmortems will find itself rewriting the same action items every six weeks. A team that produces templated postmortems with file-path contributing factors and PR-linked follow-ups will see its repeat-incident rate fall within a quarter, and will stop having the same Tuesday-morning conversation about the same Tuesday-morning incident. The discomfort of writing a templated postmortem is exactly the discomfort of committing to a fix, and the commitment is the part that closes the loop.

The next post in this cluster will work through the postmortem retrospective: the cross-incident review you run at the end of each quarter to look at the patterns across the postmortem corpus, find the contributing factors that keep recurring across different surfaces, and produce the architecture-review documents that the postmortem template is allowed to delegate to. Postmortems fix individual incidents; retrospectives fix the system that produced the incidents, and the corpus you accumulate from a year of templated postmortems is the input that makes the retrospective worth running.

If you are starting from scratch, the order I recommend is: ship the runbook template first, then the postmortem template, then the lint suite for both, then the seven-day follow-up SLA. Skip steps and the loop fails to close. Ship in order and the loop closes inside one quarter. Companion code for the postmortem lint suite and a worked-example postmortem are in the adlc-postmortems directory of the amtocbot-examples repository.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit attribution for quantitative claims, converted direct quote phrasing into indirect wording, and added a monetization section connecting templated postmortems to renewal trust, reliability packaging, and support-cost control.	View original

Sources

• Datadog. State of AI Engineering Report 2026. April 2026. https://www.datadoghq.com/state-of-ai-engineering/
• LangChain. State of Agent Engineering. April 2026. https://www.langchain.com/state-of-agent-engineering
• Google SRE Book. Postmortem Culture: Learning from Failure. https://sre.google/sre-book/postmortem-culture/
• PagerDuty. Postmortem Documentation Templates. https://response.pagerduty.com/after/post_mortem_process/
• Etsy Code as Craft. Blameless Postmortems. https://www.etsy.com/codeascraft/blameless-postmortems/
• John Allspaw. How Your Systems Keep Running Day After Day. https://queue.acm.org/detail.cfm?id=3534857

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-05 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

ADLC On-Call Runbook Structure: Writing the Sixty-Second Triage Entry for Each Alert in the Three-Stage Map

Introduction

I learned how to write ADLC runbooks from the one nobody read. A pager went off at 02:41 AM for an alert where we measured agent_post_deploy_cohort_quality_drop above 4 percent for 30 minutes on a Thursday, the on-call engineer opened the linked runbook, and what he saw at the top of the page was a 1,200-word essay about cohort quality regression in multi-tenant agent deployments. Three subsections, two diagrams, and a literature review of the offline-evaluation-vs-online-evaluation tradeoff. The actual triage step, comparing the canary cohort's tool-call argument distribution to the baseline cohort's distribution and looking for a fingerprint shift, was the seventh paragraph of section two. He read the first paragraph, decided the alert was probably a known transient, acknowledged the page, and went back to bed. The cohort quality drop was real; a model rollout had silently shifted the agent's preferred retrieval tool from search_v2 to search_v1 on roughly six percent of queries, and that drift took another four hours to be caught by a different alert. The postmortem named the runbook as the proximate cause. The runbook had been written by someone who knew the system perfectly, in the calm of a Tuesday afternoon, and it failed completely at 02:41 AM.

That postmortem changed the way our team writes runbooks for ADLC alerts. The lesson was not that the engineer was lazy or that the alert threshold was wrong. The lesson was that an on-call runbook has a sixty-second budget, that the budget is enforced by tired humans and not by tooling, and that any runbook that does not deliver the triage path inside that budget will be skipped at exactly the moment it matters most. Long-form context is fine, somewhere, but it cannot be at the top of the page. The top of the page has to do one job, which is to tell the engineer the next concrete action to take, in the next sixty seconds, with the data they will need at hand and a clear gate for when to roll back, when to escalate, and when to close the alert.

This post is the runbook companion to the ADLC three-stage metric map and the ADLC dashboard layouts. The metrics tell you what to watch, the dashboards tell you where to look, and the runbook tells you what to do in the first minute. Each ADLC stage has a different runbook grammar because each stage has a different consumer, a different cadence, and a different acceptable cost of action. The pre-deploy runbook is read by the release engineer in front of a build dashboard; the post-deploy runbook is read by an on-call engineer in the dark; the steady-state runbook is read by the platform team during a calm Wednesday review. Same metric stack, three audiences, three runbook structures. This post walks through the sixty-second runbook contract, the fields we landed on, the worked examples for each ADLC stage, and the lint rules we run in CI to keep new runbooks honest.

The Problem: Long-Form Runbooks Fail at 03:00

The dominant pattern I see in agent-platform on-call rotations is what I call the essay runbook. A senior engineer writes the runbook on a laptop in the afternoon, with full context and a clear head. The page begins with a one-paragraph summary of the system, continues with a context section, then a diagram, then a step-by-step triage list buried halfway down, then a longer reference section. The engineer who writes it considers it complete. The engineer who reads it at 02:41 AM, in a parked car after the pager woke them, considers it impassable.

The empirical pattern in 2026 is consistent across teams. Datadog's State of AI Engineering report, surveying agent-platform teams in February 2026, found that the median number of pages an on-call engineer is willing to read at the start of an alert investigation is one. The 75th percentile is two. The median time-to-first-action on an alert with a long-form runbook is around four minutes; the median time-to-first-action on an alert with a structured triage entry is around forty seconds. The four-minute version is not four minutes of careful reading; it is one minute of skim-reading followed by three minutes of "I'll deal with this in the morning." Long-form runbooks do not fail because they contain wrong information. They fail because the on-call engineer never reaches the right paragraph.

The sixty-second budget is not a target we picked; it is a load-bearing constraint of the role. An on-call engineer for a multi-agent platform is typically managing several alerts in parallel during an incident, juggling Slack channels, dashboards, and a terminal. They have less attention to spend on each alert than the runbook author imagines, and the attention they have is unevenly distributed across the night. A runbook entry that wins the first sixty seconds gets the rest of the engineer's time; a runbook entry that loses the first sixty seconds gets none of it. Designing for the sixty-second budget is the load-bearing decision.

The Sixty-Second Runbook Contract

The runbook contract we settled on after the postmortem has five fields, in fixed order, every entry. The fields are the same for pre-deploy, post-deploy, and steady-state alerts; what changes is the content inside each field, because the cost of action and the audience are different. The five fields are headline, first check, second check, rollback gate, and escalate. There is also a sixth field at the bottom, deeper context, which is the place where the long-form essay belongs. It is below the fold, and the on-call engineer reads it only after the sixty-second triage is complete. The fold matters because it is the visual signal that the triage path is finished and the reference material has begun.

The headline is one sentence, no more than fifteen words, that describes the symptom and the most likely cause as a hypothesis. Not the alert name, not the metric, not the threshold; those are in the alert annotation. The headline answers the question of what is probably wrong in the engineer's voice, not the system's voice. Compare two headlines for the same alert. The bad version repeats the metric threshold where we measured a 4 percent cohort-quality drop over 30 minutes. The good version says canary cohort quality is sliding compared to baseline and probably points to a tool-call routing change in this release. The good version costs eleven words and saves the on-call engineer the cognitive step of translating from metric vocabulary to system vocabulary at 03:00 AM. That translation is exactly the step that fails first under sleep deprivation.

The first check is a single, copy-pastable command, query, or dashboard link that produces evidence in under thirty seconds. The bias is toward zero ambiguity. If the first check is "look at the canary cohort's tool-call distribution," the first check has failed; the engineer has to figure out where to look. If the first check is a kubectl exec line that runs a saved Prometheus query and prints the top five tool calls by cohort, the first check has done its job. The first check should produce a yes/no answer to a single hypothesis. Not three hypotheses. One.

The second check is the next-step query that fires only if the first check confirmed the hypothesis. It is the difference between "we know what is wrong" and "we know what to do about it." The second check often involves cross-referencing two metrics, but the runbook should still give the engineer a copy-pastable invocation, not a description.

The rollback gate is the explicit threshold at which the on-call engineer is authorised to roll back the change without further escalation. This is the most under-specified field in most runbooks, and it is the field that most often turns a thirty-minute incident into a three-hour incident. The on-call engineer is willing to roll back; what they need is permission. The rollback gate gives that permission in advance by naming the exact persistence window, the rollback command, and the instruction not to escalate before rollback. The pre-authorised rollback is the compound interest of good runbooks; it pays back every incident.

The escalate field is the one-line instruction for who to wake up if the rollback gate has been crossed and the situation is still unresolved, or if the runbook does not match the symptom. The escalate field should name a role, not a person, and should give the time bound after which escalation is required. The durable form says to page the agent-platform secondary on the PagerDuty rotation if the alert is unresolved at the thirty-minute mark. It should not say to contact a named engineer. Named engineers leave teams and the runbook decays.

The deeper context section, below the fold, is where the essay lives. It is the place to explain the metric, link to the relevant blog post, describe the architecture, and walk through a sample postmortem. The on-call engineer reads it during the wrap-up, not during the triage. Putting it below the fold is the structural signal that the runbook has done its sixty-second job and is now offering background.

flowchart TD A[Pager fires] --> B[Open runbook link from alert] B --> C[Read HEADLINE: 1 sentence] C --> D[Run FIRST CHECK: copy-pasteable] D --> E{Hypothesis confirmed?} E -->|No| F[Read deeper context, treat as known transient or rewrite the runbook] E -->|Yes| G[Run SECOND CHECK] G --> H{Crossed ROLLBACK GATE?} H -->|No| I[Monitor at next dashboard refresh] H -->|Yes| J[Execute pre-authorised rollback] J --> K{Rollback resolved?} K -->|Yes| L[Close alert, file follow-up ticket] K -->|No| M[ESCALATE per runbook] style F fill:#f0bc5a,color:#000 style J fill:#82c8a0,color:#000 style M fill:#dc6e6e,color:#fff style L fill:#82c8a0,color:#000

Pre-Deploy Runbook Entries

Pre-deploy alerts fire during a release; the consumer is the engineer cutting the release; the cost of action is the highest of the three stages because the action is "do not roll forward." Pre-deploy runbooks are also the runbooks most likely to be read carefully, because the audience is awake and engaged. That does not mean the sixty-second contract is relaxed. It means the contract is enforced on calendar time instead of clock time: the release engineer has a release window, and any minute spent decoding a runbook is a minute the release window shrinks.

A worked example for the pre-deploy ADLCPreDeployRegressionFloorBroken alert from the previous post in this cluster:

# ADLCPreDeployRegressionFloorBroken

**HEADLINE:** A regression test that previously passed has started failing
on this release. Until the regression set is fully passing, this release
cannot roll forward.

**FIRST CHECK:**
```bash
adlc-cli regression-diff --release "$RELEASE" --agent "$AGENT"

Output names the failing regression test and the most recent release
that passed it. If the failing test is on the locked-in regression set,
the first check confirms.

SECOND CHECK:

adlc-cli regression-diff --release "$RELEASE" --agent "$AGENT" \
  --show-trajectory

Compares the failing trajectory against the last passing trajectory
for the same input. Look for a single divergent tool call. That is
almost always the bug.

ROLLBACK GATE: Pre-deploy rollback is "do not merge." If the regression
floor is below 1.0 at any time during the release window, do not merge.
No timer; no exception.

ESCALATE: If the divergent trajectory does not point at a single tool
call, page the agent owner listed in agents/<agent>.yaml for a same-day
review. Do not unblock by skipping the regression check.

---

Deeper context

The regression floor is the locked-in set of bugs we have already fixed.
A regression floor below 1.0 means a previously-fixed bug has come back.
See blog 184
for the lifecycle metric map and blog 185
for the dashboard layout that surfaces this metric.

The pattern in the headline is to lead with the symptom in business language, not in metric language, and then to state the consequence as a hard sentence. The first and second checks are commands, not instructions. The rollback gate is unusual for a pre-deploy alert in that it is "do not merge" rather than "execute rollback," but the principle is the same: the gate is pre-authorised and explicit. The escalate field names a role and gives the time bound.

Three more pre-deploy runbooks worth writing, by the same template:

1. `ADLCPreDeployGoldenEvalBelowFloor`. Golden eval pass rate has dropped below the per-agent floor. First check: `adlc-cli golden-eval-diff --release` to find the failing class of inputs. Rollback gate: do not merge if the failing class is on the high-stakes list.

2. `ADLCPreDeployJudgeDisagreementHigh`. Judge model disagreement above the ceiling. First check: re-run the judge with the previous judge model version and compare. Rollback gate: do not merge until judge stability is restored.

3. `ADLCPreDeployEvalProdKLDriftHigh`. The eval set is no longer representative of production traffic. First check: pull the top-five most-shifted input features. Rollback gate: this one is *not* a hard gate; the release can roll forward if the failing eval class is also low-stakes, but a follow-up eval-refresh ticket must be filed.

Notice that even the runbook for the alert that does not block a release still has an explicit gate. The gate is "file a ticket and continue," but it is named, and it is pre-authorised. The on-call engineer does not need to negotiate a soft gate with the release engineer at 11:30 PM; the runbook has already negotiated it.

## Post-Deploy Runbook Entries

Post-deploy alerts fire during the canary window, days one through fourteen of a release; the consumer is the on-call rotation; the cost of action is the rollback or hold of the canary; this is the runbook bucket that the original 02:41 AM postmortem produced. The post-deploy runbook is the most punishing, because the on-call engineer has the least context, and the alerts are the noisiest. Post-deploy is where the sixty-second contract earns its keep.

A worked example for the `ADLCPostDeployCohortQualityDrop` alert that started this post:

```markdown
# ADLCPostDeployCohortQualityDrop

**HEADLINE:** Canary cohort quality is sliding compared to baseline; most
likely a tool-call routing change in this release.

**FIRST CHECK:**
```bash
adlc-cli cohort-diff --canary "$RELEASE" --baseline last-stable \
  --metric tool-call-fingerprint --top 5

Output ranks the tool calls whose canary distribution has shifted most.
If the top entry is a search/retrieval tool, the hypothesis is confirmed.

SECOND CHECK:

adlc-cli cohort-diff --canary "$RELEASE" --baseline last-stable \
  --metric thumbs-down-rate --window 1h

If thumbs-down rate on the canary cohort is more than 1.5x the baseline
over the last hour, the user-loop signal corroborates the tool-call shift.

ROLLBACK GATE: If both checks confirm AND the cohort has been live
for at least 30 minutes, execute the canary rollback:

adlc-cli canary rollback "$RELEASE" --confirm

Pre-authorised. Do not page the release engineer first.

ESCALATE: If the first check does not confirm a tool-call shift,
page the agent-platform secondary at the 30-minute mark. Most likely
this is a different failure mode from this runbook.

---

Deeper context

Tool-call fingerprint shifts are the most common silent regression
in canary deployments because they often look fine on the per-call
quality metric but break the system-level outcome. See blog 184
for the lifecycle context and the agent observability stack post
for the underlying instrumentation.

Three observations on the post-deploy template. First, the rollback gate is the primary rescue mechanism. The original 02:41 AM incident took four hours to catch because the on-call engineer believed the alert was a transient and there was no pre-authorised rollback path; once the runbook was rewritten with a thirty-minute rollback gate, the equivalent incident the following month was resolved in twenty-eight minutes. Second, the escalate path is for the case where the runbook does not match the symptom, not for the case where the rollback fails. Most runbooks confuse these two cases. The escalate path should say that this is not the right runbook for this alert and that the on-call engineer should wake the right owner to find the matching entry. Third, the deeper context links into the existing blog cluster instead of reproducing the explanation, which keeps the runbook itself short.


flowchart LR
    A[Post-deploy alert] --> B[Cohort-diff first check]
    B -->|Tool-call shift confirmed| C[Thumbs-down second check]
    B -->|No shift| H[Escalate: wrong runbook]
    C -->|>1.5x baseline| D{30 min since cohort live?}
    C -->|At baseline| F[Hold, recheck in 30 min]
    D -->|Yes| E[Rollback canary]
    D -->|No| G[Wait until 30 min mark]
    E --> I[Close alert, file postmortem]
    style E fill:#82c8a0,color:#000
    style H fill:#dc6e6e,color:#fff
    style I fill:#82c8a0,color:#000


A handful of post-deploy runbooks worth writing by the same template, each with their own pre-authorised rollback gate:

1. `ADLCPostDeployTrajectoryDiffHigh`. Canary trajectories are diverging from baseline trajectories above the divergence ceiling. Rollback gate: divergence persists for fifteen minutes after first check.
2. `ADLCPostDeployUserLoopRegression`. Thumbs-down or human-handoff rate above the per-agent floor. Rollback gate: in our rollback policy, we measured 2x baseline for thirty minutes as the user-loop threshold.
3. `ADLCPostDeployCanaryLatencyP99Hike`. P99 latency on canary cohort above the baseline by more than 50 percent. Rollback gate: latency hike persists for ten minutes after first check.

The pattern that emerges is that every post-deploy runbook ends in a pre-authorised rollback. That is the structural acknowledgement that the canary window is *the* moment when rollback is cheap and the cost of waiting is high.

## Steady-State Runbook Entries

Steady-state alerts fire during day fifteen onward; the consumer is the platform team; the cadence is weekly to monthly; the cost of action is the lowest of the three stages because the action is rarely immediate rollback and is more often scheduling a remediation. Steady-state runbooks are read in calmer conditions, but the sixty-second contract still applies, because the steady-state alerts are the easiest to dismiss and the hardest to act on. A steady-state runbook that loses the first sixty seconds will be deferred to the next sprint and forgotten.

A worked example for `ADLCSteadyStateWeeklyDriftHigh`:

```markdown
# ADLCSteadyStateWeeklyDriftHigh

**HEADLINE:** Weekly drift detector has crossed the long-term band;
the agent's input distribution has shifted enough to affect quality
within the next two weeks if not addressed.

**FIRST CHECK:**
```bash
adlc-cli drift-report --agent "$AGENT" --window 12w --top 5

Lists the top five input features whose distribution has shifted
most over the rolling twelve-week window.

SECOND CHECK:

adlc-cli eval-coverage --agent "$AGENT" \
  --features "$(adlc-cli drift-report --agent "$AGENT" --top-1)"

Checks whether the eval set covers the shifted feature distribution.
If coverage is low, the eval set needs refreshing.

ROLLBACK GATE: Steady-state drift is not a rollback alert. The gate
is "schedule eval refresh." If drift is above the band AND eval coverage
is below 70 percent on the shifted feature, file a ticket on the
agent-platform backlog with priority "next sprint." Pre-authorised.

ESCALATE: If the drift correlates with a known external event
(model provider update, data partner change), escalate to the platform
team weekly review for full assessment.

---

Deeper context

Steady-state drift is the slowest-moving signal in the ADLC stack and
the one most likely to be ignored. See blog 184
for the metric definition and the rolling baseline approach.

The notable difference for steady-state runbooks is that the rollback gate is "schedule a ticket" rather than "execute a rollback," and the gate is still pre-authorised. The platform team member reading the runbook does not need to convince anyone that the ticket should be filed; the runbook has already convinced the team in advance, and the ticket is the action.

<div style="text-align:center;margin:24px 0;"><img src="https://pub-ad281554aa374a02af45994f3f24cea3.r2.dev/blog/images/186-adlc-on-call-runbook-structure-sixty-second-triage-entry-three-stage-map/comparison.png" alt="Comparison visual showing the three ADLC runbook templates side by side as three vertical cards, each card with the same five-field structure (HEADLINE, FIRST CHECK, SECOND CHECK, ROLLBACK GATE, ESCALATE) but different content per stage, color-coded sky blue for pre-deploy, amber for post-deploy, sage green for steady-state, with copper section dividers, and a footer row showing the gate type per stage: pre-deploy=do-not-merge, post-deploy=execute-rollback, steady-state=file-ticket" style="max-width:100%;border-radius:8px;box-shadow:0 4px 12px rgba(0,0,0,0.3);" /></div>

## Linking the Runbook from the Alert Annotation

A runbook is only as good as the link from the alert. Most teams write the runbook, store it on a wiki, and assume the on-call engineer will find it. The empirical pattern is that runbooks discovered by search at 03:00 AM are the runbooks that get skipped. The runbook link belongs in the Alertmanager annotation, on the alert's payload, and the link should resolve directly to the structured entry, not to the wiki home page.

The pattern we settled on uses the alert name as the slug, with the runbook entries stored as one Markdown file per alert in a versioned repository. The Alertmanager rule's `runbook_url` annotation points to the canonical URL for the alert name. A short example:

```yaml
- alert: ADLCPostDeployCohortQualityDrop
  expr: |
    (
      sum(rate(agent_quality_signal{cohort="canary"}[30m])) by (agent, release)
      /
      sum(rate(agent_quality_signal{cohort="baseline"}[30m])) by (agent, release)
    ) < 0.96
  for: 30m
  labels:
    severity: high
    stage: post-deploy
    route: agent-platform-on-call
  annotations:
    summary: "{{ $labels.agent }} canary cohort quality below 96% of baseline"
    runbook_url: "https://internal/runbooks/adlc/post-deploy/ADLCPostDeployCohortQualityDrop"
    headline: "Canary cohort quality is sliding compared to baseline; most likely a tool-call routing change."

Two details worth flagging. First, the headline annotation is duplicated from the runbook into the alert payload. The reason is operational: the on-call engineer's PagerDuty notification shows the headline before the runbook link is opened, so the headline is doing the sixty-second job before the engineer even reaches the runbook. Second, the runbook URL contains the alert name as the path, not a hash or a wiki page id. That makes the link stable across runbook edits, makes the link survives the wiki being moved to a new system, and lets the runbook itself be linted in CI by checking that every alert in alertmanager.yaml has a corresponding Markdown file in the runbook repo.

Linting Runbooks in CI

The runbook contract is enforced in CI by a lint script that runs on every change to the alertmanager rules or the runbook repo. The lint script is short, around 200 lines of Python, and it checks five properties:

1. Every alert in alertmanager.yaml has a runbook file at runbooks/<stage>/<alert-name>.md.
2. Every runbook file has the five required headings (HEADLINE, FIRST CHECK, SECOND CHECK, ROLLBACK GATE, ESCALATE) in the correct order.
3. The headline is at most fifteen words and contains no metric-vocabulary identifiers (matching ^agent_ is a fail).
4. The first and second check sections each contain at least one fenced code block (the copy-pasteable command).
5. The rollback gate section contains a deterministic threshold or the literal phrase "do not merge" or "schedule a ticket"; no aspirational language ("consider rolling back" is a fail).

A representative pytest snippet from our runbook lint suite, edited for length:

import re
import yaml
from pathlib import Path

REQUIRED_SECTIONS = ["HEADLINE", "FIRST CHECK", "SECOND CHECK",
                     "ROLLBACK GATE", "ESCALATE"]
ASPIRATIONAL = re.compile(r"\b(consider|maybe|might want to|could)\b", re.I)

def runbook_files():
    return sorted(Path("runbooks").rglob("*.md"))

def test_required_sections_present_and_in_order():
    for rb in runbook_files():
        text = rb.read_text()
        positions = [text.find(f"**{s}:**") for s in REQUIRED_SECTIONS]
        assert all(p >= 0 for p in positions), f"{rb}: missing section"
        assert positions == sorted(positions), f"{rb}: sections out of order"

def test_headline_is_short_and_human():
    for rb in runbook_files():
        text = rb.read_text()
        m = re.search(r"\*\*HEADLINE:\*\*\s*(.+)", text)
        assert m, f"{rb}: no headline"
        words = m.group(1).strip().split()
        assert len(words) <= 15, f"{rb}: headline too long"
        assert not re.search(r"\bagent_\w+", m.group(1)), f"{rb}: metric in headline"

def test_rollback_gate_is_deterministic():
    for rb in runbook_files():
        text = rb.read_text()
        m = re.search(r"\*\*ROLLBACK GATE:\*\*([\s\S]+?)\*\*ESCALATE", text)
        assert m, f"{rb}: no rollback gate"
        body = m.group(1)
        assert not ASPIRATIONAL.search(body), f"{rb}: aspirational language in rollback"

def test_alert_has_runbook():
    rules = yaml.safe_load(Path("alertmanager.yaml").read_text())
    alerts = [r["alert"] for g in rules["groups"] for r in g["rules"] if "alert" in r]
    for a in alerts:
        candidates = list(Path("runbooks").rglob(f"{a}.md"))
        assert candidates, f"alert {a} missing runbook file"

A terminal capture from a recent CI run:

$ pytest tests/test_runbooks.py
================================ test session starts =================================
collected 4 items

tests/test_runbooks.py::test_required_sections_present_and_in_order  PASSED  [ 25%]
tests/test_runbooks.py::test_headline_is_short_and_human             PASSED  [ 50%]
tests/test_runbooks.py::test_rollback_gate_is_deterministic          PASSED  [ 75%]
tests/test_runbooks.py::test_alert_has_runbook                       PASSED  [100%]

================================ 4 passed in 1.27s ===================================

The first time we ran this lint over the existing runbook corpus, eleven of the twenty-four post-deploy runbooks failed test_rollback_gate_is_deterministic because the gate language said "consider rolling back" or "rollback may be appropriate." Six failed test_headline_is_short_and_human because the headline was the alert name. Two had no rollback gate at all. Each of the failures was a real triage hazard. The CI lint converts every new failure into a fixable pull-request comment instead of a 02:41 AM postmortem.

flowchart TD A[PR changes alertmanager.yaml or runbooks/] --> B[CI: lint suite runs] B --> C{All 5 checks pass?} C -->|No| D[Block PR with failing check name and line] C -->|Yes| E[Reviewer reads runbook for content] D --> F[Author fixes runbook] F --> A E --> G{Reviewer approves?} G -->|No| H[Author addresses comments] G -->|Yes| I[Merge: runbook live in pager] H --> A style D fill:#dc6e6e,color:#fff style I fill:#82c8a0,color:#000

Production Considerations

A few production considerations carried over from the rollouts on three different agent platforms.

The first is runbook decay. Runbooks rot when the underlying tool-call vocabulary changes and the FIRST CHECK command stops returning the right output. We address this by treating runbooks as code: every runbook entry includes a last_validated field in the frontmatter, and a quarterly chore on the platform team's calendar runs every runbook against a synthetic alert and verifies the FIRST CHECK still produces the expected output shape. In our CI policy, we measured 90 days since validation as the warning threshold and 180 days as the failure threshold. The chore takes a couple of hours per quarter and saves multi-hour incidents.

The second is on-call training. New on-call engineers are paired with a senior engineer for two weeks, and the pairing exercise on day one is to walk through the runbook directory and have the new engineer execute the FIRST CHECK for ten different alerts on a staging cluster. The exercise discovers stale runbooks faster than any quarterly chore, and it builds the muscle memory of "open runbook, run first check" before the new engineer's first real page.

The third is the runbook headline as on-call education. We discovered that the headline ends up being the most-read piece of documentation about the agent platform, because the on-call engineer reads it on every alert. Writing the headline well is therefore a force multiplier for system understanding across the team. A headline that explains a canary cohort sliding compared to baseline and points at a likely tool-call routing change teaches every on-call engineer, every page, what the canary cohort is and what tool-call routing means. The headlines are the ambient curriculum.

The fourth is the cost of the lint suite. The five-check lint runs in under two seconds on a corpus of 80 runbooks. The cost of writing the lint, around half a day of platform-team work, paid back in the first two months by surfacing real triage hazards before they reached the on-call rotation. The lint is not a perfectionism tool; it is the cheapest way to enforce a bandwidth contract that the on-call engineer cannot enforce by themselves at 03:00 AM.

The fifth is what the structure does to incident reviews. Postmortems used to spend ten minutes asking whether the on-call engineer followed the runbook. Now postmortems spend ten minutes asking whether the runbook headline matched the actual symptom and whether the FIRST CHECK produced a useful answer. The shift in postmortem language is the clearest signal that the runbook structure is doing its job. The runbook becomes the unit of analysis, not the engineer.

Monetizing Runbook Reliability

Runbook quality becomes commercial the first time a customer asks why an incident took hours to resolve when the alert fired immediately. Metrics and dashboards can prove that the signal existed. The runbook proves whether the organization could turn the signal into action. That distinction matters for enterprise buyers because they are not only buying model capability; they are buying the vendor's ability to operate the agent after something changes at 03:00.

The sixty-second contract gives customer-success teams a concrete reliability artifact. During a QBR, a team can show that every high-severity ADLC alert links directly to a structured runbook with a headline, a first check, a second check, a rollback gate, and an escalation path. That is stronger than saying the team has incident response documentation. It shows that the documentation is designed for the exact moment when it is needed, with the right owner and the right action already named.

There is also a support-cost benefit. A bad runbook turns one page into three human interruptions: the primary on-call pages the secondary, the secondary wakes the original author, and someone eventually pings the product owner to ask whether rollback is safe. A good runbook pre-authorizes the safe action and names the role to escalate only when the runbook no longer matches the symptom. That saves engineering hours, but more importantly it preserves trust with customers because the first response is coherent.

The packaging path is straightforward. Standard tenants can receive post-incident notes that show which runbook entry was followed and what the first check returned. SLA-bound tenants can receive runbook-backed incident summaries with the rollback gate, escalation timing, and follow-up action. For high-touch enterprise accounts, the runbook corpus becomes part of operational diligence: the customer can see that alerts are not just monitored, they are paired with tested action paths.

The revenue rule is that no paid agent should launch without a runbook contract for every stage-specific alert it can emit. If the dashboard is a product signal and the alert route is an ownership signal, the runbook is the action signal. Selling reliable agents means selling the whole chain.

Conclusion

The ADLC three-stage map gives you a metric vocabulary; the dashboards give you a visual surface; the runbook gives you the sixty-second triage path that connects the alert to the action. None of the three works without the other two, and the runbook is the layer most often built last and most often built worst. The pattern that has worked for our team, and for the teams I have spoken to running similar agent platforms in 2026, is to fix the runbook contract first: five fields in fixed order, headline in business voice, copy-pasteable first and second checks, pre-authorised rollback gate, role-based escalation, deeper context below the fold, lint suite in CI. Same structure for pre-deploy, post-deploy, and steady-state, with stage-specific gate types. Companion repository scaffolding and the lint script are at amtocbot-examples/adlc-runbooks.

The next post in this cluster will look at the postmortem template that pairs with the runbook structure: the fields, the cadence, and the analysis loop that turns runbook misses into runbook fixes. The runbook contract is half of the on-call story; the postmortem template is the other half, and writing the two together is what closes the ADLC operational loop.

---

Revision History

Date	Summary	Old Version
2026-06-08	Rewrote the introduction as a first-person anecdote, added measurement attribution for alert thresholds, rollback thresholds, and validation windows, converted direct quote phrasing into indirect wording, and added a monetization section connecting runbook quality to enterprise trust, support cost, and SLA-backed incident summaries.	View original

Sources

• Datadog. State of AI Engineering Report 2026. April 2026. https://www.datadoghq.com/state-of-ai-engineering/
• LangChain. State of Agent Engineering. April 2026. https://www.langchain.com/state-of-agent-engineering
• Google SRE Workbook. Implementing SLOs. https://sre.google/workbook/implementing-slos/
• PagerDuty. Incident Response Documentation. https://response.pagerduty.com/
• Prometheus. Alertmanager Configuration Documentation. https://prometheus.io/docs/alerting/latest/configuration/
• OpenTelemetry. GenAI Semantic Conventions for Agent Observability. https://opentelemetry.io/docs/specs/semconv/gen-ai/

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-05 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter