Incident postmortems

Reusable failures and the estate rule each one changed. A postmortem earns a place here when the fix is a rule someone else could otherwise have to learn the same way.

Evidence routed from README.md, whose copies PR #86 removes. Until that merges, both carry it.

A label the deployment never created forced squash, silently

Read on 2026-08-10, the Ting organisation carried no labels at all — none on the org, none on any repository. nix/policy.nix named curated_label, so the merge rule's curated_label branch never matched and the rule could only ever take its squash branch. Forever, and without a sound: a label that was never created is indistinguishable from one the author chose to withhold. The rule was right; the deployment was missing a precondition.

The rule it changed. A policy parameter that names forge state must be coupled to the declaration that creates that state. nix/forgejo.nix now declares the label for the org like any other desired state, and nix/policy.nix reads its name from there rather than spelling it again — the same coupling the merge styles already have. Eval fails if the rules could ever read a label the estate does not create.

The name is lar.ad's (owner, 2026-08-11): Reviewed/Curated, one name across the estate. A label means "the approver vouched for this history", and a meaning spelled two ways is two meanings the moment anyone reads across both namespaces.

An empty mirror filter published a development branch

The hand-made push mirror on lar.ad/jostoph-rs carried an empty branch_filter — every branch. A development branch created on the canonical was published to the public repository, and deleting the source branch did not retract it.

The rule it changed. A public mirror pins main. A midgard repository has shed the fastlane flow: it carries main, takes no pull requests — a pull request against a copy the mirror overwrites on a schedule produces work the next sync discards — and holds the source mirror's public key as a writable deploy key.

A steward off the forge host hears nothing and looks healthy

Forgejo refuses to deliver webhooks to loopback and private addresses unless the instance opts in, and the estate's instance opts in to loopback only (webhook.ALLOWED_HOST_LIST = "loopback", in lar.ad/flake-ops's modules/impls/forge/forgejo.nix). A steward deployed on any other host is not one that fails loudly — it is one whose hook silently never fires, while /healthz answers.

The rule it changed. Colocate the steward with the forge on loopback until forge policy is deliberately widened; widening ALLOWED_HOST_LIST to private or external is a decision on the forge's side, not this repository's. And an empty audit log is a finding, not a quiet system — which is what jostoph kinds exists to say out loud.

Two encrypted copies of one secret fail as "the steward is down"

The webhook secret has two readers on two machines with different lifecycles: the verifier here, and the reconciler wherever nixops4 apply runs. Encrypted twice into sops, with nothing keeping the two copies equal, they diverge — and divergence presents as every delivery rejected, which reads like an outage rather than a key mismatch.

The rule it changed. The owner ruled on 2026-08-10 that the value is minted by OpenBao, one managed secret delivered to both readers, making this hook the estate's first (flake-ops#67 phase 3, via flake-ops#2). This repository only names where it lands.