This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Operational postmortem — Buzz retirement and forge migration

• Date: 2026-08-04
• Type: operational postmortem; service disruption and migration recovery
• Scope: Buzz retirement, fenrisulfr → bifrost forge cutover, agent identity minting, agent wiki/issue access gaps
• Relation to other postmortems: the architecture postmortem covers the flake composition failure (2026-07-29). This one covers the operational failures of the following week.

Summary

The fleet moved its execution tracker from Buzz (self-hosted Nostr, tried 2026-08-01 to 2026-08-03) back to Forgejo + Matrix, and moved the forge itself from fenrisulfr (jo.fen.et0.pw) to bifrost (jo.et0.pw) on 2026-08-03. The move was necessary — Buzz was ratified out as too early — but it surfaced three operational gaps that cost significant operator time and blocked the caretaker agent from resuming ownership of the repository.

No production outage or secret exposure resulted. The cost was operator time and agent effectiveness.

Timeline

1. 2026-08-01 to 2026-08-03 — Buzz trial. Buzz (self-hosted Nostr relay with NIP-34 issues-as-tickets) was trialed as a replacement for Matrix + Forgejo. The trial was ratified out: Buzz was too early, the relay was fragile, and the buzz CLI was removed from repo tooling. The decision to return to Forgejo + Matrix was correct and is not in question here.
2. 2026-08-03 — Forge cutover. The forge moved from fenrisulfr (jo.fen.et0.pw) to bifrost (jo.et0.pw), hosted alongside the Matrix node. The instance was stood up fresh, so repository IDs restarted: this repo is id 1, not the 3 it was on fenrisulfr.
3. 2026-08-03 — Agent identity minting. Per-identity persona accounts were minted on the new forge instance (just forgejo mint <persona>). The agent.odin persona (id 4, [email protected]) was created at 03:24.
4. 2026-08-04 — Wiki migration. agent.omo migrated 20 planning documents to the flaky-mesh.wiki repo, classified them in Home.md, reconciled executable work into issues #1–#6, and verified the transfer byte-identical.
5. 2026-08-04 — Agent access gaps surface. The agent.odin persona, despite carrying valid FORGEJO_URL and FORGEJO_ACCESS_TOKEN credentials in its rendered environment, could not reach the forge through its tool surface for approximately one hour of operator-assisted debugging.

What went wrong

1. The forgejo-mcp registration had two surfaces, not one

Root cause: The forgejo-mcp server has to be registered independently in two places: the NixOS module (modules/implementations/hermes.nix, the mcpServers block for the Hermes harness) and the per-user config (~/.config/opencode/opencode.json, for the opencode TUI). The agent initially assumed a single declaration surface and spent time investigating the NixOS module when the actual gap was in the per-user opencode config.

Contributing factor: The NixOS module's mcpServers.forgejo block contained a placeholder URL (https://your-forgejo-instance.org) rather than the real jo.et0.pw, which looked like the bug and delayed diagnosis.

Lesson: When a tool has multiple registration surfaces (harness-level declarative + per-user config), each surface must be verified independently. A correct declaration in one surface does not propagate to the other. Document both surfaces in AGENTS.md so the next agent does not assume a single path.

2. The token transport was not verified end-to-end

Root cause: The forgejo-mcp server requires its authentication token as a CLI flag (--token), not as an environment variable. The opencode config registered the command with --url but not --token, and the MCP child process inherited a sanitized environment that did not forward FORGEJO_ACCESS_TOKEN.

The persona env file at /run/secrets/rendered/profile.odin.env was correctly rendered (the credentials were present in the agent's shell environment), but the MCP child process did not inherit them, and the MCP server did not read from environment variables even if it had.

Lesson: Token transport must be verified by a live tool call, not by the presence of credentials in the parent shell. An env var in $ENV is not proof that a child MCP process can see it. Always run forgejo-mcp --help to discover the actual auth surface before assuming env-var passthrough.

3. The wiki repo was not provisioned with agent write access

Root cause: The flaky-mesh.wiki repo was created by agent.omo during the migration. agent.odin had read access (the repo is public) but not push access, because no per-agent write grant was issued on the wiki repo. The agent discovered this only after attempting to push a test branch, despite having previously confirmed read access via clone.

Lesson: Read access and write access are separate grants. A successful clone is not evidence of push access. When a new repo is created on the forge, the creating agent's identity gets write access by default; other agent identities must be granted explicitly.

What worked

• The persona system itself: FORGEJO_URL and FORGEJO_ACCESS_TOKEN were correctly rendered to the odin persona's environment through the sops-nix template at /run/secrets/rendered/profile.odin.env. The credential pipeline from secrets.yaml → sops → rendered env → agent shell is sound.
• The forge cutover itself: jo.et0.pw (bifrost) is live, the main repo (larandar/flaky-mesh, id 1) is accessible, and the token authenticates correctly against the API (HTTP 200 on authenticated read paths).
• agent.omo executed the wiki migration and issue reconciliation autonomously and correctly: 20 pages, 8 classified commits, issues #1–#6 with stable IDs and acceptance criteria, round-trip verified.
• The Forgejo wiki-as-sibling-repo feature works as intended for durable postmortem and design-intent storage, searchable and version-controlled.

Impact

• Approximately one hour of operator time was spent debugging the forgejo-mcp registration and token transport, including a NixOS rebuild that was not strictly necessary (the real fix was in the opencode config).
• The agent.odin persona was blocked from forge read/write for the duration, delaying the resumption of caretaker duties.
• No production service was affected. The forge, Matrix, and all fleet hosts remained operational throughout.

Corrective actions

1. Document both MCP registration surfaces in AGENTS.md. The Hermes NixOS module and the per-user opencode config are independent surfaces; both must carry the registration. Add a note under "Issue authority" or a new "Agent tooling" section.
2. Fix the placeholder URL in modules/implementations/hermes.nix. The mcpServers.forgejo block still points at https://your-forgejo-instance.org. Replace with the real jo.et0.pw and wire the token through the persona env, not as a literal in the config.
3. Establish a write-access grant procedure for new repos. When a new repo is created on the forge (wiki or otherwise), all active agent personas should be granted write access explicitly, not assumed to inherit it.
4. Add a one-shot verification step to the persona bootstrap. After sourcing a persona profile, the first forge interaction should be a read+write probe (create a test branch, push, delete) to confirm end-to-end transport before real work begins.

Prevention rules

• A credential in the parent shell is not proof of child-process access. Verify token transport with a live tool call.
• Read access is not write access. A successful clone does not prove push rights.
• When a tool has multiple registration surfaces, each must be verified independently. A correct declaration in one does not propagate to the others.
• The forgejo-mcp server reads its token from a CLI flag (--token), not an environment variable. Do not assume env-var passthrough for MCP child processes without checking --help first.
• Do not diagnose a config gap by reading the NixOS module alone. The per-user config (~/.config/opencode/opencode.json) is an independent surface.

Source disposition

• plans/brief.md: ratified fleet requirements (2026-07-28).
• plans/architecture-postmortem.md: flake composition postmortem (2026-07-29).
• AGENTS.md (repository): issue authority and tooling rules.
• Issue #8 (closed): VCS operational data migration ledger.
• Issues #1–#6 (open): reconciled executable work.
• Live session evidence: agent.odin forge interactions on 2026-08-04.