Epic: OpenBao joins the fleet, pinned to heimdallr alongside Vedanta #5

Closed
opened 2026-08-09 19:34:26 +00:00 by agent.teyla · 3 comments
Contributor

Production reconciliation (2026-08-30)

Heimdallr co-location retained; legacy custody and token delivery superseded. Follow the authoritative #63 reconciliation and OpenSpec 61. The original body below is retained as decision provenance and is not an alternative production path.


Source: design discussion during Ting/Vedanta#1's implementation, 2026-08-09.
Disposition: open — epic, needs decomposition, same posture as Ting/Ting#2.

Decision

OpenBao (open-source Vault fork) joins the fleet as secrets infrastructure for agent/service credentials, the same way Authelia is the fleet's auth for humans — but it and Vedanta both run on heimdallr, hard requirement, not optional and not portable to another host. Corrected 2026-08-09 from an earlier "deploys optionally, wherever Vedanta runs" framing, which was wrong: two independent reasons pin both to this one box.

  1. OpenBao is Authelia's counterpart for non-human principals. Authelia already lives on heimdallr because it's the identity-adjacent auth component for humans; OpenBao is that same responsibility for agents and services, so it belongs on the same machine for the same reason — not a separate, relocatable service that happens to talk to lldap.
  2. lldap will not be opened to the internet. It binds loopback today (modules/hosts/heimdallr/configuration.nix, flake-ops) and stays that way — so anything that reads or writes it, Vedanta included, has to be co-located. This was already Ting/Vedanta#1's own deploy-topology call ("nothing but identity belongs on this machine" / lldap loopback-only); OpenBao inherits the identical constraint, not a fresh one.

This also answers this epic's own previously-open question about a non-heimdallr Vedanta+OpenBao host reopening the fate-isolation argument: it doesn't, because there is no other host — both are pinned to heimdallr, full stop.

OpenBao serves two distinct data flows, not one:

  1. Credential in: gates access to the lldap super service-account credential Vedanta's mint path uses (AdminClient::login, Ting/Vedanta PR #2). Today that's a plain sops-delivered password; this replaces it so the long-lived credential never leaves OpenBao at all.
  2. Credentials out: stores/tracks whatever Vedanta itself mints (tokens issued to hands, service accounts, later the agent pool) — a durable, revocable record, not "minted once and only the holder has it."

These are different paths through the same store (likely different secret engines/policies) and should not be collapsed into one lease.

Module packaging: OpenBao ships INSIDE Vedanta's flake, conditionally

Corrected 2026-08-09, same session: OpenBao is not a separate flake-ops-level component wired in next to services.vedanta — it's a co-module bundled in Ting/Vedanta's own flake (alongside nixosModules.default's services.vedanta options from PR #2), enabled by default only when no external secret manager is configured. The shape this implies: something like services.vedanta.openbao.enable, defaulting true, with an escape hatch (an external OpenBao/Vault address option) that turns the bundled instance off when a real one already exists to point at. This keeps "just enable Vedanta" a complete, self-contained deploy on heimdallr with no second service to separately stand up by hand — the local instance is the default path, not an opt-in extra.

The mechanism: OpenBao Agent renders straight onto DOTENV_FILE

Added 2026-08-09, same session: Vedanta needs zero OpenBao-specific code. Agent's template stanza fetches the lldap super credential and renders it directly onto the same DOTENV_FILE path load_dotenv already reads (PR #2) — no OpenBao HTTP client, no auth flow, no token handling anywhere in Vedanta's Rust. Agent runs as its own sidecar systemd unit, holds the OpenBao session using the human-minted bounded token, and keeps the file current for as long as that token is valid.

This is also what makes the dead-man-switch mechanism actually hold without extra enforcement: Agent auto-renews the lease on the secret it's already holding, but it cannot extend the token past whatever max TTL the human minted it with. Once that ceiling is hit and nobody has re-minted, Agent's renewal fails, rendering stops, and Vedanta finds a stale or absent DOTENV_FILE on its own next read — the switch is a property of the token's own bounded lifetime, not something Agent, Vedanta, or this module has to separately implement or police.

The dead-man-switch flow (credential-in side)

  • A human mints a bounded-lifetime (~1 month) OpenBao access token scoped to read the lldap super credential's path. Lifetime is calibrated to the fleet's own deploy cadence — at least one deploy a month is already the norm, so re-minting rides an existing ritual instead of a new one.
  • The token must be re-minted, not auto-renewed. An OpenBao periodic token that renews itself defeats the point — the dead-man-switch only means something if a human is forced back into the loop each cycle.
  • The token is what travels through sops → wherever OpenBao Agent's auto-auth method reads it from (a token_file, most likely). Agent, not Vedanta, is what actually calls OpenBao.
  • If nobody re-mints, the token expires, Agent's renewal fails, and DOTENV_FILE goes stale — Vedanta's mint/reconcile path stops working. This is accepted, deliberately: Vedanta's path is additive and occasional — failing safe when unattended is the point, not a bug.

Explicitly decided NOT to do (yet)

  • No dead-man-switch on Authelia's lldap bind. Owner ruling, 2026-08-09: Authelia and lldap already share heimdallr's trust boundary — if the box is compromised, lldap is compromised regardless of how Authelia's credential is protected, so gating only one of the two co-located credentials adds ceremony without raising the bar. Authelia keeps its current static sops-delivered admin credential unchanged. A future auto-renewing (not human-gated) OpenBao lease for Authelia is plausible later but explicitly out of scope now — and if it happens, it must be a periodic lease, never the human-minted one Vedanta uses, because Authelia's bind is continuous/load-bearing (every login, fleet-wide, including the operator's own way back in) where Vedanta's is occasional/additive. Conflating the two would turn "nobody renewed a token" into a fleet-wide lockout.

Open questions for decomposition

  • OpenBao's own bootstrap (unseal keys / root token) has the same "who holds the master credential" turtle one level up that lldap's own admin bootstrap already has. Not solved here.
  • Sizing heimdallr for a third (fourth, with Agent) service. heimdallr's current sizing (1 CPU / 512 MB, per docs/heimdallr-identity-tickets.md) was measured against Authelia + lldap alone. OpenBao + Agent landing on the same box needs its own headroom check, not an assumption that today's sizing still holds.
  • Which secret engines (KV, Transit, something else) back the credential-in vs. credentials-out paths, and what policies scope each.
  • Whether the credentials-out side eventually replaces secrets/mint-identity-token.nu's operator-held signing key entirely (Transit engine doing the ES256 signing itself, key never leaving OpenBao) — plausible, not decided.
  • Exact shape of the "external secret manager" escape hatch (an address + auth option, presumably) and what happens to already-bundled state if a deploy flips from local to external mid-lifecycle.
  • Agent's exact auto-auth method (token_file is the obvious fit for a human-minted token, but worth confirming against Agent's docs rather than assumed) and the template syntax for rendering a multi-line DOTENV_FILE-shaped output.

Context

  • Ting/Vedanta#1 — the mint effector this epic's credential-in flow protects; AdminClient::login currently reads a plain sops-delivered password, the thing this epic replaces. Its own flake.nix/module.nix (PR #2) is where the bundled OpenBao + Agent co-modules land, and load_dotenv is the existing convention Agent's rendered file slots into unchanged.
  • Ting/Ting#2 — the pool-manager epic; the credentials-out side of this store is where its eventual minted tokens should land too, not just the hand/service-account mints Vedanta does today.
  • docs/identity-pool-schema.md, docs/librefang-hand-identity-schema.md (flake-ops) — the schemas whose "neither service account exists yet... that over-grant is noted, not solved" line this epic is the answer to.
## Production reconciliation (2026-08-30) **Heimdallr co-location retained; legacy custody and token delivery superseded.** Follow the [authoritative #63 reconciliation](https://jo.et0.pw/Ting/Ting/issues/5#issuecomment-13276) and [OpenSpec 61](https://jo.et0.pw/Ting/Ting/wiki/OpenSpec-61-forge-identity-lifecycle-contract). The original body below is retained as decision provenance and is not an alternative production path. --- Source: design discussion during Ting/Vedanta#1's implementation, 2026-08-09. Disposition: open — epic, needs decomposition, same posture as Ting/Ting#2. ## Decision [OpenBao](https://openbao.org/) (open-source Vault fork) joins the fleet as secrets infrastructure for agent/service credentials, the same way Authelia is the fleet's auth for humans — but it and Vedanta both run **on heimdallr, hard requirement, not optional and not portable to another host.** Corrected 2026-08-09 from an earlier "deploys optionally, wherever Vedanta runs" framing, which was wrong: two independent reasons pin both to this one box. 1. **OpenBao is Authelia's counterpart for non-human principals.** Authelia already lives on heimdallr because it's the identity-adjacent auth component for humans; OpenBao is that same responsibility for agents and services, so it belongs on the same machine for the same reason — not a separate, relocatable service that happens to talk to lldap. 2. **lldap will not be opened to the internet.** It binds loopback today (`modules/hosts/heimdallr/configuration.nix`, flake-ops) and stays that way — so anything that reads or writes it, Vedanta included, has to be co-located. This was already `Ting/Vedanta#1`'s own deploy-topology call ("nothing but identity belongs on this machine" / lldap loopback-only); OpenBao inherits the identical constraint, not a fresh one. This also answers this epic's own previously-open question about a non-heimdallr Vedanta+OpenBao host reopening the fate-isolation argument: it doesn't, because there is no other host — both are pinned to heimdallr, full stop. OpenBao serves **two distinct data flows**, not one: 1. **Credential in**: gates access to the lldap super service-account credential Vedanta's mint path uses (`AdminClient::login`, `Ting/Vedanta` PR #2). Today that's a plain sops-delivered password; this replaces it so the long-lived credential never leaves OpenBao at all. 2. **Credentials out**: stores/tracks whatever Vedanta itself mints (tokens issued to hands, service accounts, later the agent pool) — a durable, revocable record, not "minted once and only the holder has it." These are different paths through the same store (likely different secret engines/policies) and should not be collapsed into one lease. ## Module packaging: OpenBao ships INSIDE Vedanta's flake, conditionally Corrected 2026-08-09, same session: OpenBao is not a separate flake-ops-level component wired in next to `services.vedanta` — it's a co-module bundled in `Ting/Vedanta`'s own flake (alongside `nixosModules.default`'s `services.vedanta` options from PR #2), enabled by default **only when no external secret manager is configured**. The shape this implies: something like `services.vedanta.openbao.enable`, defaulting true, with an escape hatch (an external OpenBao/Vault address option) that turns the bundled instance off when a real one already exists to point at. This keeps "just enable Vedanta" a complete, self-contained deploy on heimdallr with no second service to separately stand up by hand — the local instance is the default path, not an opt-in extra. ## The mechanism: [OpenBao Agent](https://openbao.org/docs/agent-and-proxy/agent/) renders straight onto `DOTENV_FILE` Added 2026-08-09, same session: **Vedanta needs zero OpenBao-specific code.** Agent's `template` stanza fetches the lldap super credential and renders it directly onto the same `DOTENV_FILE` path `load_dotenv` already reads (PR #2) — no OpenBao HTTP client, no auth flow, no token handling anywhere in Vedanta's Rust. Agent runs as its own sidecar systemd unit, holds the OpenBao session using the human-minted bounded token, and keeps the file current for as long as that token is valid. This is also what makes the dead-man-switch mechanism actually hold without extra enforcement: Agent auto-renews the *lease on the secret* it's already holding, but it cannot extend the *token* past whatever max TTL the human minted it with. Once that ceiling is hit and nobody has re-minted, Agent's renewal fails, rendering stops, and Vedanta finds a stale or absent `DOTENV_FILE` on its own next read — the switch is a property of the token's own bounded lifetime, not something Agent, Vedanta, or this module has to separately implement or police. ## The dead-man-switch flow (credential-in side) - A human mints a bounded-lifetime (~1 month) OpenBao access token scoped to read the lldap super credential's path. Lifetime is calibrated to the fleet's own deploy cadence — at least one deploy a month is already the norm, so re-minting rides an existing ritual instead of a new one. - The token must be **re-minted**, not auto-renewed. An OpenBao *periodic* token that renews itself defeats the point — the dead-man-switch only means something if a human is forced back into the loop each cycle. - The token is what travels through sops → wherever OpenBao Agent's auto-auth method reads it from (a `token_file`, most likely). Agent, not Vedanta, is what actually calls OpenBao. - If nobody re-mints, the token expires, Agent's renewal fails, and `DOTENV_FILE` goes stale — Vedanta's mint/reconcile path stops working. This is accepted, deliberately: Vedanta's path is additive and occasional — failing safe when unattended is the point, not a bug. ## Explicitly decided NOT to do (yet) - **No dead-man-switch on Authelia's lldap bind.** Owner ruling, 2026-08-09: Authelia and lldap already share heimdallr's trust boundary — if the box is compromised, lldap is compromised regardless of how Authelia's credential is protected, so gating only one of the two co-located credentials adds ceremony without raising the bar. Authelia keeps its current static sops-delivered admin credential unchanged. A future auto-renewing (not human-gated) OpenBao lease for Authelia is plausible later but explicitly out of scope now — and if it happens, it must be a *periodic* lease, never the human-minted one Vedanta uses, because Authelia's bind is continuous/load-bearing (every login, fleet-wide, including the operator's own way back in) where Vedanta's is occasional/additive. Conflating the two would turn "nobody renewed a token" into a fleet-wide lockout. ## Open questions for decomposition - **OpenBao's own bootstrap** (unseal keys / root token) has the same "who holds the master credential" turtle one level up that lldap's own admin bootstrap already has. Not solved here. - **Sizing heimdallr for a third (fourth, with Agent) service.** heimdallr's current sizing (1 CPU / 512 MB, per `docs/heimdallr-identity-tickets.md`) was measured against Authelia + lldap alone. OpenBao + Agent landing on the same box needs its own headroom check, not an assumption that today's sizing still holds. - Which secret engines (KV, Transit, something else) back the credential-in vs. credentials-out paths, and what policies scope each. - Whether the credentials-out side eventually replaces `secrets/mint-identity-token.nu`'s operator-held signing key entirely (Transit engine doing the ES256 signing itself, key never leaving OpenBao) — plausible, not decided. - Exact shape of the "external secret manager" escape hatch (an address + auth option, presumably) and what happens to already-bundled state if a deploy flips from local to external mid-lifecycle. - Agent's exact auto-auth method (`token_file` is the obvious fit for a human-minted token, but worth confirming against Agent's docs rather than assumed) and the template syntax for rendering a multi-line `DOTENV_FILE`-shaped output. ## Context - `Ting/Vedanta#1` — the mint effector this epic's credential-in flow protects; `AdminClient::login` currently reads a plain sops-delivered password, the thing this epic replaces. Its own `flake.nix`/`module.nix` (PR #2) is where the bundled OpenBao + Agent co-modules land, and `load_dotenv` is the existing convention Agent's rendered file slots into unchanged. - `Ting/Ting#2` — the pool-manager epic; the credentials-out side of this store is where its eventual minted tokens should land too, not just the hand/service-account mints Vedanta does today. - `docs/identity-pool-schema.md`, `docs/librefang-hand-identity-schema.md` (flake-ops) — the schemas whose "neither service account exists yet... that over-grant is noted, not solved" line this epic is the answer to.
agent.teyla changed title from Epic: OpenBao joins the fleet, deployed optionally alongside Vedanta to Epic: OpenBao joins the fleet, pinned to heimdallr alongside Vedanta 2026-08-09 19:37:38 +00:00
Owner

Lifecycle contract reconciliation: OpenBao is now the lease authority, not merely a confirmation or KV-custody step. The canonical issue/renew/revoke path is OpenBao → private Vedanta effect → Kanidm, with the workload seeing only OpenBao. See Ting/Ting#60 and OpenSpec 61.

<!-- forge-identity-lifecycle-distribution:2026-08-29 --> Lifecycle contract reconciliation: OpenBao is now the lease authority, not merely a confirmation or KV-custody step. The canonical issue/renew/revoke path is OpenBao → private Vedanta effect → Kanidm, with the workload seeing only OpenBao. See [Ting/Ting#60](https://jo.et0.pw/Ting/Ting/issues/60) and [OpenSpec 61](https://jo.et0.pw/Ting/Ting/wiki/OpenSpec-61-forge-identity-lifecycle-contract).
Owner

Ratified reconciliation — co-location retained, lease authority corrected

This comment is the authoritative production interpretation of this Epic after the final Ting/Ting#62 ratification. Earlier sections remain provenance; where they describe OpenBao as storage, confirmation, Agent-rendered custody, or a human-reminted long-lived runtime-token gate, this notice supersedes them.

The topology decision survives: OpenBao and Vedanta are deployed on heimdallr alongside the identity boundary; workloads do not gain a route to Kanidm. Packaging may keep the services deployable together, but packaging does not merge their authorities.

The production authority split is:

  • workloads call OpenBao and never Vedanta;
  • OpenBao authorizes and owns lease issue, renewal, expiry, revocation, and one-shot delivery, but holds no Kanidm mutation authority;
  • Vedanta is a private credential effector with Kanidm mutation capability; it neither authors grants nor serves workloads;
  • Kanidm owns durable principal and credential state, not workflow policy.

Consequently, the earlier credential in proposal placing Vedanta's directory mutation credential in OpenBao, the claim that OpenBao merely confirms or stores Vedanta-minted credentials, and the OpenBao Agent → shared DOTENV_FILE runtime-delivery mechanism are not production paths. Neither a monthly human-reminted token nor any static/periodic shared bearer token may substitute for a bounded workload lease.

For every drone Forge lease, OpenBao validates the immutable AssignmentGrant, active queen tenure, generation, repository set, and liveness before invoking Vedanta privately. Vedanta causes Forgejo to mint one distinct read-only, repository-restricted token under the active queen holder's Forge account. The drone alone receives it through a five-minute one-shot wrapper. Maximum lease/token authority is 30 minutes; renewal starts at least ten minutes before expiry, creates a distinct successor, and allows at most five minutes of overlap. Exact deletion is required when Forgejo cannot encode expiry; pending deletion blocks reuse.

Provisioning the ten-member swarm-alpha pool creates its durable principals and ten immutable grants, but no Forge account, token, or active lease. Ordinary completion returns the same UUID to the pool only after explicit revocation, negative runtime liveness, checkout-credential rotation, and generation advance; its next allocation uses a new immutable grant.

OpenBao bootstrap/unseal and the private delivery of Vedanta's own administrative credential remain implementation concerns, but they must preserve the authority split above and cannot make OpenBao the holder of Kanidm mutation authority.

Canonical authority: OpenSpec 61, final #62 projection, combined specification SHA-256 3af93de7e11433a89f4181ba7ea7624063ca78af81bdc1f40c6947f11f9198cb.

<!-- forge-identity-lifecycle-reconciliation:63:issue-5 --> ## Ratified reconciliation — co-location retained, lease authority corrected This comment is the authoritative production interpretation of this Epic after the final Ting/Ting#62 ratification. Earlier sections remain provenance; where they describe OpenBao as storage, confirmation, Agent-rendered custody, or a human-reminted long-lived runtime-token gate, this notice supersedes them. The topology decision survives: OpenBao and Vedanta are deployed on heimdallr alongside the identity boundary; workloads do not gain a route to Kanidm. Packaging may keep the services deployable together, but packaging does not merge their authorities. The production authority split is: - workloads call OpenBao and never Vedanta; - OpenBao authorizes and owns lease issue, renewal, expiry, revocation, and one-shot delivery, but holds no Kanidm mutation authority; - Vedanta is a private credential effector with Kanidm mutation capability; it neither authors grants nor serves workloads; - Kanidm owns durable principal and credential state, not workflow policy. Consequently, the earlier `credential in` proposal placing Vedanta's directory mutation credential in OpenBao, the claim that OpenBao merely confirms or stores Vedanta-minted credentials, and the OpenBao Agent → shared `DOTENV_FILE` runtime-delivery mechanism are not production paths. Neither a monthly human-reminted token nor any static/periodic shared bearer token may substitute for a bounded workload lease. For every drone Forge lease, OpenBao validates the immutable `AssignmentGrant`, active queen tenure, generation, repository set, and liveness before invoking Vedanta privately. Vedanta causes Forgejo to mint one distinct read-only, repository-restricted token under the active queen holder's Forge account. The drone alone receives it through a five-minute one-shot wrapper. Maximum lease/token authority is 30 minutes; renewal starts at least ten minutes before expiry, creates a distinct successor, and allows at most five minutes of overlap. Exact deletion is required when Forgejo cannot encode expiry; pending deletion blocks reuse. Provisioning the ten-member `swarm-alpha` pool creates its durable principals and ten immutable grants, but no Forge account, token, or active lease. Ordinary completion returns the same UUID to the pool only after explicit revocation, negative runtime liveness, checkout-credential rotation, and generation advance; its next allocation uses a new immutable grant. OpenBao bootstrap/unseal and the private delivery of Vedanta's own administrative credential remain implementation concerns, but they must preserve the authority split above and cannot make OpenBao the holder of Kanidm mutation authority. Canonical authority: [OpenSpec 61](https://jo.et0.pw/Ting/Ting/wiki/OpenSpec-61-forge-identity-lifecycle-contract), [final #62 projection](https://jo.et0.pw/Ting/Ting/issues/62#issuecomment-13267), combined specification SHA-256 `3af93de7e11433a89f4181ba7ea7624063ca78af81bdc1f40c6947f11f9198cb`.
Owner

Closing per this issue's own "Production reconciliation (2026-08-30)" note: heimdallr co-location is retained, but legacy custody/token-delivery is superseded by the authoritative #63 reconciliation and OpenSpec 61. No open thread here; the current work is tracked under Epic #60's rollout. Grooming pass, freeholder-authorized.

Closing per this issue's own "Production reconciliation (2026-08-30)" note: heimdallr co-location is retained, but legacy custody/token-delivery is superseded by the authoritative [#63 reconciliation](https://jo.et0.pw/Ting/Ting/issues/5#issuecomment-13276) and [OpenSpec 61](https://jo.et0.pw/Ting/Ting/wiki/OpenSpec-61-forge-identity-lifecycle-contract). No open thread here; the current work is tracked under Epic #60's rollout. Grooming pass, freeholder-authorized.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
Ting/Ting#5
No description provided.