Ting/Vedanta

Watch

Fork 0

Nix service: webhook + reconciliation timer + lldap access + OTLP #1

New issue

Closed

opened 2026-08-09 18:32:38 +00:00 by agent.teyla · 4 comments

agent.teyla commented

2026-08-09 18:32:38 +00:00

Collaborator

Vedanta needs to actually run somewhere, not just build. Scope, decided 2026-08-09:

Deploy target: heimdallr itself. heimdallr's own doc (modules/hosts/heimdallr/configuration.nix in flake-ops) says "nothing but identity belongs on this machine" because a credential issuer must not share fate with the workloads it issues for — but Vedanta is identity infrastructure, not a workload in that sense, and lldap only binds loopback (127.0.0.1:3890) there today. Running off-box would mean opening lldap up beyond loopback, which is new attack surface on the identity host that the pool schema deliberately left as an open question rather than a default. On-box wins on the fate-isolation reading and costs nothing new.
Webhook: already substantially there — jostoph::service::serve's POST /v1/evaluate (jostoph-rs) IS the webhook surface (mint-request events in, a Decision out). What's missing is running it as a standing systemd service instead of a cargo run.
Reconciliation: NOT webhook-driven, a separate systemd-timer-triggered oneshot (vedanta reconcile). No reconciliation rules exist yet — the owner named Matrix specifically as a thing that will drift silently (room/account state can diverge from lldap with no event ever firing), so periodic sweep is required in addition to the webhook path, not instead of it. First cut is a stub that connects and no-ops; rules land as separate tickets once decided, same posture as principal-shape being "the shape of the first rule, not the finished service."
lldap access: a connection the binary can prove at startup (bind check), config via env — the actual write-scope/privilege question (docs/identity-pool-schema.md's "neither service account exists yet, both act through lldap_admin for now, that over-grant is noted not solved") is unchanged by this ticket, just wired.
OTLP: no bundled collector-side module — Vedanta pushes its own traces/metrics via tracing-opentelemetry + opentelemetry-otlp, reading a plain otlpEndpoint/enable pair off the NixOS module (same minimal shape as modron/modules/telemetry.nix), so whatever's watching the fleet's telemetry (Victoria, on tamiel) picks it up the same way anything else pushing OTLP does.

Out of scope here: heimdallr's actual live secrets/deploy wiring (a real credential onto a real identity host) — this ticket is the binary + the NixOS module it ships, not the flake-ops-side consumption.

Vedanta needs to actually run somewhere, not just build. Scope, decided 2026-08-09: - **Deploy target: heimdallr itself.** heimdallr's own doc (`modules/hosts/heimdallr/configuration.nix` in flake-ops) says "nothing but identity belongs on this machine" because a credential issuer must not share fate with the workloads it issues for — but Vedanta *is* identity infrastructure, not a workload in that sense, and lldap only binds loopback (127.0.0.1:3890) there today. Running off-box would mean opening lldap up beyond loopback, which is new attack surface on the identity host that the pool schema deliberately left as an open question rather than a default. On-box wins on the fate-isolation reading and costs nothing new. - **Webhook**: already substantially there — `jostoph::service::serve`'s `POST /v1/evaluate` (jostoph-rs) IS the webhook surface (mint-request events in, a `Decision` out). What's missing is running it as a standing systemd service instead of a `cargo run`. - **Reconciliation**: NOT webhook-driven, a separate systemd-timer-triggered oneshot (`vedanta reconcile`). No reconciliation rules exist yet — the owner named Matrix specifically as a thing that will drift silently (room/account state can diverge from lldap with no event ever firing), so periodic sweep is required in addition to the webhook path, not instead of it. First cut is a stub that connects and no-ops; rules land as separate tickets once decided, same posture as `principal-shape` being "the shape of the first rule, not the finished service." - **lldap access**: a connection the binary can prove at startup (bind check), config via env — the actual write-scope/privilege question (`docs/identity-pool-schema.md`'s "neither service account exists yet, both act through lldap_admin for now, that over-grant is noted not solved") is unchanged by this ticket, just wired. - **OTLP**: no bundled collector-side module — Vedanta pushes its own traces/metrics via `tracing-opentelemetry` + `opentelemetry-otlp`, reading a plain `otlpEndpoint`/`enable` pair off the NixOS module (same minimal shape as `modron/modules/telemetry.nix`), so whatever's watching the fleet's telemetry (Victoria, on tamiel) picks it up the same way anything else pushing OTLP does. Out of scope here: heimdallr's actual live secrets/deploy wiring (a real credential onto a real identity host) — this ticket is the binary + the NixOS module it ships, not the flake-ops-side consumption.

agent.teyla referenced this issue from a commit

2026-08-09 18:52:21 +00:00

vendor: jostoph-rs at 029a102c, path dependency — PROVISIONAL

agent.teyla referenced this issue from a commit

2026-08-09 18:52:21 +00:00

serve/reconcile: split the binary on its two triggers, wire lldap + OTLP + dotenv

agent.teyla referenced this issue from a pull request that will close it,

2026-08-09 18:52:38 +00:00

Nix service: webhook + reconciliation timer + lldap access + OTLP #2

agent.teyla commented

2026-08-09 20:09:32 +00:00

Author

Collaborator

Vedanta's scope, crystallized after the OpenBao design thread (`Ting/Ting#5`, `Ting/Ting#2`)

Worth recording here directly, since it narrows what "the finished service" (this issue's own phrase, borrowed from principal-shape's doc comment) actually ends up being, past what PR #2 implements today.

Vedanta's ongoing job is initial mint + OpenBao registration, one-shot per identity — not runtime secret brokering. Concretely, per newly minted identity:

Create the lldap user (built, PR #2 — AdminClient::create_user).
Register a matching OpenBao entity + entity alias, so that identity has something distinct to authenticate to OpenBao as.

That's it. Everything after that binding exists is OpenBao's own native mechanism, not Vedanta code:

Credential leasing & rotation-at-checkout — non-renewable leases mean a pool slot's reissue is "ask for a fresh lease," not custom rotation logic (Ting/Ting#2's comment thread).
Per-queen/per-agent isolation — identity-templated ACL paths ({{identity.entity.id}}) scope each entity to only its own roster/secrets, enforced by OpenBao before a request ever reaches a secrets engine (Ting/Ting#2).
PKI issuance, if/when that lands — OpenBao's own PKI + SSH engines, no separate CA needed once OpenBao is mandatory anyway (Ting/Ting#5).
Delivery — OpenBao Agent renders straight onto the same DOTENV_FILE path Vedanta's own load_dotenv already reads; nothing downstream needs to speak OpenBao's API at all (Ting/Ting#5).

The identity-templated policy itself is fleet-wide and written once — it is NOT per-mint work. What's per-mint is narrowly the entity/alias registration, which is real, unbuilt work (not yet in PR #2's AdminClient) and the concrete next scope item once #5 exists to register against.

## Vedanta's scope, crystallized after the OpenBao design thread (`Ting/Ting#5`, `Ting/Ting#2`) Worth recording here directly, since it narrows what "the finished service" (this issue's own phrase, borrowed from `principal-shape`'s doc comment) actually ends up being, past what PR #2 implements today. **Vedanta's ongoing job is initial mint + OpenBao registration, one-shot per identity — not runtime secret brokering.** Concretely, per newly minted identity: 1. Create the lldap user (built, PR #2 — `AdminClient::create_user`). 2. Register a matching OpenBao entity + entity alias, so that identity has something distinct to authenticate to OpenBao as. That's it. Everything after that binding exists is OpenBao's own native mechanism, not Vedanta code: - **Credential leasing & rotation-at-checkout** — non-renewable leases mean a pool slot's reissue is "ask for a fresh lease," not custom rotation logic (`Ting/Ting#2`'s comment thread). - **Per-queen/per-agent isolation** — identity-templated ACL paths (`{{identity.entity.id}}`) scope each entity to only its own roster/secrets, enforced by OpenBao before a request ever reaches a secrets engine (`Ting/Ting#2`). - **PKI issuance, if/when that lands** — OpenBao's own PKI + SSH engines, no separate CA needed once OpenBao is mandatory anyway (`Ting/Ting#5`). - **Delivery** — OpenBao Agent renders straight onto the same `DOTENV_FILE` path Vedanta's own `load_dotenv` already reads; nothing downstream needs to speak OpenBao's API at all (`Ting/Ting#5`). The identity-templated *policy* itself is fleet-wide and written once — it is NOT per-mint work. What's per-mint is narrowly the entity/alias registration, which is real, unbuilt work (not yet in PR #2's `AdminClient`) and the concrete next scope item once #5 exists to register against.

agent.teyla commented

2026-08-09 20:09:57 +00:00

Author

Collaborator

Addendum: reconciliation includes deleting downstream mirrors, and that's NOT a boundary violation

Following up on the scope summary above — reconcile (PR #2's stub, currently a no-op past the lldap bind check) needs to be bidirectional against Matrix and Forgejo specifically, not just create-what's-missing: an lldap identity that's gone but still has a Matrix account or Forgejo account is exactly the kind of silent drift reconcile exists to catch (Ting/Vedanta#1's own justification: "Matrix in particular will drift... must be tracked in cron job").

Worth being explicit about a boundary question this could otherwise blur: Vedanta's charter (README) is additive only — sibling to Jostoph, which is subtractive only (rotate/disable lldap accounts). Deleting a Matrix or Forgejo account during reconciliation does NOT cross that line, because lldap is not the thing being deleted from — Matrix/Forgejo are downstream mirrors of lldap's state, and keeping a mirror accurate is inherently bidirectional (create what's missing, remove what's stale) regardless of which direction a given correction happens to point. The additive/subtractive split is specifically about who may act on the directory (lldap) — Jostoph disables/rotates there, Vedanta never does. Reconciling other systems against the directory's current truth is a different privilege domain entirely, and stating it here so a future decomposition doesn't misread "Vedanta deletes a Matrix user" as scope creep into Jostoph's territory.

## Addendum: reconciliation includes deleting downstream mirrors, and that's NOT a boundary violation Following up on the scope summary above — `reconcile` (PR #2's stub, currently a no-op past the lldap bind check) needs to be bidirectional against Matrix and Forgejo specifically, not just create-what's-missing: an lldap identity that's gone but still has a Matrix account or Forgejo account is exactly the kind of silent drift `reconcile` exists to catch (Ting/Vedanta#1's own justification: "Matrix in particular will drift... must be tracked in cron job"). Worth being explicit about a boundary question this could otherwise blur: Vedanta's charter (README) is **additive only** — sibling to Jostoph, which is subtractive only (rotate/disable lldap accounts). Deleting a Matrix or Forgejo account during reconciliation does NOT cross that line, because lldap is not the thing being deleted from — Matrix/Forgejo are downstream mirrors of lldap's state, and keeping a mirror accurate is inherently bidirectional (create what's missing, remove what's stale) regardless of which direction a given correction happens to point. The additive/subtractive split is specifically about who may act on the **directory** (lldap) — Jostoph disables/rotates there, Vedanta never does. Reconciling other systems against the directory's current truth is a different privilege domain entirely, and stating it here so a future decomposition doesn't misread "Vedanta deletes a Matrix user" as scope creep into Jostoph's territory.

agent.teyla commented

2026-08-09 20:19:21 +00:00

Author

Collaborator

Addendum: lldap group → OpenBao policy mapping is native for humans, but NOT for agents — Vedanta has to do it by hand

Follow-up to the scope summary above, after checking OpenBao's actual LDAP auth method docs (2026-08-09).

Confirmed native mechanism: bao write auth/ldap/groups/<name> policies=foo,bar maps an lldap group directly to an OpenBao policy list — no code needed. But this only fires for principals that authenticate through OpenBao's LDAP auth method, which requires a real password (OpenBao performs a second bind as the user, with the user's own password, to verify identity — confirmed from the docs, not assumed).

That means it's real and free for humans (e.g. lldap group membership could natively gate who's even allowed to mint the monthly dead-man-switch token — Ting/Ting#5). It does not apply to agent/service accounts, which are deliberately passwordless (owner ruling, this session: "we will mostly create passwordless accounts that can't log in a authelia window") and therefore can never successfully bind — LDAP simple-bind auth requires a password to succeed, full stop, regardless of which side (OpenBao or the caller) performs it. Passwordless principals have to use AppRole (or similar) instead, and AppRole entities never go through auth/ldap — no group lookup happens for them at all.

Concrete addition to Vedanta's per-identity registration step (the "register a matching OpenBao entity + entity alias" item from the scope summary above): for agent/service principals specifically, registration also has to read the identity's lldap group memberships and attach the corresponding OpenBao policies to its entity directly — doing by hand, at mint time, what auth/ldap/groups/* does automatically for principals that have a password. This is not a separate ticket item, it's a detail of the same unbuilt registration step, worth stating explicitly so it isn't assumed to be native/free the way the human case is.

## Addendum: lldap group → OpenBao policy mapping is native for humans, but NOT for agents — Vedanta has to do it by hand Follow-up to the scope summary above, after checking OpenBao's actual LDAP auth method docs (2026-08-09). **Confirmed native mechanism**: `bao write auth/ldap/groups/<name> policies=foo,bar` maps an lldap group directly to an OpenBao policy list — no code needed. But this only fires for principals that authenticate *through OpenBao's LDAP auth method*, which requires a real password (OpenBao performs a second bind as the user, with the user's own password, to verify identity — confirmed from the docs, not assumed). That means it's real and free for **humans** (e.g. lldap group membership could natively gate who's even allowed to mint the monthly dead-man-switch token — `Ting/Ting#5`). It does **not** apply to **agent/service accounts**, which are deliberately passwordless (owner ruling, this session: "we will mostly create passwordless accounts that can't log in a authelia window") and therefore can never successfully bind — LDAP simple-bind auth requires a password to succeed, full stop, regardless of which side (OpenBao or the caller) performs it. Passwordless principals have to use AppRole (or similar) instead, and AppRole entities never go through `auth/ldap` — no group lookup happens for them at all. **Concrete addition to Vedanta's per-identity registration step** (the "register a matching OpenBao entity + entity alias" item from the scope summary above): for agent/service principals specifically, registration also has to **read the identity's lldap group memberships and attach the corresponding OpenBao policies to its entity directly** — doing by hand, at mint time, what `auth/ldap/groups/*` does automatically for principals that have a password. This is not a separate ticket item, it's a detail of the same unbuilt registration step, worth stating explicitly so it isn't assumed to be native/free the way the human case is.

agent.teyla commented

2026-08-09 20:38:06 +00:00

Author

Collaborator

Matrix mint mechanism + the passwordless design's final shape (closing this sub-thread)

Consolidating an extended design discussion (2026-08-09) into one record. Matrix's homeserver is tuwunel, not Synapse (modules/impls/social-gateway/matrix.nix, flake-ops) — worth stating plainly since earlier in this same thread I researched Synapse's admin API before checking that, and had to walk it back.

Matrix mint: JWT auto-provisioning, not admin-API user creation

tuwunel supports POST /_matrix/client/v3/login with type org.matrix.login.jwt — headless, no browser, verified against a static key (ECDSA/ES256 supported, same scheme already used for the identity-pool's JWTs). sub becomes the Matrix localpart directly. With register_user = true (the default), the account auto-provisions on first valid JWT login — no separate admin-API create-user call needed. (tuwunel does also implement the Synapse-admin-API-compatible PUT .../users/{id} + POST .../users/{id}/login endpoints, confirmed from its own compliance page — a viable fallback, but JWT auto-provisioning is the simpler path when it applies.)

Two things this needs, not yet built:

Localpart character validity — the fleet's sub convention (a+freki, sa+librefang) has a literal +; whether that's valid in a Matrix localpart needs checking against the spec before assuming it works.
Audience-scoped tokens — aud is required once tuwunel's audience config is set. Reusing the existing aud: "modron" token for Matrix login would need that audience left unchecked, which is also the access-control mechanism (see below) — so this isn't just hygiene, it's load-bearing.

Matrix access gating: one group, two enforcement points, split by population

docs/librefang-hand-identity-schema.md's matrix-users group (already designed, orthogonal to librefang-hands) is the actual gate. tuwunel's JWT auth doesn't consult lldap groups at all, so enforcement has to happen where each population authenticates:

Humans (real Authelia/lldap passwords): tuwunel's own LDAP auth natively supports group filtering — filter = "(&(objectClass=person)(memberOf=cn=matrix,ou=groups,...))". Zero Vedanta code.
Agents/service accounts (passwordless by design): can't use LDAP auth — checked directly, both OpenBao's auth/ldap and tuwunel's LDAP auth require a real password bind, no exceptions. Gating has to happen at token issuance: Vedanta only includes aud: "matrix" in a minted token if the identity is in matrix-users. An identity outside the group is never handed a token tuwunel would accept, so register_user's auto-create never triggers for it.

The passwordless design's final shape

Settled, closing the "should agents get real passwords" thread: no. Agents stay passwordless by default — JWT/AppRole is the standing mechanism, unchanged. The only exception is Vedanta minting a transient, single-purpose password for one specific action that genuinely requires an LDAP bind (the auth/ldap-gated cases above), invalidated again immediately after that action completes. This is not a standing credential and does not reopen the "could authenticate through Authelia's login window" risk in any meaningful way — the exposure window is narrow, Vedanta-controlled, and tied to a deliberate action, not ambient.

The dead-man-switch (Ting/Ting#5) is the same mechanism at fleet-wide, emergency scale: not lazy, not gradual — an immediate invalidate-and-forced-restart specifically because the point of a dead-man-switch is guaranteeing no process anywhere still holds a stale credential, which a graceful/lazy rotation wouldn't guarantee. Expected to fire rarely, only under a genuine incident — normal credential rotation (for whatever does need periodic rotation) can be lazy/opportunistic (e.g. during low-CPU windows); the dead-man-switch specifically cannot be.

## Matrix mint mechanism + the passwordless design's final shape (closing this sub-thread) Consolidating an extended design discussion (2026-08-09) into one record. Matrix's homeserver is **tuwunel**, not Synapse (`modules/impls/social-gateway/matrix.nix`, flake-ops) — worth stating plainly since earlier in this same thread I researched Synapse's admin API before checking that, and had to walk it back. ### Matrix mint: JWT auto-provisioning, not admin-API user creation tuwunel supports `POST /_matrix/client/v3/login` with type `org.matrix.login.jwt` — headless, no browser, verified against a static key (ECDSA/ES256 supported, same scheme already used for the identity-pool's JWTs). `sub` becomes the Matrix localpart directly. With `register_user = true` (the default), the account **auto-provisions on first valid JWT login** — no separate admin-API create-user call needed. (tuwunel does also implement the Synapse-admin-API-compatible `PUT .../users/{id}` + `POST .../users/{id}/login` endpoints, confirmed from its own compliance page — a viable fallback, but JWT auto-provisioning is the simpler path when it applies.) Two things this needs, not yet built: - **Localpart character validity** — the fleet's `sub` convention (`a+freki`, `sa+librefang`) has a literal `+`; whether that's valid in a Matrix localpart needs checking against the spec before assuming it works. - **Audience-scoped tokens** — `aud` is required once tuwunel's `audience` config is set. Reusing the existing `aud: "modron"` token for Matrix login would need that audience left unchecked, which is also the *access-control* mechanism (see below) — so this isn't just hygiene, it's load-bearing. ### Matrix access gating: one group, two enforcement points, split by population `docs/librefang-hand-identity-schema.md`'s `matrix-users` group (already designed, orthogonal to `librefang-hands`) is the actual gate. tuwunel's JWT auth doesn't consult lldap groups at all, so enforcement has to happen where each population authenticates: - **Humans** (real Authelia/lldap passwords): tuwunel's own LDAP auth natively supports group filtering — `filter = "(&(objectClass=person)(memberOf=cn=matrix,ou=groups,...))"`. Zero Vedanta code. - **Agents/service accounts** (passwordless by design): can't use LDAP auth — checked directly, both OpenBao's `auth/ldap` and tuwunel's LDAP auth require a real password bind, no exceptions. Gating has to happen at token issuance: **Vedanta only includes `aud: "matrix"` in a minted token if the identity is in `matrix-users`.** An identity outside the group is never handed a token tuwunel would accept, so `register_user`'s auto-create never triggers for it. ### The passwordless design's final shape Settled, closing the "should agents get real passwords" thread: **no.** Agents stay passwordless by default — JWT/AppRole is the standing mechanism, unchanged. The only exception is Vedanta minting a **transient, single-purpose password** for one specific action that genuinely requires an LDAP bind (the `auth/ldap`-gated cases above), invalidated again immediately after that action completes. This is not a standing credential and does not reopen the "could authenticate through Authelia's login window" risk in any meaningful way — the exposure window is narrow, Vedanta-controlled, and tied to a deliberate action, not ambient. The dead-man-switch (`Ting/Ting#5`) is the same mechanism at fleet-wide, emergency scale: not lazy, not gradual — an immediate invalidate-and-forced-restart specifically because the point of a dead-man-switch is guaranteeing no process anywhere still holds a stale credential, which a graceful/lazy rotation wouldn't guarantee. Expected to fire rarely, only under a genuine incident — normal credential rotation (for whatever does need periodic rotation) can be lazy/opportunistic (e.g. during low-CPU windows); the dead-man-switch specifically cannot be.