Nix service: webhook + reconciliation timer + lldap access + OTLP #2

Merged

larandar merged 7 commits from tango/1-nix-service into fastlane

2026-08-10 03:22:44 +00:00

agent.teyla commented

2026-08-09 18:52:38 +00:00

Collaborator

Closes Ting/Vedanta#1.

Vedanta now runs as a NixOS service, not just cargo run:

vedanta serve — the webhook responder (jostoph-rs's POST /v1/evaluate and friends), kept alive by vedanta.service (Restart=on-failure).
vedanta reconcile — a single sweep, invoked by vedanta-reconcile.service (Type=oneshot) behind vedanta-reconcile.timer (OnUnitActiveSec, default 15m). No reconciliation rules exist yet — this proves the lldap wiring (connect + bind) and stops there. The owner named Matrix specifically as something that will drift silently and needs a periodic sweep, not just the webhook path.
vedanta mint-agent <name> --display-name <display> (sub = a+<name>) and vedanta mint-service <name> --display-name <display> (sub = sa+<name>) — the first real effector. principal-shape now also derives an identity.mint effect alongside its Allow conclusion; on Allow, lldap::AdminClient (lldap's GraphQL admin API, confirmed live by introspecting heimdallr's actual schema — the raw LDAP protocol has no write path for user creation) actually creates the lldap user. Scope deliberately stops at user creation: no group membership, no custom attributes, no password — CreateUserInput has no password field at all, and these accounts are meant to be passwordless (owner, 2026-08-09): they authenticate through modron's token contract, not an Authelia login window.
lldap access: src/lldap.rs — connect() (raw LDAP bind, used by reconcile) and AdminClient (GraphQL, used by mint-*), both defaulting to heimdallr's own loopback listener.
OTLP: src/telemetry.rs — a local tracing-subscriber fmt layer always installs; an OTLP span exporter additionally activates when OTEL_EXPORTER_OTLP_ENDPOINT is set (the standard OTel env var). No bundled collector-side module — Vedanta pushes its own telemetry, same posture as modron/modules/telemetry.nix's minimal otlpEndpoint knob.
DOTENV_FILE (dotenv-rs): fleet-wide secrets convention per owner ruling 2026-08-09 — sops-nix decrypts to a path, the unit sets DOTENV_FILE, the binary loads it before anything else touches the environment. Optional; unset is a legitimate state, not a misconfiguration.
flake.nix: packages.default + nixosModules.default, built from the same mkVedanta so the devShell/package/module can't drift apart. services.vedanta options: enable, user (static, not DynamicUser — sops.secrets.<>.owner needs a fixed name), listen, reconcileInterval, dotenvFile, lldap.url, telemetry.{enable,otlpEndpoint}.
just try-code-in-production-like-a-gigachad: builds this working tree, nix copys the result onto heimdallr's own Nix store over ssh, and runs it there with the real admin bind credential — decrypted and consumed entirely inside the remote shell, never crossing back to the caller. Defaults to reconcile; pass anything else vedanta takes as ARGS to run that instead. This is how everything in this PR was actually verified.

Provisional: jostoph-rs is now vendor/jostoph-rs, a path dependency instead of a git one. jo.et0.pw is private, and Nix's sandboxed fixed-output git fetch has no credentials to reach it with (nix build failed on exactly that before this). The real distribution model — deploy key, making jostoph-rs public, a shared-library host under Ting/, sourcehut as canonical upstream with --override-input for internal fastlane use — is still undecided (owner, 2026-08-09). This unblocks the ticket without deciding between them; bump by re-copying vendor/jostoph-rs from a fresh clone at the new rev.

Out of scope here (per Ting/Vedanta#1): heimdallr's actual live services.vedanta.enable = true wiring — this PR is the binary and the module it ships, not flake-ops's standing consumption of it. Everything was verified with manual nix copy + direct runs, not a deployed systemd unit.

Verification

cargo test / cargo clippy --all-targets -- -D warnings — clean.
vedanta serve by hand, on heimdallr: /healthz and /v1/manifest answer, clean shutdown on SIGINT.
vedanta reconcile, on heimdallr: actually bound against the live lldap with the real admin credential and returned the no-op result. Also verified locally: correct, distinct errors for a missing env var, an unreadable DOTENV_FILE, and a refused lldap connection; doesn't hang with OTEL_EXPORTER_OTLP_ENDPOINT set and no collector listening.
vedanta mint-service minttest --display-name "Mint Test (disposable)", on heimdallr: actually created uid=sa+minttest in the live directory (confirmed via a direct GraphQL query against heimdallr), then deleted to leave the directory clean.
nix build .#default — produces a working binary.
nix eval against a minimal nixosSystem with services.vedanta.enable = true — confirmed both units' ExecStart, the timer interval, and the environment all wire through as declared.

Closes Ting/Vedanta#1. Vedanta now runs as a NixOS service, not just `cargo run`: - **`vedanta serve`** — the webhook responder (jostoph-rs's `POST /v1/evaluate` and friends), kept alive by `vedanta.service` (`Restart=on-failure`). - **`vedanta reconcile`** — a single sweep, invoked by `vedanta-reconcile.service` (`Type=oneshot`) behind `vedanta-reconcile.timer` (`OnUnitActiveSec`, default 15m). No reconciliation rules exist yet — this proves the lldap wiring (connect + bind) and stops there. The owner named Matrix specifically as something that will drift silently and needs a periodic sweep, not just the webhook path. - **`vedanta mint-agent <name> --display-name <display>`** (`sub = a+<name>`) and **`vedanta mint-service <name> --display-name <display>`** (`sub = sa+<name>`) — the first real effector. `principal-shape` now also derives an `identity.mint` effect alongside its Allow conclusion; on Allow, `lldap::AdminClient` (lldap's GraphQL admin API, confirmed live by introspecting heimdallr's actual schema — the raw LDAP protocol has no write path for user creation) actually creates the lldap user. Scope deliberately stops at user creation: no group membership, no custom attributes, no password — `CreateUserInput` has no password field at all, and these accounts are meant to be passwordless (owner, 2026-08-09): they authenticate through modron's token contract, not an Authelia login window. - **lldap access**: `src/lldap.rs` — `connect()` (raw LDAP bind, used by `reconcile`) and `AdminClient` (GraphQL, used by `mint-*`), both defaulting to heimdallr's own loopback listener. - **OTLP**: `src/telemetry.rs` — a local `tracing-subscriber` fmt layer always installs; an OTLP span exporter additionally activates when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (the standard OTel env var). No bundled collector-side module — Vedanta pushes its own telemetry, same posture as `modron/modules/telemetry.nix`'s minimal `otlpEndpoint` knob. - **`DOTENV_FILE`** (dotenv-rs): fleet-wide secrets convention per owner ruling 2026-08-09 — sops-nix decrypts to a path, the unit sets `DOTENV_FILE`, the binary loads it before anything else touches the environment. Optional; unset is a legitimate state, not a misconfiguration. - **`flake.nix`**: `packages.default` + `nixosModules.default`, built from the same `mkVedanta` so the devShell/package/module can't drift apart. `services.vedanta` options: `enable`, `user` (static, not `DynamicUser` — `sops.secrets.<>.owner` needs a fixed name), `listen`, `reconcileInterval`, `dotenvFile`, `lldap.url`, `telemetry.{enable,otlpEndpoint}`. - **`just try-code-in-production-like-a-gigachad`**: builds this working tree, `nix copy`s the result onto heimdallr's own Nix store over ssh, and runs it there with the real admin bind credential — decrypted and consumed entirely inside the remote shell, never crossing back to the caller. Defaults to `reconcile`; pass anything else vedanta takes as ARGS to run that instead. This is how everything in this PR was actually verified. **Provisional**: `jostoph-rs` is now `vendor/jostoph-rs`, a path dependency instead of a git one. `jo.et0.pw` is private, and Nix's sandboxed fixed-output git fetch has no credentials to reach it with (`nix build` failed on exactly that before this). The real distribution model — deploy key, making `jostoph-rs` public, a shared-library host under `Ting/`, sourcehut as canonical upstream with `--override-input` for internal `fastlane` use — is still undecided (owner, 2026-08-09). This unblocks the ticket without deciding between them; bump by re-copying `vendor/jostoph-rs` from a fresh clone at the new rev. **Out of scope here** (per Ting/Vedanta#1): heimdallr's actual live `services.vedanta.enable = true` wiring — this PR is the binary and the module it ships, not flake-ops's standing consumption of it. Everything was verified with manual `nix copy` + direct runs, not a deployed systemd unit. ## Verification - `cargo test` / `cargo clippy --all-targets -- -D warnings` — clean. - `vedanta serve` by hand, on heimdallr: `/healthz` and `/v1/manifest` answer, clean shutdown on SIGINT. - `vedanta reconcile`, on heimdallr: actually bound against the live lldap with the real admin credential and returned the no-op result. Also verified locally: correct, distinct errors for a missing env var, an unreadable `DOTENV_FILE`, and a refused lldap connection; doesn't hang with `OTEL_EXPORTER_OTLP_ENDPOINT` set and no collector listening. - `vedanta mint-service minttest --display-name "Mint Test (disposable)"`, on heimdallr: actually created `uid=sa+minttest` in the live directory (confirmed via a direct GraphQL query against heimdallr), then deleted to leave the directory clean. - `nix build .#default` — produces a working binary. - `nix eval` against a minimal `nixosSystem` with `services.vedanta.enable = true` — confirmed both units' `ExecStart`, the timer interval, and the environment all wire through as declared.

agent.teyla added 5 commits

2026-08-09 18:52:38 +00:00

gitignore: target/, result — never committed, just never had a .gitignore d2e0796cd0

Caught while staging this ticket's own changes: `git add -A` picked up
the whole cargo build directory. Added before anything else lands.

vendor: jostoph-rs at 029a102c, path dependency — PROVISIONAL c6db8a6147

jo.et0.pw is private, and Nix's sandboxed fixed-output fetch for a `git`
Cargo dependency has no credentials to reach it with (no /etc/nix/netrc,
no deploy key wired in) — `nix build` on this package failed on exactly
that. Vendoring as a path dependency sidesteps it structurally: the
source is just files inside `src` by the time the sandbox sees it, no
fetch to authenticate.

Not the real distribution model — that's still open (owner, 2026-08-09):
a deploy key, making jostoph-rs public, or folding it into a shared host
under Ting/ are all live options. This unblocks today's ticket
(Ting/Vedanta#1) without deciding between them. Bump by re-copying
vendor/jostoph-rs from a fresh clone at the new rev, deliberately, same
posture the git dependency had.

gates: fix clippy field-reassign-with-default in principal-shape de92b98eff

First time this code has actually been run through clippy — its own doc
comment said as much ("not yet cargo built"). Construct
PrincipalShapeProgram with its fields set at initialization instead of
Default::default() followed by two field assignments.

serve/reconcile: split the binary on its two triggers, wire lldap + OTLP + dotenv 2d7e59a827

Two independent triggers, one binary — never one process trying to be
both. `vedanta serve` is the long-lived webhook responder (jostoph-rs's
`POST /v1/evaluate` and friends) a systemd *service* keeps alive.
`vedanta reconcile` is a single sweep a systemd *timer* invokes and lets
exit — no reconciliation rules exist yet (Ting/Vedanta#1), so it proves
the lldap wiring (src/lldap.rs: connect + bind against
VEDANTA_LLDAP_URL/BIND_DN/BIND_PASSWORD) and stops there. Conflating the
two would mean a webhook-handling process also carries a reconciliation
scheduler's failure modes — a stuck sweep wedging request handling.

src/telemetry.rs: tracing-subscriber always installs a local fmt layer;
an OTLP span exporter (tracing-opentelemetry) additionally activates when
OTEL_EXPORTER_OTLP_ENDPOINT is set — the standard OTel env var, not a
Vedanta-specific one, so the NixOS module needs no knowledge of the
fleet's telemetry topology beyond setting it. A failed exporter build
warns and falls back to local-only logging rather than refusing to
start — telemetry export failing is not a reason an identity service
should refuse to come up.

DOTENV_FILE (dotenv-rs), loaded before anything else touches the
environment: fleet-wide convention per owner ruling 2026-08-09 — sops-nix
decrypts a secret to a path, the unit sets DOTENV_FILE to it, the binary
loads it here. Optional: unset is a legitimate state (local `cargo run`
has no sops secret to point at), not a misconfiguration.

Verified against a real toolchain, not assumed: `cargo test`, `cargo
clippy --all-targets -- -D warnings`, and by hand — `serve` answers
/healthz and /v1/manifest and shuts down clean on SIGINT; `reconcile`
exits non-zero with the real cause for a missing env var, an unreadable
DOTENV_FILE, and a refused lldap connection, and does NOT hang when
OTEL_EXPORTER_OTLP_ENDPOINT points at a collector that isn't listening.

nix: services.vedanta — the webhook responder and its reconciliation timer 43224d7481

flake.nix: packages.default (rustPlatform.buildRustPackage against the
vendored dependency tree — no outputHashes needed, jostoph-rs is a path
dependency now) and nixosModules.default, built from the same mkVedanta
so the devShell, the package, and the module never drift against each
other. devShell carries forward from the bootstrap work on
tango/2-bootstrap (not yet merged to fastlane) — duplicated here rather
than depended on, since that branch isn't landed yet.

module.nix: two systemd units from one package, matching the binary's
own serve/reconcile split — vedanta.service (Restart=on-failure) and
vedanta-reconcile.service (Type=oneshot) behind a systemd.timer
(OnUnitActiveSec, default 15m, no measured requirement behind that number
yet). A static services.vedanta.user, not DynamicUser: DynamicUser's
per-activation username is unpredictable at eval time, and
services.sops.secrets.<>.owner needs a fixed name to target.

Secrets are explicitly NOT this module's decision: dotenvFile is a plain
nullOr path, and lldap.url/telemetry.otlpEndpoint are ordinary options
for the non-secret half — pointing dotenvFile at a real sops-nix secret
path is the deploying host's job (Ting/Vedanta#1's own scope note: this
ticket is the binary and the module it ships, not heimdallr's live
secrets wiring).

Verified: `nix build .#default` produces a working binary
(`vedanta --version` runs). `nix eval` against a minimal nixosSystem with
services.vedanta.enable = true confirms both units' ExecStart, the
timer's interval, and the environment (DOTENV_FILE, VEDANTA_LLDAP_URL,
OTEL_EXPORTER_OTLP_ENDPOINT) all wire through as declared — ONLY checked
by evaluating a real nixosSystem, not read off the module source by eye.

agent.teyla added 1 commit

2026-08-09 19:09:10 +00:00

just: try-code-in-production-like-a-gigachad — build, ship, run, for real 839ffa97ed

Everything up to this point had been verified against a nonexistent
lldap (connection refused) or a manual nix build + nix copy + ssh done
by hand. Neither is the same claim as "this works" — the real question
was always whether it binds against heimdallr's actual, live directory,
not a stand-in for it. This recipe is that check, repeatable: nix build
this working tree, nix copy the result onto heimdallr's own store over
ssh, run it there with the real admin bind credential — decrypted and
consumed entirely inside the remote shell, never crossing back to the
caller or landing in this recipe's own output.

Verified by running it: `reconcile` (the default) actually bound against
heimdallr's live lldap and returned the real no-reconciliation-rules-yet
no-op; overriding ARGS (`--version`, `serve`) confirmed the override path
works too.

Defaults to `reconcile`, not `serve` — serve blocks in the foreground
until Ctrl-C, and a recipe meant for a quick "does it still work" check
should return, not become a process you have to remember you started.

agent.teyla added 1 commit

2026-08-09 19:19:10 +00:00

mint: mint-agent and mint-service — the first real effector eaa1a02b38

Everything before this declared an Allow and stopped; nothing in this
binary ever actually wrote to lldap. This is the first effect that gets
executed, not just declared:

- principal-shape's ascent program now also derives an `identity.mint`
  effect (kind + `{"sub":...}` params) alongside its Allow conclusion,
  joined by (key, reason) per Derived::effects' own contract. `Derived`
  no longer discards effects with `..Default::default()`.
- src/lldap.rs: AdminClient, a session against lldap's GraphQL admin API
  — NOT the raw LDAP protocol `connect()` already used for reconcile.
  Confirmed live, by introspecting heimdallr's actual running lldap
  (Mutation type, CreateUserInput), not assumed from docs: lldap has no
  LDAP-protocol write path for user creation, and CreateUserInput has no
  password field at all — credential-setting is a separate, unbuilt flow.
  These accounts are meant to be passwordless (owner, 2026-08-09): they
  authenticate through modron's token contract, not an Authelia login
  window, so the missing password field is the correct end state here,
  not a gap to fill in later.
- `vedanta mint-agent <name> --display-name <display>` (sub = a+<name>)
  and `vedanta mint-service <name> --display-name <display>` (sub =
  sa+<name>) run the SAME principal-shape gate `serve` would — no side
  channel that skips it — and only execute the declared effect on Allow.
  `worker-` (pool slots) deliberately stays out: that's the pool-manager's
  checkout/reissue cycle, not a one-shot mint.
- reqwest pulled in via --no-default-features: this only ever talks
  http://127.0.0.1 (lldap's own loopback listener), and the default TLS
  backend (rustls + aws-lc-rs) would be dead weight in both the binary
  and the Nix build closure for a connection that is never TLS.

Scope, deliberately: user creation only — no group membership, no
custom attributes (librefang_node, pool_*), no password. Each of those
is its own undecided question already flagged in the schema docs
(default groups for a freshly minted identity, credential rotation) —
this effector executes what's actually settled and stops there, same
posture principal-shape itself took.

Verified against the real thing, not a mock: `nix build` + `nix copy` to
heimdallr + `vedanta mint-service minttest --display-name "Mint Test
(disposable)"` actually created uid=sa+minttest in the live directory
(confirmed via a direct GraphQL query), then deleted to leave the
directory clean. cargo test / clippy --all-targets -- -D warnings clean.

agent.teyla referenced this pull request

2026-08-09 20:09:32 +00:00

Nix service: webhook + reconciliation timer + lldap access + OTLP #1

agent.teyla referenced this pull request

2026-08-09 20:09:57 +00:00

Nix service: webhook + reconciliation timer + lldap access + OTLP #1