Decommission mesh0 — full OIDC trust model replaces wireguard membership #32
Labels
No labels
blocked
conflicted
curated
in-progress
in-review
needs-decision
No milestone
No project
No assignees
1 participant
No due date set.
Dependencies
No dependencies set
Reference
larandar/flaky-mesh#32
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stable ID: FM-MESH-DECOMMISSION
Source: owner ruling 2026-08-06
Source status: ratified decision, not yet tracked
Disposition: open — epic, needs decomposition
Blocked by: nothing (but every step inside it blocks on prior steps)
Decision
Wireguard mesh0 is dead. Full OIDC.
Owner ruling 2026-08-06, fleet-wide. The mesh has been dying for months
(keepalive-only, packet loss on mobile, bad on mobile generally). The
direction is now star-and-pull with public
et0.pwnames — services reachclients over public DNS, authenticated through Authelia OIDC, not through
wireguard mesh membership.
This is the existing memory direction ("star-and-pull, public et0.pw names")
made canonical: mesh0 stops being the trust boundary.
What this means concretely
Trust boundary shifts
wg0, firewall gates on the mesh interface.*.meshDNS names resolve onlyinside mesh0.
gated by Authelia
forward_auth(or equivalent).et0.pwnames resolvepublicly.
What dies
policies/mesh.nix— wireguard interface, peer config, keepalivecontracts/mesh-nodes.nix— the mesh registryintegrations/mesh-dns.nix—*.meshDNS resolveroutputs/hosts/*/secrets.yaml—wireguard/private_keyentriesaspects.policies.mesh.privateKeyFilereferencenetworking.firewall.interfaces.${mesh_interface}ruleWhat replaces it
browser-facing service. Already partially in place — heimdallr runs
Authelia, and
policies/identityexists.et0.pw) replaces*.meshnames. Cloudflare proxycertificates already cover the vault names (owner ruling from the audit).
forward_authreplaces mesh-interface firewall scoping forservices that need authentication. (Exception:
git cloneover HTTPS —see backlog.md, Authelia-as-Forgejo-login-source.)
from
et0.pwrather than reaching a mesh address.Scope of THIS epic
This is a tracking issue. Decomposition happens in sub-tickets. Rough phases:
OIDC readiness. Authelia on heimdallr must be production-grade before
anything stops trusting mesh0. The deferred Authelia-as-forgejo-login-source
(backlog.md) unblocks. Caddy
forward_authneeds to be a working pattern.Per-host migration. Each host switches from mesh-interface firewall
scoping to OIDC-gated public ingress. This is the natural unit of work —
one host at a time, each verifiable.
Mesh teardown. Once every host has migrated, the wireguard interface,
peer config, mesh DNS, and mesh secrets are removed fleet-wide.
Documentation. The mesh decommission and the OIDC trust model need a
wiki page. The deploy law and the migration plan both reference mesh0 as a
live assumption — those need updating.
Known interactions
OIDC-gated public ingress. First host to adopt the new posture.
transferred.
reaching bifrost's homeserver. That path changes too.
not deferred work.
blanket passwordless sudo on servers becomes higher-risk (public ingress).
Open questions
before any more feature work), or a gradual one (each host migrates when
touched)? The owner's phrasing suggests decided, not exploratory.
just sshresolves viatofu). With mesh0 gone, SSH moves to public addresses + key-only auth.
Does the justfile's
known_hostsmodel survive?mesh0. Public scraping needs authentication or a different transport.
mesh0 gone, the key rotation/trust model may need adjustment.
Acceptance
policies/mesh,contracts/mesh-nodes,integrations/mesh-dnsdeleted*.meshDNS name resolvesjust checkgreenProvenance
Owner ruling 2026-08-06, stated during modron deploy-rs user story
refinement. Consistent with the existing memory direction ("star-and-pull,
public et0.pw names"), now made the canonical fleet direction.
Wireguard mesh0 status (from memory): keepalive-only, packet loss on mobile,
bad on mobile generally. The mesh has been functionally dead for
communication; this makes it formally dead for trust.
Owner ruling 2026-08-06: timeline and difficulty
Gradual, but there isn't much. The fleet is small and most services are
trivially behind
forward_auth. Two are hard:The two hard migrations
Matrix (tuwunel on bifrost)
/.well-known/matrix/*delegation must survive the move frommesh-only to public. A delegation that serves 200 with subtly wrong JSON
fails federation silently — the same trap DEFERRED.md already flags.
mesh0 (
0.0.0.0:6167, plain HTTP, mesh-is-the-encryption). Moving to publicmeans TLS + a public hostname (
matrix.et0.pwalready exists).@freki:et0.pw,@geri:et0.pw) assumedmesh-only plain HTTP to bifrost:6167. That path disappears.
Forgejo HTTPS clone
git cloneover HTTPS is performed by git, not a browser — it cannotfollow a redirect to a login page. Forward_auth breaks git over HTTPS
(backlog.md, DEFERRED.md both flag this).
forward_auth). Authelia OIDC → Forgejo
oauth2auth source. Web UI goesthrough Authelia; git over HTTPS keeps using Forgejo tokens.
of the mesh decommission, not deferrable work.
The easy rest
Everything else browser-facing is
forward_authagainst Authelia:dashboards, whoami, victoria, librefang, the Caddy-served sites. One pattern,
applied per host. The hard part is already built — Caddy supports
forward_authnatively, and heimdallr already runs Authelia.Updated phasing
pattern proven.
the hard one first.
well-known delegation verified.
forward_auth, gradual.What this settles from the open questions
"indefinitely deferred." The two hard ones get explicit tickets.
DEFERRED.md to blocked-on-nothing.
Sub-tickets to file
Owner question 2026-08-06: SSH discovery without mesh0
The easy-seeming questions (DNS, forward_auth, TLS) are settled or scoped.
This one is genuinely open: how does
just sshfind a distant node'saddress when mesh0 is gone?
The diagnostic that settles it
Are vpsFree public addresses stable?
Today
just sshreads the address from OpenTofu state (tofu output -raw address). That works because tofu tracks what vpsAdmin granted. The questionis whether that address survives reboots, migrations, or vpsFree-side changes.
If stable (static public IPv4/IPv6): DNS suffices.
bifrost.et0.pwpoints at the public address;
ssh [email protected]. The tofu state andknown_hostsmodel survive unchanged — just the address type changes frommesh to public. One-time DNS record per host.
If unstable (SLAAC with privacy extensions, or address rotation): DNS
goes stale on every change. A discovery system is needed — dynamic DNS, a
coordination service, or the host publishing its current address somewhere
queryable.
This is a five-minute check against a running host — does the tofu-recorded
address match
ip addrafter a reboot? Owner or Fizz can answer it; I can'treach the hosts from here.
Options, if discovery is needed
Recorded so the choice has a named surface, not to decide now:
Dynamic DNS. Each host updates its own
et0.pwrecord on boot (or onaddress change). ddclient, a NixOS systemd service, or a Caddy-level
hook. Simple, no new infrastructure, depends on a DNS API.
OpenTofu as discovery. tofu already tracks the address. If it stays
authoritative,
just sshkeeps working as-is — tofu output is thediscovery query. This is the current mechanism; the question is whether
it's stable enough to rely on without a mesh fallback.
A coordination/mesh-discovery service. Something like a lightweight
registry where hosts publish their current address and clients query it.
Tailscale's coordination server does this; a bare version is a single HTTP
endpoint returning
{host: address}. Overkill if a simpler option works.Magic DNS / overlay. A Tailscale/Headscale/Zerotier-style overlay that
gives each host a stable name regardless of its underlying address. This
is essentially rebuilding mesh0 with a better protocol. May be the right
answer if mobile client connectivity also needs it (the original mesh0
failure was on mobile).
Why this isn't just an SSH question
SSH is the first consumer, but not the only one. Monitoring scraping
(VictoriaMetrics reaching exporters), secrets transport (sops using the host's
SSH key), and any agent reaching a distant service all depend on knowing where
a host is. If discovery is needed for SSH, it's needed for those too.
Not decided here
This stays an open question on #32 until the stability diagnostic is run. The
mechanism choice follows the answer — don't pick a discovery system before
knowing whether one is needed.
Filed as open question #2 (SSH), refined
The original Q2 on #32 asked "does the known_hosts model survive?" This
refines it: known_hosts survives any mechanism that produces a stable address.
The question is upstream of known_hosts — it's about address stability.
Owner refinement 2026-08-06: CF proxy kills DNS-for-SSH
The DNS answer is wrong, and not because addresses might be unstable.
CF proxy hides origin IPs by design.
dig bifrost.et0.pwreturnsCloudflare edge IPs, not the host's real address. CF only proxies HTTP(S),
not raw TCP — so you cannot SSH to a CF-resolved address. Ever.
And when CF itself is unreachable (owner: "sometimes we need alternative
access"), DNS resolution fails entirely. CF becomes a single point of failure
for any path that depends on it.
What this eliminates
mode, exposing the real IP) works mechanically, but defeats the proxy
protection for that hostname and leaks the origin IP.
browser traffic) because CF proxies those. It does not work for SSH,
monitoring scraping, sops key trust, or any non-HTTP TCP.
What this means for the remaining options
The discovery question is now sharper: non-HTTP TCP needs a path that
bypasses CF entirely. That's SSH, VictoriaMetrics scraping, and any
agent-to-service reachability that isn't HTTP.
The surviving options, refined:
OpenTofu state as the primary discovery. Tofu already tracks the real
address, bypasses CF, and is the current mechanism. If vpsFree addresses
are stable (still the open diagnostic), tofu is both the primary and the
CF-independent fallback.
just sshalready uses this path and it works.Host-published address, not through CF. Each host publishes its real
address somewhere queryable that doesn't depend on CF — a lightweight
registry endpoint, a DNS TXT record on a non-proxied zone, or an append
to a known location (Forgejo, a status file). This is the "alternative
access" the owner is asking about: when CF is down, how do you find the
host?
Overlay network (Tailscale/Headscale/Wireguard-as-discovery). Gives
each host a stable address that doesn't depend on DNS or CF at all. This
is rebuilding mesh0 with better properties — the original mesh0 failure
was mobile clients and packet loss, not the concept of an overlay. An
overlay that handles mobile well (Tailscale's DERP relays, for instance)
could serve both SSH discovery and mobile client connectivity.
The real question
It's not "do we need discovery" — CF proxy means we do, for everything
that isn't HTTP(S). The question is: is tofu state reliable enough to be
the sole discovery path, or do we need a second independent channel?
The owner's "sometimes we need alternative access" suggests tofu alone is
not enough — there are scenarios where neither CF DNS nor the current
tofu-tracked address gets you in. Those scenarios need naming before picking
a mechanism.
Owner decision 2026-08-06: Headscale
Freenet evaluated (interesting talk — hole-punching, no coordination server),
but it's a content-routing platform, not a TCP tunnel. Wrong shape for this.
Headscale. Self-hosted Tailscale control server. Replaces mesh0's role
(stable addresses for non-HTTP TCP, mobile-friendly) with better properties
(DERP relay fallback, battle-tested NAT traversal, NixOS first-class module).
Resolves open question #2 on this issue: discovery without CF for SSH,
monitoring scraping, sops transport, and mobile clients. Headscale gives every
host a stable
100.x.y.zaddress that bypasses CF entirely.Filed as #35.