Fix cluster bring-up defects found converging the anvil nodes #11

Merged

josh merged 3 commits from feat/foundry-k3s-cluster-4057a8 into main 2026-08-06 14:39:08 -05:00

Conversation 0 Commits 3 Files changed 23 +515 -104

josh commented 2026-08-06 14:04:03 -05:00

Owner

Copy link

Everything here came out of actually converging the three nodes. Static checks — yamllint, ansible-inventory --list, --syntax-check, ansible-lint — passed on all of it.

The one that mattered

Cluster traffic was about to run over Tailscale. k3s_server_address was the hostname foundry-white, which resolves on foundry — where MagicDNS is accepted — to the node's 100.x tailnet address. Every agent connection and every flannel packet would have gone through userspace WireGuard on a Raspberry Pi: the exact outcome the LAN/tailnet split was designed to prevent, and one that would have looked like nothing worse than "the cluster feels slow".

Now pinned to the LAN address, with k3s_node_ip set per node so kubelet does not choose among the Tailscale, Docker bridge and CNI addresses these hosts also carry.

Environment findings

Both Pis were multi-homed — Ethernet and Wi-Fi up on the same subnet, each with a default route. That gives kubelet and flannel a second address to guess wrong about and causes ARP flux, while providing no useful failover (same subnet, same switch, same upstream). The raspberry_pi role now blacklists the Wi-Fi driver rather than editing netplan, because the wifis stanza lives in a cloud-init-owned file that may be rewritten.

Addresses now match the DHCP reservations: foundry 10.10.10.10, foundry-white 10.10.10.20, foundry-black 10.10.10.30.

Bugs static checks could not catch

Bug	Why it slipped through
Origins-Pattern used Allowed-Origins syntax	apt-config dump reported the config as present and correct while unattended-upgrade crashed on startup with a ValueError. The dry-run check caught it — which is exactly why that check exists.
ansible_managed in copy: content:	Only the template action plugin injects it. Three tasks failed at runtime.
systemd-timesyncd assumed	Ubuntu 26.04 ships chrony. The role now detects the daemon; what it gates on is the kernel's synchronised state, which timedatectl reports identically for either.
Tailscale apt source conflict	The install script leaves a legacy one-line source whose Signed-By differs from the deb822 source this role writes. Two entries for one repository with different keyrings makes apt refuse to read any source list.
Missing journald drop-in directory	Does not exist on a fresh image.

A defect in my own guard logic

The delegated agent-readiness check and the node-label reconciler tested whether the server was in ansible_play_hosts_all. But playbooks/k3s.yml converges the server in an earlier play, so that was false during normal push runs too — both skipped silently on every run. Agents were never verified cluster-side and labels were never reconciled.

They now test inventory group membership, which is what actually differs under ansible-pull: inventory-pull.yml deliberately has no k3s_server group at all.

Traefik moves wholly to anvil

Rather than foundry owning the bundled chart while anvil owns everything about it, K3s's Traefik stays disabled and anvil installs the ingress controller outright. No split ownership, no HelmChartConfig reaching across the boundary.

foundry keeps the svccontroller.k3s.cattle.io/enablelb label on foundry-black. That is what confines ServiceLB and protects Caddy's ports — a property of the node, not of the workload. With Traefik disabled it matters more, since the label is now the only thing standing between an anvil deployment and foundry's :80/:443. It is already applied, so it is in place before anvil can create its first LoadBalancer Service.

The durable rule, now stated in the README: foundry owns where things may run; anvil owns what runs.

kubectl on all three nodes

Each node gets a managed kubeconfig and an ~/.kube/config symlink. Two per-node differences, both deliberate:

• The Pis point at the server's LAN address, not the MagicDNS name — they run --accept-dns=false to keep tailscaled's resolver away from CoreDNS, so MagicDNS does not resolve there. It also means kubectl on a Pi keeps working when Tailscale is down.
• The server's admin kubeconfig is group-readable by the owner. On the Pis kubectl is a symlink to the k3s binary, and that wrapper prefers /etc/rancher/k3s/k3s.yaml over ~/.kube/config whenever it exists — then hard-fails if it cannot read it. Root-only ownership made kubectl unusable on the server despite a perfectly good kubeconfig in the owner's home directory. This grants cluster-admin to an account that already has passwordless sudo on that node, so it is not an escalation.

One piece of cosmetic noise is documented rather than fixed: the k3s wrapper also reads /etc/rancher/k3s/config.yaml, which stays 0600 because it holds the join token, so every kubectl call on a Pi prints a permission warning to stderr. Results and exit codes are unaffected.

Verified state

All three nodes Ready, correct internal IPs, no taints, enablelb=true on foundry-black and nowhere else, zero Traefik pods. playbooks/k3s.yml, playbooks/raspberry-pi.yml, playbooks/auto-updates.yml and playbooks/tailscale.yml all converge to changed=0 on a repeat run.

Everything here came out of actually converging the three nodes. Static checks — yamllint, `ansible-inventory --list`, `--syntax-check`, `ansible-lint` — passed on all of it. ## The one that mattered **Cluster traffic was about to run over Tailscale.** `k3s_server_address` was the hostname `foundry-white`, which resolves *on foundry* — where MagicDNS is accepted — to the node's `100.x` tailnet address. Every agent connection and every flannel packet would have gone through userspace WireGuard on a Raspberry Pi: the exact outcome the LAN/tailnet split was designed to prevent, and one that would have looked like nothing worse than "the cluster feels slow". Now pinned to the LAN address, with `k3s_node_ip` set per node so kubelet does not choose among the Tailscale, Docker bridge and CNI addresses these hosts also carry. ## Environment findings **Both Pis were multi-homed** — Ethernet and Wi-Fi up on the same subnet, each with a default route. That gives kubelet and flannel a second address to guess wrong about and causes ARP flux, while providing no useful failover (same subnet, same switch, same upstream). The `raspberry_pi` role now blacklists the Wi-Fi driver rather than editing netplan, because the `wifis` stanza lives in a cloud-init-owned file that may be rewritten. Addresses now match the DHCP reservations: `foundry` 10.10.10.10, `foundry-white` 10.10.10.20, `foundry-black` 10.10.10.30. ## Bugs static checks could not catch | Bug | Why it slipped through | |---|---| | `Origins-Pattern` used `Allowed-Origins` syntax | `apt-config dump` reported the config as present and correct while `unattended-upgrade` crashed on startup with a `ValueError`. The dry-run check caught it — which is exactly why that check exists. | | `ansible_managed` in `copy: content:` | Only the `template` action plugin injects it. Three tasks failed at runtime. | | `systemd-timesyncd` assumed | Ubuntu 26.04 ships `chrony`. The role now detects the daemon; what it gates on is the kernel's synchronised state, which `timedatectl` reports identically for either. | | Tailscale apt source conflict | The install script leaves a legacy one-line source whose `Signed-By` differs from the deb822 source this role writes. Two entries for one repository with different keyrings makes apt refuse to read *any* source list. | | Missing journald drop-in directory | Does not exist on a fresh image. | ## A defect in my own guard logic The delegated agent-readiness check and the node-label reconciler tested whether the server was in `ansible_play_hosts_all`. But `playbooks/k3s.yml` converges the server in an *earlier play*, so that was false during normal push runs too — both **skipped silently on every run**. Agents were never verified cluster-side and labels were never reconciled. They now test inventory group membership, which is what actually differs under `ansible-pull`: `inventory-pull.yml` deliberately has no `k3s_server` group at all. ## Traefik moves wholly to `anvil` Rather than `foundry` owning the bundled chart while `anvil` owns everything about it, K3s's Traefik stays disabled and `anvil` installs the ingress controller outright. No split ownership, no `HelmChartConfig` reaching across the boundary. `foundry` keeps the `svccontroller.k3s.cattle.io/enablelb` label on `foundry-black`. That is what confines ServiceLB and protects Caddy's ports — a property of the node, not of the workload. With Traefik disabled it matters *more*, since the label is now the only thing standing between an `anvil` deployment and foundry's :80/:443. It is already applied, so it is in place before `anvil` can create its first `LoadBalancer` Service. The durable rule, now stated in the README: **`foundry` owns where things may run; `anvil` owns what runs.** ## kubectl on all three nodes Each node gets a managed kubeconfig and an `~/.kube/config` symlink. Two per-node differences, both deliberate: - **The Pis point at the server's LAN address**, not the MagicDNS name — they run `--accept-dns=false` to keep tailscaled's resolver away from CoreDNS, so MagicDNS does not resolve there. It also means kubectl on a Pi keeps working when Tailscale is down. - **The server's admin kubeconfig is group-readable by the owner.** On the Pis `kubectl` is a symlink to the k3s binary, and that wrapper prefers `/etc/rancher/k3s/k3s.yaml` over `~/.kube/config` whenever it exists — then hard-fails if it cannot read it. Root-only ownership made kubectl unusable on the server despite a perfectly good kubeconfig in the owner's home directory. This grants cluster-admin to an account that already has passwordless sudo on that node, so it is not an escalation. One piece of cosmetic noise is documented rather than fixed: the k3s wrapper also reads `/etc/rancher/k3s/config.yaml`, which stays `0600` because it holds the join token, so every `kubectl` call on a Pi prints a permission warning to stderr. Results and exit codes are unaffected. ## Verified state All three nodes `Ready`, correct internal IPs, no taints, `enablelb=true` on `foundry-black` and nowhere else, zero Traefik pods. `playbooks/k3s.yml`, `playbooks/raspberry-pi.yml`, `playbooks/auto-updates.yml` and `playbooks/tailscale.yml` all converge to `changed=0` on a repeat run.

josh added 1 commit 2026-08-06 14:04:03 -05:00

Fix cluster bring-up defects found converging the anvil nodes ... 2409844223

Everything here came out of actually running the playbooks against the three
nodes. Static checks passed on all of it.

Cluster traffic was about to run over Tailscale. k3s_server_address was the
hostname `foundry-white`, which resolves on foundry - where MagicDNS is accepted -
to the node's 100.x tailnet address. Every agent connection and every flannel
packet would have gone through userspace WireGuard on a Raspberry Pi, which is
the exact outcome the LAN/tailnet split was meant to avoid. It is now the LAN
address, and k3s_node_ip is pinned per node so kubelet does not choose among the
Tailscale, Docker bridge and CNI addresses these hosts also carry.

Both Pis were multi-homed, with Ethernet and Wi-Fi up on the same subnet and a
default route each. That gives kubelet and flannel a second address to guess
wrong about and causes ARP flux, without providing useful failover. The
raspberry_pi role now blacklists the Wi-Fi driver rather than editing netplan,
since the wifis stanza lives in a cloud-init-owned file that may be rewritten.

Origins-Pattern had the wrong syntax entirely. The `Ubuntu:resolute` shorthand
belongs to Allowed-Origins; in Origins-Pattern it makes unattended-upgrade crash
on startup while `apt-config dump` still reports the configuration as present and
correct. This is precisely the failure the dry-run check exists to catch, and it
did. Allowed-Origins is now cleared as well, so the file is genuinely
authoritative rather than unioned with the distribution's defaults.

Two guards tested the wrong thing. The delegated agent-readiness check and the
node-label reconciler keyed on the server being in ansible_play_hosts_all, but
playbooks/k3s.yml converges the server in an earlier play - so both skipped
silently on every push run. They now test inventory group membership, which is
what actually differs under ansible-pull: inventory-pull.yml has no k3s_server
group at all.

Also:

- ansible_managed is injected by the template action plugin and is not available
  to copy's content. Three tasks referenced it and failed at runtime.
- Ubuntu 26.04 ships chrony, not systemd-timesyncd. The role now detects the
  daemon; what it actually gates on is the kernel's synchronised state, which
  timedatectl reports identically for either.
- Tailscale's install script leaves a legacy one-line apt source whose Signed-By
  differs from the deb822 source this role writes. Two entries for one repository
  with different keyrings makes apt refuse to read its source list at all. The
  role now removes the file it supersedes, so it can take over a host that was
  enrolled by hand - which is how the Pis were bootstrapped.
- The journald drop-in directory does not exist on a fresh image.
- Bringing the Wi-Fi link down and unloading the driver reported changed on every
  run; the module cannot be unloaded while cfg80211 holds it, so the removal now
  reports changed only on success.

Traefik stays disabled and moves wholly to anvil rather than being split across
both repositories. foundry keeps the ServiceLB node label, which is what confines
ingress to foundry-black and protects Caddy's ports - a property of the node
rather than of the workload.

kubectl now works as the owner on all three nodes. The Pis point at the server's
LAN address because --accept-dns=false means MagicDNS does not resolve there, and
the server's admin kubeconfig is group-readable by the owner because on those
nodes kubectl is a symlink to the k3s binary, and that wrapper prefers
/etc/rancher/k3s/k3s.yaml and hard-fails when it cannot read it.

josh added 1 commit 2026-08-06 14:14:44 -05:00

Store the K3s join token outside config.yaml ... 282a54c456

On nodes without a standalone kubectl - the Raspberry Pis, where the K3s
installer symlinks kubectl to the k3s binary - the wrapper parses
/etc/rancher/k3s/config.yaml on every invocation. That file had to be mode 0600
because the join token was inline, so every kubectl call printed three
`open /etc/rancher/k3s/config.yaml: permission denied` warnings to stderr.
Harmless, but noisy enough to bury the output you actually asked for.

K3s supports token-file on both server and agent, so the secret moves to
/etc/rancher/k3s/token at mode 0600 root-owned - exactly the protection it had
before, and the same protection K3s gives the copy it maintains itself. With no
secret left in config.yaml, that file becomes 0644 and the wrapper reads it
cleanly.

This is preferable to the alternative of installing a standalone kubectl on the
Pis purely to sidestep the wrapper: no extra binary to keep in step with the
cluster version, and the config file being readable is useful in its own right.

no_log moves from the config template to the token task, so --diff stays
informative for configuration changes while still never printing the secret.

Rolled out to foundry-black first to keep a mistake away from the control plane,
then to the rest. Verified afterwards: all three nodes Ready, etcd healthy, the
ServiceLB label still on foundry-black alone, no Traefik pods, and
playbooks/k3s.yml converging to changed=0 across the cluster.

josh added 1 commit 2026-08-06 14:37:01 -05:00

Make the K3s server delegation safe when the group is absent ... 5ae27ea59e

The hourly ansible-pull converge on foundry has been failing since the cluster
work landed:

  fatal: [foundry -> {{ groups['k3s_server'] | first }}]: FAILED! =>
    "object of type 'dict' has no attribute 'k3s_server'"

Three tasks delegate to the K3s server, and each was guarded with a `when` that
checks the group is non-empty. That guard never gets a chance: Ansible templates
delegate_to during task setup, before evaluating `when`, so referencing
groups['k3s_server'] raises on any run where the group does not exist -
regardless of how carefully the task is gated. inventory-pull.yml deliberately
omits the group, which makes ansible-pull exactly that case.

The target is now resolved once in tasks/main.yml, into k3s_server_available and
k3s_server_host, with the host falling back to inventory_hostname when the group
is missing. That value is always templatable, and every task using it stays gated
on k3s_server_available so the fallback never actually runs. Two fail_msg strings
referenced the group directly as well; one of them sat on an assert that is not
gated, so it would have raised while reporting an unrelated failure.

Worth noting what this cost while it was broken. The pull run reached ok=278
before dying on the last task, so it applied most of a converge from the mirror -
including rewriting foundry's K3s config from the pre-fix code, pointing the
agent at the server by hostname. On foundry that hostname resolves through
MagicDNS to the tailnet address, so each hourly run quietly moved the agent onto
WireGuard and restarted it, until the next push run put it back.

Verified by running the real thing on foundry: ansible-pull's exact invocation
against inventory-pull.yml, both playbooks/k3s.yml and the full foundry.yml, with
the whole repo present. Both complete with failed=0, and the k3s play reports
changed=0. The server-dependent tasks skip cleanly - 23 skipped in pull mode
against 19 in push mode, which is the difference showing up where it should.

josh merged commit 9ae213a0d9 into main 2026-08-06 14:39:08 -05:00

josh referenced this pull request from a commit 2026-08-06 14:39:09 -05:00

Merge pull request 'Fix cluster bring-up defects found converging the anvil nodes' (#11) from feat/foundry-k3s-cluster-4057a8 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

josh/foundry!11

No description provided.