Build the anvil K3s cluster across foundry and two Raspberry Pis #10

Merged

josh merged 1 commit from feat/foundry-k3s-cluster-4057a8 into main 2026-08-06 13:04:52 -05:00

Conversation 0 Commits 1 Files changed 48 +2429 -301

josh commented 2026-08-06 13:03:24 -05:00

Owner

Copy link

Replaces the single-node K3s server on foundry with a three-node cluster named anvil, spanning foundry and the new Raspberry Pi hosts foundry-white and foundry-black.

Topology

Node	Hardware	K3s role	Root storage	Also does
foundry-white	Raspberry Pi, Ubuntu Server 26.04 arm64	server (--cluster-init, embedded etcd)	1 TB USB SSD	dedicated control plane
foundry	x86_64, Ubuntu 26.04	agent	NVMe root + 120 GiB LV for K3s data	Compose workloads, Caddy on :80/:443
foundry-black	Raspberry Pi, Ubuntu Server 26.04 arm64	agent	microSD	cluster ingress

All three nodes run Ubuntu, so this needs no OS-family compatibility work — the existing distribution == "Ubuntu" asserts all still pass and the Tailscale role is unchanged. Only the k3s role's architecture clause was widened.

etcd lives on the SSD-backed node because its fsync-per-transaction pattern is the one workload a microSD card cannot sustain. Ingress lives on the microSD node precisely because it is an agent: no etcd, and klipper-lb is pure DNAT.

The part that carries real risk

Traefik now ships with the cluster, the opposite of the previous setup. That is safe only because ServiceLB is confined to foundry-black via the svccontroller.k3s.cattle.io/enablelb label. An unlabelled cluster treats every node as eligible, and a klipper-lb pod on foundry installs a DNAT rule in the host's nat PREROUTING chain that intercepts packets before they reach Caddy's listening socket.

That failure is silent — every service keeps reporting healthy while plex/git/bitwarden.emkaytec.com go dark. Three independent mitigations: k3s_traefik_enabled=false for the initial build, foundry-black joins first so it is labelled before foundry appears, and the documented rebuild sequence gates on a label check.

Other notable decisions

• Intra-cluster traffic uses LAN addresses; only operator kubeconfigs use Tailscale. Routing the cluster over the tailnet would push every pod-to-pod packet through userspace WireGuard on a Pi.
• Inventory gains three orthogonal group axes — OS family, board hardware, cluster role. inventory-pull.yml mirrors the structure, and the k3s role asserts group membership because drift there fails safe but silently.
• k3s.yml moves after caddy.yml so a K3s outage never stops foundry's hourly pull from converging Docker and Caddy.
• Pre-shared vault join token rather than harvesting from the server, which keeps agent convergence order-independent and works under ansible-pull, where there is no SSH path to the server. Note it is write-once: K3s derives the bootstrap encryption key from it.
• foundry is torn down and rebuilt, not converted. Its 120 GiB LV is reused — the uninstall script empties a mounted data directory while leaving the mount and fstab entry intact.

New roles

• raspberry_pi — memory cgroup, swap, clock sync, microSD write reduction. The cmdline.txt edit is a token set-union, never a substitution, because mangling that single-line vfat file makes the board unbootable. An unconditional runtime gate proves the controller is live regardless of how it got enabled.
• auto_updates — unattended-upgrades on the Pis only. Automatic reboot is off while the cluster has one etcd member and one ingress node. Verifies the effective apt configuration via apt-config dump plus a dry run, not the rendered file.
• k3s_teardown — never imported by foundry.yml, double confirmation, and a guard against the uninstall script's silent no-op when a second k3s*.service exists.

Verification

yamllint, ansible-inventory --list and syntax checks on all four playbooks pass; CI now covers the two non-imported playbooks and validates the inventory. ansible-lint reports zero findings in the new roles.

Beyond that, a throwaway harness (not committed) exercised the expressions static checks cannot reach. It caught two real bugs, both fixed here: the origins verification counted the apt config key itself rather than its entries, and the swap fstab regex keyed off the mount point rather than the filesystem-type field — which would have silently left Ubuntu's /swap.img enabled. It also confirmed the cgroup token union is idempotent, the endpoint-rewrite and noatime regexes are correct and re-runnable, and all three config.yaml variants render as valid YAML with no bare disable: key.

Archive metadata was checked against live sources rather than assumed: ports.ubuntu.com serves resolute arm64 with Origin: Ubuntu, the security pocket is Suite: resolute-security / Codename: resolute (so Ubuntu's origin:archive shorthand is the correct form), and Tailscale publishes a resolute suite with a keyring.

Before merging

group_vars/all/99-private.yml needs k3s_api_endpoint (foundry-white's MagicDNS name) and k3s_extra_tls_sans. Without them kubectl works over the LAN only; with the endpoint set but missing from the SANs, kubectl fails TLS verification.

Replaces the single-node K3s server on `foundry` with a three-node cluster named `anvil`, spanning `foundry` and the new Raspberry Pi hosts `foundry-white` and `foundry-black`. ## Topology | Node | Hardware | K3s role | Root storage | Also does | | --- | --- | --- | --- | --- | | `foundry-white` | Raspberry Pi, Ubuntu Server 26.04 arm64 | server (`--cluster-init`, embedded etcd) | 1 TB USB SSD | dedicated control plane | | `foundry` | x86_64, Ubuntu 26.04 | agent | NVMe root + 120 GiB LV for K3s data | Compose workloads, Caddy on :80/:443 | | `foundry-black` | Raspberry Pi, Ubuntu Server 26.04 arm64 | agent | microSD | cluster ingress | All three nodes run Ubuntu, so this needs no OS-family compatibility work — the existing `distribution == "Ubuntu"` asserts all still pass and the Tailscale role is unchanged. Only the k3s role's architecture clause was widened. etcd lives on the SSD-backed node because its fsync-per-transaction pattern is the one workload a microSD card cannot sustain. Ingress lives on the microSD node precisely because it is an agent: no etcd, and klipper-lb is pure DNAT. ## The part that carries real risk Traefik now ships with the cluster, the opposite of the previous setup. That is safe **only** because ServiceLB is confined to `foundry-black` via the `svccontroller.k3s.cattle.io/enablelb` label. An unlabelled cluster treats every node as eligible, and a klipper-lb pod on `foundry` installs a DNAT rule in the host's `nat` PREROUTING chain that intercepts packets *before* they reach Caddy's listening socket. That failure is silent — every service keeps reporting healthy while `plex`/`git`/`bitwarden.emkaytec.com` go dark. Three independent mitigations: `k3s_traefik_enabled=false` for the initial build, `foundry-black` joins first so it is labelled before `foundry` appears, and the documented rebuild sequence gates on a label check. ## Other notable decisions - **Intra-cluster traffic uses LAN addresses; only operator kubeconfigs use Tailscale.** Routing the cluster over the tailnet would push every pod-to-pod packet through userspace WireGuard on a Pi. - **Inventory gains three orthogonal group axes** — OS family, board hardware, cluster role. `inventory-pull.yml` mirrors the structure, and the k3s role asserts group membership because drift there fails safe but silently. - **`k3s.yml` moves after `caddy.yml`** so a K3s outage never stops foundry's hourly pull from converging Docker and Caddy. - **Pre-shared vault join token** rather than harvesting from the server, which keeps agent convergence order-independent and works under `ansible-pull`, where there is no SSH path to the server. Note it is write-once: K3s derives the bootstrap encryption key from it. - **`foundry` is torn down and rebuilt**, not converted. Its 120 GiB LV is reused — the uninstall script empties a mounted data directory while leaving the mount and fstab entry intact. ## New roles - `raspberry_pi` — memory cgroup, swap, clock sync, microSD write reduction. The `cmdline.txt` edit is a token set-union, never a substitution, because mangling that single-line vfat file makes the board unbootable. An unconditional runtime gate proves the controller is live regardless of how it got enabled. - `auto_updates` — unattended-upgrades on the Pis only. Automatic reboot is off while the cluster has one etcd member and one ingress node. Verifies the *effective* apt configuration via `apt-config dump` plus a dry run, not the rendered file. - `k3s_teardown` — never imported by `foundry.yml`, double confirmation, and a guard against the uninstall script's silent no-op when a second `k3s*.service` exists. ## Verification `yamllint`, `ansible-inventory --list` and syntax checks on all four playbooks pass; CI now covers the two non-imported playbooks and validates the inventory. `ansible-lint` reports zero findings in the new roles. Beyond that, a throwaway harness (not committed) exercised the expressions static checks cannot reach. It caught two real bugs, both fixed here: the origins verification counted the apt config key itself rather than its entries, and the swap fstab regex keyed off the mount point rather than the filesystem-type field — which would have silently left Ubuntu's `/swap.img` enabled. It also confirmed the cgroup token union is idempotent, the endpoint-rewrite and `noatime` regexes are correct and re-runnable, and all three `config.yaml` variants render as valid YAML with no bare `disable:` key. Archive metadata was checked against live sources rather than assumed: `ports.ubuntu.com` serves `resolute` arm64 with `Origin: Ubuntu`, the security pocket is `Suite: resolute-security` / `Codename: resolute` (so Ubuntu's origin:archive shorthand is the correct form), and Tailscale publishes a `resolute` suite with a keyring. ## Before merging `group_vars/all/99-private.yml` needs `k3s_api_endpoint` (foundry-white's MagicDNS name) and `k3s_extra_tls_sans`. Without them kubectl works over the LAN only; with the endpoint set but missing from the SANs, kubectl fails TLS verification.

josh added 1 commit 2026-08-06 13:03:24 -05:00

Build the anvil K3s cluster across foundry and two Raspberry Pis ... dd8e2c9353

Replace the single-node K3s server on foundry with a three-node cluster named
anvil, spanning foundry and the new Raspberry Pi hosts foundry-white and
foundry-black. All three nodes run Ubuntu 26.04, so this needs no OS-family
compatibility work: the existing Ubuntu asserts all still pass and the Tailscale
role is unchanged.

foundry-white is the server, using cluster-init and embedded etcd rather than
the default SQLite. There is one server today, but etcd is what makes promoting
additional servers possible later without rebuilding the control plane. It boots
from a USB SSD, which is why it holds the datastore - etcd's
fsync-per-transaction pattern is the one workload a microSD card cannot sustain.

foundry and foundry-black join as agents, and all three nodes stay schedulable.
foundry is torn down and rebuilt rather than converted in place: its config
predates the token and topology model the cluster needs. Its 120 GiB logical
volume is reused, because the installer's uninstall script empties a mounted
data directory while leaving the mount and fstab entry intact.

Traefik now ships with the cluster, which is the opposite of the previous setup.
That is safe only because ServiceLB is confined to foundry-black through the
svccontroller.k3s.cattle.io/enablelb label: an unlabelled cluster treats every
node as eligible, and a klipper-lb pod on foundry would install a DNAT rule that
intercepts packets before they reach Caddy's socket on ports 80 and 443. That
failure is silent - services keep reporting healthy while the public sites go
dark - so the rebuild sequence gates on the label check, the agent play joins
foundry-black first, and k3s_traefik_enabled allows the initial build to run
with ingress off.

Intra-cluster traffic uses LAN addresses and only operator kubeconfigs use
Tailscale. Routing the cluster over the tailnet would push every pod-to-pod
packet through userspace WireGuard on a Raspberry Pi.

Other notable pieces:

- Inventory gains three orthogonal group axes - OS family, board hardware and
  cluster role. inventory-pull.yml mirrors the structure, and the k3s role
  asserts group membership because drift there would otherwise fail silently.
- k3s.yml moves after caddy.yml in foundry.yml so a K3s outage never stops
  foundry's hourly pull from converging Docker and Caddy.
- The join token is a pre-shared vault secret rather than harvested from the
  server, which keeps agent convergence order-independent and works under
  ansible-pull, where there is no SSH path to the server.
- New raspberry_pi role handles the memory cgroup, swap, clock and microSD
  writes. The cmdline.txt edit is a token set-union, never a substitution, since
  mangling that file makes the board unbootable.
- New auto_updates role configures unattended-upgrades on the Pis only, with
  automatic reboot off while the cluster has a single etcd member and a single
  ingress node. It verifies the effective apt configuration rather than the
  rendered file.
- New k3s_teardown role, never imported by foundry.yml, with double confirmation
  and a guard against the uninstall script's silent no-op when a second
  k3s*.service exists.

josh merged commit d65217d3bc into main 2026-08-06 13:04:52 -05:00

josh referenced this pull request from a commit 2026-08-06 13:04:52 -05:00

Merge pull request 'Build the anvil K3s cluster across foundry and two Raspberry Pis' (#10) from feat/foundry-k3s-cluster-4057a8 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

josh/foundry!10

No description provided.