Foundry

Foundry is the infrastructure and bootstrap repository for a personal Ubuntu homelab. Ansible provisions the hosts, their storage and networking, and the three-node anvil K3s cluster that spans them. Existing Docker service configuration remains here while those workloads are migrated to Kubernetes.

Note: the copy on GitHub is an intentional read-only mirror. Primary development and CI run on a self-hosted Forgejo platform at git.emkaytec.com. The foundry host's ansible-pull converges from this public GitHub mirror so it does not depend on the Forgejo server being reachable.

Foundry and Anvil

The homelab is split across two repositories with a deliberate boundary:

Repository	Owns
foundry	Ubuntu host configuration, users, Tailscale, disks and mounts, system packages, unattended security updates, K3s installation and lifecycle, cluster topology and node labels, and transitional Docker Compose workloads
anvil	Kubernetes desired state, including namespaces, RBAC, Argo CD applications, Helm releases, ingress routing, application workloads, and Crossplane providers and configurations

The dividing line that keeps this unambiguous is foundry owns where things may run; anvil owns what runs.

Ingress is the clearest worked example. K3s's packaged Traefik is disabled, so anvil installs and owns the ingress controller outright - no split ownership, no HelmChartConfig reaching across the boundary to retune something foundry installed. What foundry keeps is the node label svccontroller.k3s.cattle.io/enablelb, applied only to foundry-black. That puts K3s's ServiceLB into allow-list mode, so whatever anvil deploys can bind host ports on that node and nowhere else - which is what protects Caddy's hold on ports 80 and 443 on foundry. Which physical machine may terminate public traffic is a property of the hardware, not of the workload.

The intended reconciliation flow is:

1. Ansible from foundry prepares the hosts and converges a healthy K3s cluster with foundry-white as the etcd-backed server. It does not continuously apply application manifests.
2. A bootstrap entry point from anvil installs Argo CD and registers the root application. After bootstrap, Argo CD manages itself and the rest of the cluster from anvil.
3. anvil follows the same Forgejo-primary, GitHub-read-only mirror pattern as this repository. Argo CD reads the GitHub mirror so restoring the cluster does not depend on the self-hosted Forgejo service already being available.
4. Crossplane runs inside K3s and reconciles cloud resources in AWS, GCP, and Azure from definitions in anvil. Provider credentials and other secrets are supplied out of band and are never committed to either repository.

Docker Compose services and Caddy remain managed by foundry until each workload has a Kubernetes replacement ready in anvil. Each migration should coordinate the Kubernetes deployment, persistent data, ingress cutover, and removal of the old Compose service so ownership never becomes ambiguous.

Cluster Topology

Node	Hardware	K3s role	Root storage	Also does
foundry-white	Raspberry Pi, Ubuntu Server 26.04 arm64	server (--cluster-init, embedded etcd)	1 TB USB SSD	nothing else - a dedicated control plane
foundry	x86_64, Ubuntu 26.04	agent	200 GiB NVMe root, plus a 120 GiB logical volume for K3s data	Docker Compose workloads, Caddy on ports 80 and 443
foundry-black	Raspberry Pi, Ubuntu Server 26.04 arm64	agent	microSD	cluster ingress - the only node ServiceLB may bind ports on

Each placement is deliberate:

• etcd runs on the SSD-backed node. etcd is the one genuinely write-intensive component in the cluster, and a microSD card cannot sustain its fsync-per-transaction pattern for long.
• Ingress runs on the microSD-backed node, which is fine precisely because it is an agent. Agents run no etcd; their steady-state writes are container image layers pulled once, container logs capped at the kubelet, and a few kilobytes of pod state. Only foundry-black carries the svccontroller.k3s.cattle.io/enablelb label, which puts K3s's ServiceLB into allow-list mode so no load balancer pod is ever scheduled onto foundry.
• Caddy keeps ports 80 and 443 on foundry. The router still forwards to foundry, and the existing *.emkaytec.com sites are untouched. Cluster ingress is reachable over the LAN and the tailnet, and can later be fronted by adding a caddy_sites entry pointing at foundry-black:80.
• All three nodes are schedulable. The server carries no control-plane taint.

What This Manages

• Baseline Ansible connectivity for every managed host
• Apt mirror pinning for faster Ubuntu package installs
• General-purpose system packages such as ffmpeg
• Human user accounts, shell setup, SSH access, and developer CLIs
• Tailscale enrollment and advertised subnet routes
• Thunderbolt-attached storage mounted at /mnt/store
• Tailnet-only SMB sharing from /mnt/store/share
• Raspberry Pi board configuration: memory cgroup, swap, clock, and microSD write reduction
• Unattended security updates on the Raspberry Pi nodes
• The three-node anvil K3s cluster - server and agent roles, the shared join token, node labels, and a dedicated internal-NVMe logical volume for foundry's K3s data
• K3s host access and cluster bootstrap prerequisites for the companion anvil GitOps repository
• Docker Engine and the Plex, Jellyfin, Forgejo, official Bitwarden Lite, and Caddy Compose services under docker/, with Caddy reverse proxying them at *.emkaytec.com
• Self-managed ansible-pull convergence through a systemd timer
• Explicit system package maintenance through a dedicated update playbook

Repository Layout

.
  ansible/       Ansible inventory, playbooks, roles, and operator docs
  control-node/  Image for running Ansible locally without installing it
  docker/        Compose files deployed by the Ansible Docker role

The detailed runbook lives in ansible/README.md. Start there for manual server prerequisites, collection installation, baseline provisioning, individual playbook commands, vault usage, and maintenance tasks. Local Ansible runs expect ansible/.vault-password to exist because ansible/ansible.cfg points at it; the file is ignored by git.

Common Commands

Run commands from the Ansible directory so the repo-local ansible.cfg is used:

cd ansible
ansible-playbook foundry.yml

Running Without Installing Ansible

The repo-root compose.yml builds a disposable control-node container from control-node/ so the playbooks can run from any machine with Docker, no local Ansible install required. From the repo root:

make dev                                      # interactive shell
make dev CMD="ansible-playbook foundry.yml"   # one-off run

make dev resolves the Ansible vault password from 1Password on the host (where Touch ID works) before handing off to docker compose run --rm ansible. The raw compose command works too; vaulted playbooks then just need FOUNDRY_VAULT_PASSWORD exported first (see below).

The container mounts the repo at /workspace, starts in /workspace/ansible so the repo-local ansible.cfg applies, and mounts the host's ~/.ssh read-only so the foundry-admin, foundry-white-admin and foundry-black-admin SSH aliases and known_hosts carry over. SSH auth flows through the ssh-agent socket the Docker VM forwards at /run/host-services/ssh-auth.sock — Colima and Docker Desktop both expose it at that path. The entrypoint strips macOS-only UseKeychain/IdentityAgent lines from the container's copy of the SSH config, so the 1Password agent works through that forwarding instead of its Mac socket path.

Two pieces of one-time Docker VM configuration are required (already applied to the local Colima profile in ~/.colima/default/colima.yaml):

• the repo's volume must be shared into the VM — for Colima that means a mounts: entry for /Volumes/Bolt (note that listing any custom mount replaces the defaults, so ~ and /tmp/colima must be re-listed too); for Docker Desktop, add it under File Sharing
• agent forwarding must be on — forwardAgent: true for Colima, which is what populates /run/host-services/ssh-auth.sock in the VM

The vault password is not stored on disk. ansible/.vault-password is a dual-mode helper script: when FOUNDRY_VAULT_PASSWORD is set it prints that, otherwise it reads op://Private/Foundry Ansible Vault/password through the 1Password CLI. On the Mac the op path runs directly; inside the container (which has no op) the make dev target resolves the password on the host first and passes it in through the FOUNDRY_VAULT_PASSWORD environment variable declared in compose.yml. The container entrypoint wires the helper into Ansible (ANSIBLE_VAULT_PASSWORD_FILE) only when that variable is set, so password-less sessions still run everything except vault decryption.

This compose file is purely a local development tool; the Compose files that Ansible deploys to the server live in docker/.

Run package maintenance explicitly:

cd ansible
ansible-playbook playbooks/system-updates.yml

The update playbook reports whether a reboot is required. To reboot automatically only when required:

cd ansible
ansible-playbook playbooks/system-updates.yml -e system_updates_reboot=true

It runs against every host one at a time. Because foundry-white currently holds the only etcd member, use --limit to target a single node rather than rebooting the cluster in one pass.

This is the deliberate path, and it is separate from the automatic one. The Raspberry Pi nodes also run unattended-upgrades, restricted to the Ubuntu archives and configured never to reboot on their own; see ansible/README.md.

Local Private Values

Committed Ansible defaults are intentionally public-safe. Host-specific human user details belong in ansible/group_vars/all/99-private.yml, which is ignored by git and loaded by Ansible when present.

Two cluster values belong there as well, if you want kubectl to work from outside the LAN:

• k3s_api_endpoint — the address written into client kubeconfigs. It defaults to the server's LAN name; set it to the server's MagicDNS name to reach the cluster from anywhere on the tailnet.
• k3s_extra_tls_sans — every additional name or address the API server's certificate must cover. If k3s_api_endpoint is not listed here, kubectl fails TLS verification.