Stop roles overwriting each other's directory ownership #12

Merged
josh merged 2 commits from feat/foundry-k3s-cluster-4057a8 into main 2026-08-06 15:11:15 -05:00
Owner

Two roles were managing the same directory with different intentions, so every converge applied both and reported a change. There were two instances of it, at different paths.

/mnt/store

storage set root:root 0755; samba then set josh:foundry 2775. Beyond the noise, it left the share root owned by root for the part of every run between the two roles.

They were each right about a different thing, which is why neither looked wrong on its own:

  • storage cares about the mount point — a bare directory that should be root-owned so a failed or missing mount cannot silently accept writes onto the root filesystem with group-writable permissions.
  • samba cares about the share root — which, once the G-DRIVE is mounted, is the same path but a different inode entirely: the mounted filesystem's root, which has to be group-writable and setgid for the share to work.

storage now enforces ownership only while the path is not a mount point, and otherwise leaves the attributes alone. Both properties hold and neither role overwrites the other.

/var/log/foundry

The same conflict at a second path, found while checking the remaining churn. Both roles wanted foundry_owner:foundry_group, so they agreed on everything except the mode: rclone asked for 0750, self_pull for 2775.

2775 wins. The directory is shared rather than either role's own, so setgid keeps new files in the foundry group whichever identity writes them — a root-run process cannot leave a file the group is unable to manage. The log files themselves are 0640, so widening the directory does not widen access to their contents. 0750 is tighter and correct while the rclone timer is the only writer, but it breaks silently the moment anything else writes there.

rclone's hardcoded mode becomes rclone_backup_log_dir_mode so the value sits next to the path it applies to, and both roles' defaults now carry a comment naming the other — nothing in the repo previously recorded that these two share a directory.

Verification

Full foundry.yml converge: failed=0 on all three nodes. Foundry's churn drops from 10 changed tasks to 6, and the four that disappear are exactly the conflicting ones:

Task Before After
storage : Ensure G-DRIVE mount point exists changed every run gone
samba : Ensure Samba share directories exist changed every run gone
rclone : Ensure rclone backup log directory exists changed every run gone
self_pull : Ensure foundry state directories exist changed every run gone

Final state on the host is stable and correct — /mnt/store, /var/log/foundry and /var/lib/foundry at 2775 josh:foundry, /etc/foundry at 750 root:foundry. Cluster healthy, ServiceLB label still on foundry-black alone, git.emkaytec.com returning 200.

Not addressed

The remaining 6 changed tasks are pre-existing in developer_clis and are not conflicts — the AWS CLI re-downloading and reinstalling itself on every run (5 tasks), and npm -g @latest (1). They are tasks that cannot report accurately rather than roles fighting each other.

One related fragility surfaced while verifying, worth recording even though it is out of scope here: developer_clis makes unauthenticated GitHub API calls on every converge to check for new releases, against a 60-requests-per-hour limit shared across the whole source IP. Repeated converges exhausted it and failed a run with 403 rate limit exceeded on api.github.com/repos/sst/opencode/releases. The hourly ansible-pull spends from the same budget, so a converge can fail for reasons unrelated to the infrastructure. Authenticating those calls or caching the version check would fix it.

Two roles were managing the same directory with different intentions, so every converge applied both and reported a change. There were two instances of it, at different paths. ## `/mnt/store` `storage` set `root:root 0755`; `samba` then set `josh:foundry 2775`. Beyond the noise, it left the share root owned by root for the part of every run between the two roles. They were each right about a different thing, which is why neither looked wrong on its own: - **`storage`** cares about the *mount point* — a bare directory that should be root-owned so a failed or missing mount cannot silently accept writes onto the root filesystem with group-writable permissions. - **`samba`** cares about the *share root* — which, once the G-DRIVE is mounted, is the same path but a different inode entirely: the mounted filesystem's root, which has to be group-writable and setgid for the share to work. `storage` now enforces ownership only while the path is not a mount point, and otherwise leaves the attributes alone. Both properties hold and neither role overwrites the other. ## `/var/log/foundry` The same conflict at a second path, found while checking the remaining churn. Both roles wanted `foundry_owner:foundry_group`, so they agreed on everything except the mode: `rclone` asked for `0750`, `self_pull` for `2775`. `2775` wins. The directory is shared rather than either role's own, so setgid keeps new files in the `foundry` group whichever identity writes them — a root-run process cannot leave a file the group is unable to manage. The log files themselves are `0640`, so widening the directory does not widen access to their contents. `0750` is tighter and correct while the rclone timer is the only writer, but it breaks silently the moment anything else writes there. rclone's hardcoded mode becomes `rclone_backup_log_dir_mode` so the value sits next to the path it applies to, and both roles' defaults now carry a comment naming the other — nothing in the repo previously recorded that these two share a directory. ## Verification Full `foundry.yml` converge: `failed=0` on all three nodes. Foundry's churn drops from **10 changed tasks to 6**, and the four that disappear are exactly the conflicting ones: | Task | Before | After | | --- | --- | --- | | `storage : Ensure G-DRIVE mount point exists` | changed every run | gone | | `samba : Ensure Samba share directories exist` | changed every run | gone | | `rclone : Ensure rclone backup log directory exists` | changed every run | gone | | `self_pull : Ensure foundry state directories exist` | changed every run | gone | Final state on the host is stable and correct — `/mnt/store`, `/var/log/foundry` and `/var/lib/foundry` at `2775 josh:foundry`, `/etc/foundry` at `750 root:foundry`. Cluster healthy, ServiceLB label still on `foundry-black` alone, `git.emkaytec.com` returning 200. ## Not addressed The remaining 6 changed tasks are pre-existing in `developer_clis` and are not conflicts — the AWS CLI re-downloading and reinstalling itself on every run (5 tasks), and `npm -g @latest` (1). They are tasks that cannot report accurately rather than roles fighting each other. One related fragility surfaced while verifying, worth recording even though it is out of scope here: `developer_clis` makes **unauthenticated GitHub API calls on every converge** to check for new releases, against a 60-requests-per-hour limit shared across the whole source IP. Repeated converges exhausted it and failed a run with `403 rate limit exceeded` on `api.github.com/repos/sst/opencode/releases`. The hourly `ansible-pull` spends from the same budget, so a converge can fail for reasons unrelated to the infrastructure. Authenticating those calls or caching the version check would fix it.
Both roles managed /mnt/store and disagreed about it, so each converge applied
both and reported a change: storage set root:root 0755, samba then set
josh:foundry 2775. Beyond the noise, it left the share root owned by root for
the part of every run between the two roles.

They were each right about a different thing. storage cares about the mount
point - a bare directory that should be root-owned so a failed or missing mount
cannot silently accept writes onto the root filesystem with group-writable
permissions. samba cares about the share root, which once the G-DRIVE is mounted
is the same path but a different inode entirely: the mounted filesystem's root,
which has to be group-writable and setgid for the share to work.

storage now enforces ownership only while the path is not a mount point, and
otherwise leaves the attributes alone. Both properties hold, and neither role
overwrites the other.

Verified: playbooks/storage.yml and playbooks/samba.yml both converge to
changed=0, and /mnt/store keeps josh:foundry 2775.

A second instance of the same conflict exists at /var/log/foundry, where rclone
wants 0750 and self_pull wants 2775. Owner and group agree there, so it is a
mode-only disagreement and is left for a separate change.
Agree on the /var/log/foundry mode between rclone and self_pull
All checks were successful
CI / validate (push) Successful in 13s
b8537e904c
Second instance of the conflict fixed in the previous commit, at a different
path. Both roles create /var/log/foundry and both want it owned by
foundry_owner:foundry_group, so they agreed on everything except the mode:
rclone asked for 0750, self_pull for 2775. Each converge applied both and
reported a change, and the directory's mode depended on which role ran last.

2775 wins. The directory is shared rather than either role's own, so setgid
keeps new files in the foundry group whichever identity writes them - a
root-run process cannot leave a file the group is unable to manage. The log
files themselves are 0640, so widening the directory does not widen access to
their contents. 0750 would be tighter and is correct while the rclone timer is
the only writer, but it breaks silently the moment anything else writes there.

rclone's hardcoded mode becomes rclone_backup_log_dir_mode so the value is
visible next to the path it applies to, and both roles' defaults now carry a
comment naming the other, since nothing else in the repo records that these two
share a directory.

Verified: playbooks/self-pull.yml converges to changed=0 and the live directory
keeps 2775 josh:foundry. The rclone side is a mode substitution to the value
already on disk, so it is a no-op; confirming that end to end needs the vault,
which is currently unavailable - see below.
josh merged commit cfb52865ca into main 2026-08-06 15:11:15 -05:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
josh/foundry!12
No description provided.