npilosov/nix-desktop
Watch
0
0
Fork
You've already forked nix-desktop
 1
Code Issues Pull requests 2 Projects Releases Packages Wiki Activity Actions

kanidm: compose the shared IdP on caco-zero-three, roster to the host #16

Open
dron wants to merge 1 commit from feat/kanidm-caco into main
pull from: feat/kanidm-caco
merge into: npilosov:main
Conversation 1 Commits 1 Files changed 4 +80
dron commented 2026-09-10 17:21:48 +03:00
Collaborator
Copy link

Second commit of the Kanidm work on the tenant box, caco-zero-three.

The shared IdP module (services/kanidm) stopped being heimdall's alone and
now reads its roster from the host. This applies it to the tenant's box the
same way heimdall carries it.

What lands

  • composition.nix pulls in ../../services/kanidm, appended after the
    other service modules for the reason this list is order-sensitive. The
    module brings the server, the client, the idm.<domain> certificate and the
    Caddy vhost. Its oauth2 clients are all gated on services this box does not
    run (no matrix, no grafana, no forgejo), so only the core IdP lands.

  • identity.nix is the tenant's roster, kept with the host rather than the
    service — the one thing two deployments of the module can never share. It
    starts with a single person on purpose, dron the operator who owns this
    box, so the IdP can be brought up and verified end to end before the rest of
    the roster is enrolled. One flat group (idm-users) to start; there are no
    oauth2 clients to map a scope to yet, so an admin layer is premature.

  • immutable-etc.nix pins kanidm to uid 987 / gid 981. The module
    defaults to 999, which is nscd's on this box, so the pin overrides it with
    a fresh allocation continuing the file's walk into free ground (987/981
    verified free on the box), keeping /var/lib/kanidm on a stable owner across
    deploys.

Deliberately out of scope (operator steps at deploy time)

  • The sops secret kanidm-idm-admin-password is still to be added to
    secrets/hosts/caco-zero-three.yaml.
  • The idm.cacodaemon.space DNS record is still to be created.

Both are operator steps, not part of this change. sops-nix fails at
activation, not eval, so CI stays green on this commit; the secret and the
DNS record become load-bearing at nixos-rebuild time.

Eval

Local semantic eval is not feasible on this box (1.5 GB disk / 4 GB RAM, a full
eval needs ~1.5 GB free). Verified instead by nix-instantiate --parse on all
four files plus source-level tracing of every option the module reads
(bootstrap = false on caco, nd.services.matrixTuwunel.serverName,
nd.services.observability.grafanaDomain, nd.backups.state, the Caddy
snippets common/acme_challenge/proxy_headers, and the secret-hygiene
test, which only checks files are encrypted and does not assert every declared
sops secret has a YAML key).

Second commit of the Kanidm work on the tenant box, caco-zero-three. The shared IdP module (`services/kanidm`) stopped being heimdall's alone and now reads its roster from the host. This applies it to the tenant's box the same way heimdall carries it. ## What lands - **`composition.nix`** pulls in `../../services/kanidm`, appended after the other service modules for the reason this list is order-sensitive. The module brings the server, the client, the `idm.<domain>` certificate and the Caddy vhost. Its oauth2 clients are all gated on services this box does not run (no matrix, no grafana, no forgejo), so only the core IdP lands. - **`identity.nix`** is the tenant's roster, kept with the host rather than the service — the one thing two deployments of the module can never share. It starts with a single person on purpose, `dron` the operator who owns this box, so the IdP can be brought up and verified end to end before the rest of the roster is enrolled. One flat group (`idm-users`) to start; there are no oauth2 clients to map a scope to yet, so an admin layer is premature. - **`immutable-etc.nix`** pins kanidm to uid `987` / gid `981`. The module defaults to `999`, which is nscd's on this box, so the pin overrides it with a fresh allocation continuing the file's walk into free ground (987/981 verified free on the box), keeping `/var/lib/kanidm` on a stable owner across deploys. ## Deliberately out of scope (operator steps at deploy time) - The sops secret `kanidm-idm-admin-password` is still to be added to `secrets/hosts/caco-zero-three.yaml`. - The `idm.cacodaemon.space` DNS record is still to be created. Both are operator steps, not part of this change. sops-nix fails at **activation**, not eval, so CI stays green on this commit; the secret and the DNS record become load-bearing at `nixos-rebuild` time. ## Eval Local semantic eval is not feasible on this box (1.5 GB disk / 4 GB RAM, a full eval needs ~1.5 GB free). Verified instead by `nix-instantiate --parse` on all four files plus source-level tracing of every option the module reads (`bootstrap = false` on caco, `nd.services.matrixTuwunel.serverName`, `nd.services.observability.grafanaDomain`, `nd.backups.state`, the Caddy snippets `common`/`acme_challenge`/`proxy_headers`, and the `secret-hygiene` test, which only checks files are encrypted and does not assert every declared sops secret has a YAML key).
kanidm: compose the shared IdP on caco-zero-three, roster to the host
All checks were successful
Check / check (push) Successful in 4m18s
Details
a193c5a16c 
Step two of putting an IdP on caco-zero-three. The shared module
(services/kanidm) stopped being heimdall's alone and now reads its roster
from the host; this applies it to the tenant's box.

  - composition.nix pulls in ../../services/kanidm, appended after the other
    service modules for the reason this list is order-sensitive. The module
    brings the server, the client, the idm.<domain> certificate and the Caddy
    vhost; its oauth2 clients are all gated on services this box does not run,
    so only the core IdP lands.

  - identity.nix is the tenant's roster, kept with the host rather than the
    service -- the one thing two deployments of the module can never share. It
    starts with a single person on purpose, dron the operator who owns this
    box, so the IdP can be brought up and verified end to end before the rest
    of the roster is enrolled. One flat group (idm-users) to start; there are
    no oauth2 clients to map a scope to yet, so an admin layer is premature.

  - immutable-etc.nix pins kanidm to uid 987 / gid 981. The module defaults to
    999, which is nscd's on this box, so the pin overrides it with a fresh
    allocation continuing the file's walk into free ground (987/981 verified
    free on the box), keeping /var/lib/kanidm on a stable owner across deploys.

The sops secret kanidm-idm-admin-password is still to be added to
secrets/hosts/caco-zero-three.yaml and the idm.cacodaemon.space DNS record is
still to be created -- both are operator steps at deploy time, not part of
this change; sops-nix fails at activation, not eval, so CI stays green.
npilosov commented 2026-09-10 22:57:07 +03:00
Owner
Copy link

Review of PR #16: feat/kanidm-caco (head a193c5a)

The PR is one commit touching four files under hosts/caco-zero-three/. The changes you have staged locally in the working tree (the awg-gateway ones) aren't part of this PR, and the review ignores them. Verdict: don't merge yet. The PR description says a missing sops key only fails at activation, but it actually breaks caco's build. Every deploy to caco will fail until the key is added.

Blocking

  1. The missing sops key fails the build, not activation (hosts/caco-zero-three/composition.nix:60). Pulling in services/kanidm declares the secret kanidm-idm-admin-password, and secrets/hosts/caco-zero-three.yaml doesn't have that key. sops-nix checks every declared key at build time by default (validateSopsFiles = true), so building caco's system fails with "the key 'kanidm-idm-admin-password' cannot be found". The per-push CI stays green only because it skips the tenant-build-* targets and fleet-check only evaluates. After merge, every deploy-caco fails, including unrelated changes like llama-swap tuning, until someone adds the key.
  2. The nightly cache-builder goes red (packages/command-suite/cmd/fleet-check/main.go:406). It runs fleet-check with no --exclude, so it builds caco's system and hits the failure above. fleet-check stops at the first failed build, which means every check after caco is skipped and old cache roots are never pruned. The date stamp also doesn't get written, so the whole build repeats and fails again every night.

Should fix

  1. The certificate is requested before the DNS record exists (services/kanidm/default.nix:202). The PR leaves the idm.cacodaemon.space DNS record as a later manual step, so the certificate request fails on every switch and every renewal attempt. The box then shows as degraded permanently, which hides real failures later. Kanidm starts on the temporary self-signed cert, so Caddy's upstream TLS check fails and the site returns 502. The repeated failures also count against Let's Encrypt's rate limit.
  2. Caddy can reload before the cert file exists (services/kanidm/default.nix:469). Caddy is only ordered after useACMEHost certificates, not after the file this vhost loads directly. On heimdall this sorted itself out because a later successful certificate order restarted Caddy. On caco no order ever succeeds, so Caddy stays on its old config.
  3. The group name doesn't match the shared module (hosts/caco-zero-three/identity.nix:52). The roster names the everyone-group idm-users, but the shared module hardcodes matrix-users and admin-users, and web-gate, jellyfin, grafana, forgejo, tuwunel and the launcher all rely on those names. Adding any of those clients on caco will fail evaluation. Renaming the group later, with autoRemove on, deletes it and creates a new one with a new UUID.
  4. The identity-stays-separate test is out of date (tests/caco-tenant.nix:200). Its reasoning ("pointing its Matrix logins at our Kanidm") no longer holds: kanidm.enable on caco would now use caco's own Kanidm, not heimdall's. The test now blocks the obvious next step (Matrix login through the tenant's own identity server). The same outdated explanation is in services.nix:78-83 and in tuwunel's kanidm.enable description.
  5. Group members get reset on every restart. idm-users keeps the upstream default overwriteMembers = true. If a tenant operator adds someone by hand with kanidm group add-members, that person is silently removed on the next kanidm restart, which happens on every deploy and every secret or cert rotation.
  6. Kanidm state isn't backed up (services/kanidm/default.nix:147). caco doesn't import services/backups, so nd.backups.state.kanidm has no effect. /var/lib/kanidm is the only copy of every passkey, TOTP and password enrolled there. At minimum, write down that it's deliberately left out, the way services-auth.nix does for Authelia.
  7. Kanidm can read every certificate key on the box (services/kanidm/default.nix:225). Being in the acme group gives it read access to every ACME private key on caco, including the TURN certificate's, not just its own.

Comment errors

  • identity.nix:19 says dron is the only human with an SSH key, but users.nix also gives keys to amenocturne and slopych, and default.nix:86-87 says the same. The follow-on claim is also wrong: the others aren't in the roster at all, so they can't enroll anything.
  • identity.nix:9 ("Pinned to the vhost… so the two can never drift") was copied from heimdall. Nothing in this file pins a domain; the pin is in services/kanidm.
  • default.nix:37 says the roster adds no lists, but persons.dron.groups becomes the list-typed groups..members. It's harmless today, but the stated reason for putting the import last is wrong.

Design questions (not bugs)

  • CI blind spot: this is the second missing-sops-key incident after int-secret. Building system.build.sops-nix-manifest for every host on each push is cheap (no CUDA) and would have caught this. So would a secret-hygiene check comparing declared keys against the YAML.
  • uid/gid default: the shared module sets mkDefault 999 for the uid/gid, which only suits heimdall. It happened to collide with nscd on caco. Dropping the default and pinning 999 in heimdall's config would make every host choose explicitly.
  • Two identity stores: caco already runs Authelia for llm.* and ha.*. This PR adds a second, unlinked set of accounts for the same people, and doesn't say whether Authelia is being replaced or kept.

The quickest path to mergeable is to add the sops key and create the DNS record before merging, rather than after. Then rename the group to matrix-users/admin-users and update the tripwire test. I haven't posted anything to the PR. I can leave these as review comments, or fix the smaller items on the branch.

Review of PR #16: feat/kanidm-caco (head a193c5a) The PR is one commit touching four files under hosts/caco-zero-three/. The changes you have staged locally in the working tree (the awg-gateway ones) aren't part of this PR, and the review ignores them. Verdict: don't merge yet. The PR description says a missing sops key only fails at activation, but it actually breaks caco's build. Every deploy to caco will fail until the key is added. Blocking 1. The missing sops key fails the build, not activation (hosts/caco-zero-three/composition.nix:60). Pulling in services/kanidm declares the secret kanidm-idm-admin-password, and secrets/hosts/caco-zero-three.yaml doesn't have that key. sops-nix checks every declared key at build time by default (validateSopsFiles = true), so building caco's system fails with "the key 'kanidm-idm-admin-password' cannot be found". The per-push CI stays green only because it skips the tenant-build-* targets and fleet-check only evaluates. After merge, every deploy-caco fails, including unrelated changes like llama-swap tuning, until someone adds the key. 2. The nightly cache-builder goes red (packages/command-suite/cmd/fleet-check/main.go:406). It runs fleet-check with no --exclude, so it builds caco's system and hits the failure above. fleet-check stops at the first failed build, which means every check after caco is skipped and old cache roots are never pruned. The date stamp also doesn't get written, so the whole build repeats and fails again every night. Should fix 3. The certificate is requested before the DNS record exists (services/kanidm/default.nix:202). The PR leaves the idm.cacodaemon.space DNS record as a later manual step, so the certificate request fails on every switch and every renewal attempt. The box then shows as degraded permanently, which hides real failures later. Kanidm starts on the temporary self-signed cert, so Caddy's upstream TLS check fails and the site returns 502. The repeated failures also count against Let's Encrypt's rate limit. 4. Caddy can reload before the cert file exists (services/kanidm/default.nix:469). Caddy is only ordered after useACMEHost certificates, not after the file this vhost loads directly. On heimdall this sorted itself out because a later successful certificate order restarted Caddy. On caco no order ever succeeds, so Caddy stays on its old config. 5. The group name doesn't match the shared module (hosts/caco-zero-three/identity.nix:52). The roster names the everyone-group idm-users, but the shared module hardcodes matrix-users and admin-users, and web-gate, jellyfin, grafana, forgejo, tuwunel and the launcher all rely on those names. Adding any of those clients on caco will fail evaluation. Renaming the group later, with autoRemove on, deletes it and creates a new one with a new UUID. 6. The identity-stays-separate test is out of date (tests/caco-tenant.nix:200). Its reasoning ("pointing its Matrix logins at our Kanidm") no longer holds: kanidm.enable on caco would now use caco's own Kanidm, not heimdall's. The test now blocks the obvious next step (Matrix login through the tenant's own identity server). The same outdated explanation is in services.nix:78-83 and in tuwunel's kanidm.enable description. 7. Group members get reset on every restart. idm-users keeps the upstream default overwriteMembers = true. If a tenant operator adds someone by hand with kanidm group add-members, that person is silently removed on the next kanidm restart, which happens on every deploy and every secret or cert rotation. 8. Kanidm state isn't backed up (services/kanidm/default.nix:147). caco doesn't import services/backups, so nd.backups.state.kanidm has no effect. /var/lib/kanidm is the only copy of every passkey, TOTP and password enrolled there. At minimum, write down that it's deliberately left out, the way services-auth.nix does for Authelia. 9. Kanidm can read every certificate key on the box (services/kanidm/default.nix:225). Being in the acme group gives it read access to every ACME private key on caco, including the TURN certificate's, not just its own. Comment errors - identity.nix:19 says dron is the only human with an SSH key, but users.nix also gives keys to amenocturne and slopych, and default.nix:86-87 says the same. The follow-on claim is also wrong: the others aren't in the roster at all, so they can't enroll anything. - identity.nix:9 ("Pinned to the vhost… so the two can never drift") was copied from heimdall. Nothing in this file pins a domain; the pin is in services/kanidm. - default.nix:37 says the roster adds no lists, but persons.dron.groups becomes the list-typed groups.<n>.members. It's harmless today, but the stated reason for putting the import last is wrong. Design questions (not bugs) - CI blind spot: this is the second missing-sops-key incident after int-secret. Building system.build.sops-nix-manifest for every host on each push is cheap (no CUDA) and would have caught this. So would a secret-hygiene check comparing declared keys against the YAML. - uid/gid default: the shared module sets mkDefault 999 for the uid/gid, which only suits heimdall. It happened to collide with nscd on caco. Dropping the default and pinning 999 in heimdall's config would make every host choose explicitly. - Two identity stores: caco already runs Authelia for llm.* and ha.*. This PR adds a second, unlinked set of accounts for the same people, and doesn't say whether Authelia is being replaced or kept. The quickest path to mergeable is to add the sops key and create the DNS record before merging, rather than after. Then rename the group to matrix-users/admin-users and update the tripwire test. I haven't posted anything to the PR. I can leave these as review comments, or fix the smaller items on the branch.
All checks were successful
Check / check (push) Successful in 4m18s
Details
This pull request has changes conflicting with the target branch.
  • hosts/caco-zero-three/immutable-etc.nix
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/kanidm-caco:feat/kanidm-caco
git switch feat/kanidm-caco

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff feat/kanidm-caco
git switch feat/kanidm-caco
git rebase main
git switch main
git merge --ff-only feat/kanidm-caco
git switch feat/kanidm-caco
git rebase main
git switch main
git merge --no-ff feat/kanidm-caco
git switch main
git merge --squash feat/kanidm-caco
git switch main
git merge --ff-only feat/kanidm-caco
git switch main
git merge feat/kanidm-caco
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
npilosov/nix-desktop!16
No description provided.