nix-desktop

A single NixOS flake that builds six managed machines — a workstation, laptop, home server, VPS, handheld, and a separately owned GPU server — including ~20 services, an AmneziaWG mesh, a self-hosted recursive DNS stack, secrets split by trust boundary, and guarded deployments that roll themselves back when a host comes up unhealthy.

A host names the modules it composes in one place, and its identity in another. hosts/heimdall/composition.nix:

[
  ../../roles/server
  ../../services/jellyfin
  ../../services/forgejo
  ../../services/matrix-tuwunel
  ../../services/observability
  ../../services/backups
]

and hosts/inventory.nix carries only mesh membership and deployability:

heimdall = {
  deployment.enable = true;
  mesh = {
    address = "10.90.100.1/24";
    publicKey = "aHOI2skr4Ei3U96KpRONeR/gHRa8qLudvvmTXyzx+HI=";
    endpoint = "…:51820";
  };
};

Everything else — module set, dependency closure, mesh peer lists, mesh subnet, internal DNS records, /etc/hosts entries — is derived from that entry.

The Fleet

Host	Kind	Runs
thor	Workstation	Hyprland desktop, work VPN + tooling, gaming; Lix base system, pinned 7.1 kernel with the in-kernel AmneziaWG module
loki	Laptop	Hyprland desktop, work VPN + tooling
heimdall	Home server, mesh hub	Jellyfin, Navidrome, Forgejo, Matrix (Tuwunel + LiveKit + Element), Nix binary cache, Transmission, Grafana/VictoriaMetrics/VictoriaLogs, nightly restic backups
vidar	VPS, mesh spoke	AmneziaWG gateway, Knot Resolver + Blocky DNS, public DoH endpoint, coturn, append-only restic backup target, observability agent
magni	Anbernic RG353V handheld	AArch64 SD image, RetroArch, fast local boot, bootstrap Wi-Fi, USB gadget recovery
caco-zero-three	Tenant-owned GPU server	CUDA llama.cpp/llama-swap, Hermes WebUI, Tuwunel/LiveKit, ntfy, Authelia and Podman workloads; isolated as a mesh client and deployed through its tenant selector

Design Notes

The current flake and software decisions, completed changes, deferred external actions, and rollout checks are recorded in docs/modernization-2026-09-06.md. Deployment verification, evaluation and packaging changes, runtime tests, and the remaining recovery setup are described in docs/flake-improvements-2026-09-08.md.

Four things in here are worth more than the rest:

Guarded deployment with automatic rollback. deploy <host> switch runs the old generation's own health probes as a preflight, pre-builds and dry-activates the entire selection, then switches hosts one at a time. Before each activation it arms a dead-man's-switch on the target that restores the exact captured generation unless it is explicitly confirmed. Confirmation requires both that /run/current-system points at the expected closure and that that closure's health probes pass. A failed switch requests rollback immediately; losing SSH mid-deploy leaves the timer armed, so a host that becomes unreachable recovers on its own. Implementation in the compiled command suite under packages/command-suite, and the failure paths — preflight failure, unhealthy activation, failed switch — are tested against mocked colmena and ssh in tests/deploy-safe-switch.nix.

Topology derived from a single source of truth. lib/topology.nix computes the mesh subnet, listen ports, per-host peer lists, and internal DNS from the inventory. The hub is named there (hub = "heimdall"); any other host with an endpoint is dialed directly instead of through it. lib/inventory.nix fails the evaluation if the hub is missing, is not a managed host, or has no endpoint.

Secrets split by trust boundary. Each host decrypts with its own SSH host key via sops-nix; there is no shared age key. Files are scoped so that a workstation cannot read cluster credentials and a server cannot read Wi-Fi or work secrets. See Secrets.

Systemd hardening as reusable profiles. modules/options/hardening.nix defines graduated confinement profiles rather than per-service copy-paste, with the exclusions documented and justified — ProcSubset=pid is left out because JVM and Go runtimes read /proc/meminfo at startup, and JIT runtimes get a profile without MemoryDenyWriteExecute.

Architecture

The repo is organized around four layers:

• modules/: shared option definitions and genuinely universal core behavior
• roles/: reusable host capabilities such as desktop, server, or work
• services/: single-purpose service definitions (Jellyfin, Navidrome, Tuwunel, etc.) selected in each host's composition.nix
• hosts/: machine-specific composition, hardware, storage, and local overrides

The important design constraint is that the shared baseline should stay portable. Personal or environment-specific defaults belong in host-layer modules, not in modules/base.nix.

Layout

.
├── flake.nix
├── flake/                  # Colmena, app/package, check, and installer outputs
├── hosts/
│   ├── inventory.nix
│   └── <hostname>/
├── sites/                  # operator identity and per-tenant/site defaults
├── modules/
│   ├── base.nix
│   ├── options/
│   └── core/
├── roles/
│   ├── workstation/
│   ├── desktop/
│   ├── work/
│   ├── gaming/
│   └── server/
├── services/
│   └── <service>/
├── lib/
├── overlays/
├── packages/
└── secrets/

Composition Model

Each declared host is built from:

1. modules/base.nix
2. the modules listed in hosts/<hostname>/composition.nix
3. the host module in hosts/<hostname>/

composition.nix is a flat list of module paths, not a module with imports:

[
  ../../roles/desktop
  ../../roles/work
]

lib/fleet.nix splices that list directly into the host's module list. The shape matters to derivations, not to behaviour. Nesting these one level deeper — inside a module's imports — or reordering the list reorders every list- and string-valued option that more than one module contributes to: environment.systemPackages, Caddy's globalConfig, tmpfiles and nftables rules, Home Manager files. Measured on 2026-09-15 by reversing every host's list: derivations moved, behaviour did not. No two packages on any host provide the same file, so PATH precedence cannot shift, and the lines that moved were ones whose order nothing reads (Caddy global options, listen addresses, firewall rules on disjoint interfaces). So lists are kept sorted, and a reorder is checked with nix run .#nix-eval-and-diff -- --contents <host>, which builds what moved and reports it as reordered rather than as a new hash.

Roles import the roles they build on, so the dependency graph is stated in the tree rather than computed: gaming imports desktop, and desktop and work import workstation. The module system closes that graph transitively and deduplicates by path, so listing a role twice costs nothing. Services do the same where they have a hard dependency — blocky imports knot-resolver.

Fleet discovery and NixOS host construction live in lib/fleet.nix. flake.nix composes that result, while flake/ contains the Colmena, installer, app/package, and check output builders.

hosts/inventory.nix describes mesh membership and deployability only:

{
  my-host = {
    mesh = {
      address = "10.90.100.9/24";
      publicKey = "...";
    };
  };
}

lib/inventory.nix types it once with lib.evalModules, so a malformed entry reports the option path rather than a hand-written message. Everything downstream reads that typed value — lib/fleet.nix, lib/topology.nix, the Colmena hive, the checks, and each host through the inventory module argument and config.nd.fleet — so every default is stated once. Setting deployment.enable = true adds the host to both the Colmena hive and the guarded deploy command; hosts without it remain available through nixosConfigurations.

The same file checks the fleet-wide invariants, once per evaluation rather than once per host: the named hub is a managed mesh host with an endpoint, no duplicate mesh addresses or public keys, no name shared between a managed host and a mesh-only peer, no mesh client with an endpoint, and no auto-deploy on a host belonging to another tenant.

Shared vs Local Config

Portable defaults live in modules/.

Local defaults are split by authority under sites/:

• operator.nix contains the administrator's user, SSH, Git, cache and local checkout settings.
• nestoris.nix imports the operator and adds the fleet owner's domains, DNS and Wi-Fi configuration.
• cacodaemon.nix imports the operator but carries the tenant's domains and ACME contact separately.

These files hold values that should not be assumed for every user of the flake:

• primary user
• SSH authorized keys
• Git identity
• private cache trust
• server domain
• fleet DNS-over-HTTPS endpoint
• desktop Wi-Fi profiles
• local repo path for nh

Existing machines import the applicable site explicitly from their host modules. New machines do not need to use one. They can define only the settings they need in their own host module or introduce a new trust boundary under sites/.

Important Options

The shared option surface for local customization is mostly under nd.*:

• nd.meta.primaryUser: main interactive user for the host
• nd.meta.bootstrap: temporary mode for installs and recovery
• nd.meta.stateVersion.system: NixOS compatibility version from initial installation
• nd.meta.stateVersion.home: Home Manager compatibility version from initial user setup
• nd.user.openssh.authorizedKeys: SSH keys for the primary user
• nd.user.git.name / nd.user.git.email: Git identity
• nd.nix.substituters / nd.nix.trustedPublicKeys: extra cache settings
• nd.security.privEsc: run0 for interactive hosts or doas for deployment targets
• nd.desktop.iwd.countryCode: IWD regulatory domain
• nd.desktop.iwd.profiles: Wi-Fi profiles rendered from SOPS secrets
• nd.paths.repoRoot: checkout path used by nh

See modules/options/ for the authoritative option definitions — one file per concern (meta, security, user, network, nix, desktop, paths, theme), plus fleet.nix, which exposes the typed inventory as nd.fleet. Per-service knobs, such as nd.services.observability.backendHost, live in that service's directory.

Roles

Roles are self-contained directories under roles/. Each role owns both its composition and its implementation modules.

Notable role boundaries:

• cli: portable fish, Git/Jujutsu and small shell tools; suitable for Magni
• workstation: credentials, TPM-backed SSH/signing, superfile and local interactive tools; imports cli
• development: language servers, compilers, uv, GitHub CLI and Distrobox with rootless Podman; imports workstation
• operations: AWG, DNS, networking, SOPS and storage administration; imports workstation
• ai: Claude Code and Codex; imports workstation; Thor's self-hosted OpenCode client stays host-local
• desktop: graphical environment (Hyprland with no bar, and mako for notifications; hypridle auto-lock and the login keyring), kmscon, default apps, and the session variables naming a terminal and browser; imports workstation
• server: bash login shell, headless diagnostics, Caddy, and automatic upgrades
• work: mail, employer-specific certificates, VPN, Ansible/YAML tooling, and work integrations; imports workstation
• gaming: Steam, emulators, gamemode, and the image-quality pipeline; imports desktop

modules/base.nix contains shared options, core system behavior, and the two pieces of user tooling every host gets: helix, themed and configured but without language servers (modules/core/user/helix.nix), and Waypipe in the system profile so either end of an SSH-forwarded graphical application can find it (modules/core/network/waypipe.nix). Everything else user-facing belongs to a role so headless machines do not inherit a workstation environment accidentally.

Game image quality

Three layers, applied at the earliest point that can still fix the problem:

Reconstruction (per game, optiscaler-install). Most of what reads as "blurry" in a modern title is its own upscaler resolving a low internal resolution, and no amount of post-processing puts back detail that was never presented. OptiScaler hooks a game's existing DLSS/FSR2+/XeSS inputs and substitutes a better resolver. optiscaler-install <game-dir> installs it next to a game's executable and prints the WINEDLLOVERRIDES line for that title's launch options.

--profile=xess is the default and uses XeSS 3.0.1, which runs on this machine's RDNA2 card through DP4a with no forcing. --profile=fsr4 forces FSR 4.1.1 through the INT8 path; upstream supports INT8 on RDNA4/RDNA3 dGPUs only, and on RADV it is reported to work at a higher upscaler cost, so it is opt-in and worth checking for ghosting. Installs are imperative state inside the Steam library, not something a rebuild reconciles -- the generated remove_optiscaler.sh is the uninstall.

Texture LOD (session-wide, roles/gaming/optiscaler.nix). A game that picks mip levels for its internal render resolution hands the display textures blurrier than it can show. DXVK_CONFIG carries a mild global samplerLodBias for Direct3D 9 and 11; Direct3D 12 goes through vkd3d-proton, which has no equivalent knob, so there the lever is optiscaler-install --mip-bias=. Anisotropy is deliberately not forced globally -- DXVK's own documentation warns it breaks passes that rely on bilinear filtering -- and is instead per game via --aniso.

Post-processing (vkBasalt, roles/gaming/vkbasalt.nix). Last, and least able to help: the layer only ever sees the finished swapchain image. Effects run left to right, so presets follow four ordering rules -- anti-aliasing first, deband before any sharpener, exactly one sharpener and it goes last, and never a morphological AA pass after a game's own TAA. That last rule is why the default is rcas rather than the cmaa2:dls it used to be: almost everything modern here ships TAA or an upscaler, and that chain was softening an already-resolved frame and then sharpening it back up.

vkBasalt reads exactly one config file, chosen at process start, with no include mechanism -- so a preset is selected by pointing VKBASALT_CONFIG_FILE at it before the game launches. vkbasalt-preset list shows what is available; vkbasalt-preset <name> prints the launch-option line for a Steam title, and vkbasalt-preset <name> -- <cmd> runs something with it directly. The presets and the generated default come from one attribute set, so the default cannot drift from the preset it names. SUPER + Home toggles the layer off entirely.

CLI profiles

server and cli are the two shell profiles, and nothing imports both. Workstation and development add progressively more specialized capabilities:

	server	cli	workstation	development
login shell	bashInteractive	fish	inherited	inherited
tools	headless diagnostics	Git/Jujutsu and small shell tools	credentials, file/media and local interactive tools	compilers, language servers and containers
delivery	environment.systemPackages	Home Manager	Home Manager	Home Manager + system Podman
helix	baseline only	baseline only	baseline only	baseline + nil, rust-analyzer, lua-language-server

The shell is set by each profile at normal priority rather than in modules/core/user/user.nix, so a host that ever imports both fails loudly instead of silently picking one. Server tools go in environment.systemPackages because smartctl and dig are run as root, and a root shell has no Home Manager profile on PATH.

Session variables live with what they name — EDITOR with helix, TERMINAL with foot, BROWSER with firefox, the portal variables with xdg-portal.nix — rather than in one fleet-wide file, so headless hosts never advertise a terminal or browser they do not have.

Services

Services live under services/ and are opt-in per host: add the module's path to hosts/<hostname>/composition.nix. Every service assumes the server role, which those hosts list alongside them.

A service module is imported only where it is wanted, so it does not gate itself behind an enable flag. Modules that need to know whether another service is present ask that service's real upstream option — config.services.forgejo.enable, config.services.jellyfin.enable. The one exception is observability, which is an assembly of VictoriaMetrics, VictoriaLogs, Grafana, vmagent, vector and node-exporter with no single upstream option, so it declares nd.services.observability.enable itself.

Adding a web service

A service fronted by Caddy declares itself once, and the Caddy vhost, the systemd hardening floor, the deployment health check and the Grafana dashboard all follow:

nd.services.web.jellyfin = {
  port = 8096;              # loopback port Caddy proxies to
  unit = "jellyfin";        # systemd unit, without .service
  job = "jellyfin";         # Prometheus job, if it ships an exporter
  probed = true;            # blackbox-probed, so the dashboard gets TLS expiry
  health = "/health";       # loopback health path, or omit for none
  required = false;         # true = failing it fails the deploy and rolls back
  hardening = "runtime";    # nd.hardening.floor tier, or omit for none
  blockMetrics = true;      # 404 /metrics on the public vhost
};

required is the deployment-safety tier. Reserve true for services whose absence costs access to the host, its communications, or the ability to deploy at all — Forgejo is true, Jellyfin is not. Set healthUnit = false for a service whose unit goes active well before it actually serves, so it is judged by its endpoint instead; harmonia is the case that exists for.

The served host is <vhost>.<serverDomain>, where vhost defaults to the attribute name — set it when they differ (harmonia serves cache, ntfy serves push). Set host instead when the domain is configurable in its own right and need not sit under the fleet domain, as Grafana's does.

Then add the module to hosts/<hostname>/composition.nix. Nothing needs touching in services/observability.

This deliberately does not cover every vhost. One that differs in kind rather than in name — Matrix delegation, LiveKit's split JWT path, the DoH endpoint, a static site — keeps writing its own extraConfig, and its dashboard entry stays hand-written in services/observability/dashboards.nix. An escape hatch wide enough to express those would defeat the point.

If the service holds state worth keeping, it also declares its own backup contract; see Secrets's neighbour nd.backups.state:

nd.backups.state.jellyfin.paths = [ "/var/lib/jellyfin/data" ];

Currently defined services:

• awgGateway: AmneziaWG VPN gateway
• matrixTuwunel: Matrix server (Tuwunel + LiveKit)
• elementWeb: Element Web client
• elementCall: self-hosted Element Call UI
• nixCache: Nix binary cache (Harmonia + cache builder)
• dufs: web file manager
• forgejo: Git hosting
• minecraftServer: PaperMC server with RCON
• jellyfin: Jellyfin media server
• navidrome: Navidrome music server
• transmission: Transmission BitTorrent daemon
• wgDashboard: WGDashboard web UI
• turn: coturn relay for Matrix and Element Call (UDP fast path with TLS fallback)
• backupTarget: append-only restic REST server (vidar)
• backups: nightly restic backups of heimdall service state to vidar
• observability: VictoriaMetrics, VictoriaLogs, Grafana, vmalert, Alertmanager, and host agents

The directories under services/ are the authoritative list.

Music library workflow

Heimdall manages Navidrome's files with Beets. New music is staged in /media/downloads/music; run music-library-import over SSH to review MusicBrainz/AcoustID matches. Every match defaults to skip until explicitly accepted. Accepted imports are copied into /media/music as 160-kbps VBR Opus with the verified metadata and cover art embedded. Each source audio file is deleted only after its Opus conversion succeeds.

The existing library is repaired in small batches with music-library-retag '/media/music/Artist/Album'. This writes tags in place but never re-encodes existing Opus files. music-library-audit produces a TSV report of missing tags and non-Opus formats; it also counts cue sheets, which must be split or converted album-by-album instead of having their referenced audio blindly replaced. music-library-beet exposes read-only Beets queries such as music-library-beet stats and music-library-beet list.

vmalert evaluates the alert rules in services/observability/alerting.nix, one instance against VictoriaMetrics and one against VictoriaLogs, and Alertmanager routes whatever fires. Every metrics rule is unit-tested against synthetic series by the observability-alert-rules check, which must prove it both fires and stays quiet. Grafana only draws dashboards; its own alerting is switched off. Firing and resolved notifications go to the static matrix-alertmanager-receiver Go binary from nixpkgs, bound only to Heimdall's loopback interface. It converts Alertmanager's webhook payload to a Matrix message and sends it to the private #alerts:matrix.nestoris.online room through local Tuwunel. The dedicated @grafana-alerts:matrix.nestoris.online bot (named before the move off Grafana) runs in a hardened systemd service. Setting nd.services.observability.heartbeat.enable routes an always-firing Watchdog alert to an external dead-man URL instead, so heimdall or its alerting going dark is reported from outside. Silence alerts from a shell on heimdall with amtool. Its token and recovery password live only in secrets/hosts/heimdall.yaml; SOPS renders the token into the service's private environment file at activation time.

Host Modules

Within hosts/<hostname>/, keep modules named by concern:

• hardware.nix
• storage.nix / disks.nix
• network.nix
• ssh.nix
• kernel.nix
• nix.nix
• hyprland.nix
• vpn.nix

Avoid catch-all custom.nix files when the configuration can be named more precisely.

Common Commands

Switch a named host:

nh os switch . -H <hostname>
# or
sudo nixos-rebuild switch --flake .#<hostname>

Deploy a server through Colmena:

deploy vidar
deploy heimdall build
deploy all dry-activate
deploy all health
deploy all switch

The deployment command accepts only the fleet's own deployable hosts -- heimdall, magni, vidar -- or the explicit all selector. Its default goal is the guarded switch, so deploy all and deploy all switch are equivalent. unsafe-switch exposes a direct Colmena switch for emergency use without health checks or automatic rollback.

Hosts owned by someone else are deployed by their own command and are not reachable from deploy at all:

nix run .#deploy-caco -- all switch

deployment.tenant in the inventory is what splits them. Each binary's all covers only its own owner's hosts, so no selector spans owners, and a tenant host may not set deployment.autoDeploy, which selects hosts for the manually dispatched fleet deployment. The Colmena hive and the build checks still cover every deployable host; only the deploy selectors are separated.

The same command runs from CI as the Deploy caco-zero-three workflow (.forgejo/workflows/deploy-caco.yaml), started by hand from the Actions tab. It requires the newest Check run for the exact main commit to have succeeded, and deploying someone else's machine stays a deliberate act. It reaches the box by name over the mesh rather than through the tenant's router forward that thor's ssh config uses. Nothing upstream substitutes that closure; the per-push Check run builds and roots it on heimdall, so the deploy downloads it instead of compiling.

Each deployed generation carries its own deployment-guard, including the systemd, HTTP, and DNS probes declared by that generation. health executes the guard belonging to the currently active generation without changing the host. Generations created before this mechanism fall back to checking the systemd system state.

switch checks the old generation with its own guard, pre-builds and dry-activates the whole selection, then switches hosts one at a time. It obtains the expected system and guard paths from the same Colmena configuration used for those operations. Immediately before switching, the uploaded target guard runs interruption-safety checks; the LiveKit host refuses to switch while it has active rooms or participants. --allow-busy overrides only that gate when an interruption is intentional. The currently deployed guard then arms a timer that restores the exact generation captured before activation. Its duration is the configured five-minute activation allowance plus the configured health-retry delays, so health checks do not consume the switch budget. The target guard cancels the timer only after /run/current-system points at the expected new closure and that closure's own health checks pass. A failed switch or exhausted health retries requests the rollback through the captured guard immediately; loss of SSH leaves the timer armed. A globally degraded system remains deployable when every explicitly required unit and application probe is healthy.

The safety windows can be tuned for an unusually slow host:

DEPLOY_ROLLBACK_TIMEOUT_SEC=600 \
DEPLOY_HEALTH_ATTEMPTS=36 \
DEPLOY_HEALTH_INTERVAL_SEC=5 \
  deploy heimdall switch

Update flake inputs:

nix flake update

Apply the repository's safe formatters to Nix and Go sources:

nix fmt

Run the read-only quality gate over formatting, dead Nix bindings, Statix diagnostics, Go tests and vet, and Forgejo workflows:

nix run .#lint

The command builds the same sandboxed checks.<system>.lint derivation used by CI and never rewrites the working copy. Jujutsu pushes remain the native, transport-only jj git push; jj fix is still available as an explicit Nix-formatting convenience.

A plain-git clone can run that gate automatically before every push:

git config core.hooksPath .githooks

.githooks/pre-push lints the revision being pushed rather than the working copy, so a dirty tree cannot block a push and a clean tree cannot hide a bad commit; it builds the derivation CI builds, so a pass here is a pass there. Bypass a single push with git push --no-verify. Enable this in every clone that pushes to main: CI is otherwise the only backstop, and a red runner is found by whoever looks next rather than by whoever caused it. Jujutsu fires no git hooks at all, so a jj working copy is unaffected and still needs nix run .#lint by hand.

Evaluate every host and package output, then build every check for the current system:

nix run .#fleet-check

The driver starts a fresh Nix evaluator for every host and package output, then builds each check separately. Package evaluation covers every system exposed by the flake even when checks are built only for the runner's native system. This keeps peak evaluator memory near one host or package graph; a monolithic nix flake check retains the full fleet graph and exceeds the memory limit on Heimdall.

Forgejo builds the dedicated lint gate first, then runs the same fleet command on every push. The second pass over lint is already present in the Nix store. The check runner lives in a Nix-built NixOS systemd-nspawn container on Heimdall and accepts jobs only from npilosov/nix-desktop. The container has a private network and user namespace, blocks forwarded access to private networks, and exposes no inbound ports. The runner executes one job at a time and its container is limited to two CPUs, 5 GiB of memory (with reclaim pressure above 4 GiB), 2 GiB of swap, and 1024 tasks. Nix, Git, and the remaining runner tools are part of the container closure; Podman, Docker, and Node.js are absent.

The workflow uses Git directly for its exact-SHA checkout and the runner's native backend inside that boundary. fleet-check evaluates each NixOS host and package output and builds each native check in a separate process, including the Thor and Loki toplevels, which releases evaluator memory between targets and keeps the job within the container limit. As with standard NixOS containers, it receives the host's read-only Nix store and Nix daemon socket; the host daemon still enforces Heimdall's two-job/two-core limit. The private-network rule applies to direct container traffic: a source-specific route sends public egress through Heimdall's LAN while the forward firewall rejects private destinations. Nix builds are performed by the host daemon. DNS uses a stub bound only to the container's private host address and follows Heimdall's monitored, cached DoH path. Job homes and workspaces live under /run and are discarded whenever the runner service or container restarts. The host starts the runner only after container@forgejo-ci has configured the host end of the private veth. Its health monitor also verifies and repairs the source-specific egress rule if a network reactivation flushes it.

Checking and deploying are two workflows. Check runs on every push with cancel-in-progress, so a newer commit on a ref cancels the check still running for an older one instead of letting it rebuild closures nobody wants. Deploy is workflow_dispatch only: started from the Actions tab, it checks out the current main on a separate nix-deploy host runner, verifies the newest flake-check.yaml run for that exact SHA through Forgejo's API, and runs deploy-ci all switch. This is the normal guarded fleet deployment: both targets are pre-built and dry-activated, switched serially, health-checked against the exact generation, and automatically rolled back on failure. Keeping the deployment executor outside forgejo-ci lets Heimdall restart the check container without terminating its own deployment. The deployment runner stays alive across a switch and picks up runner-definition changes on reboot.

Deploying is deliberately manual. Separate concurrency groups let checks be cancelled without interrupting a deployment and its rollback timer. The shared .forgejo/require-check.sh gate rejects missing, failed, pending, cancelled, skipped, or unapproved checks; an older success cannot mask a newer failed run. Superseded check runs fail before building, so an entirely skipped run cannot serve as successful verification. Local deploy remains available for recovery and does not depend on Forgejo's API.

The deployment runner receives its SSH identity as a systemd credential and copies it at mode 0600 into its ephemeral /run home before invoking OpenSSH; the check container cannot access it. Both server host keys are pinned declaratively. The restricted public key is authorized only for the system account deploy, on Heimdall, Vidar and the separately dispatched Caco host. That account can enter a small validated activation/rollback protocol through doas; it does not share the human account's SSH authorization or general doas rule.

After changing runner configuration, verify both runners:

deploy all switch
ssh heimdall systemctl status \
  container@forgejo-ci \
  forgejo-runner-health \
  'gitea-runner-heimdall\x2ddeploy'
ssh heimdall systemctl --machine=forgejo-ci status 'gitea-runner-*'

The encrypted forgejo-runner-token in secrets/hosts/heimdall.yaml is a repository-scoped registration token. If it is ever rotated, generate it for the npilosov/nix-desktop scope and replace that SOPS value before deploying. The same file contains the dedicated forgejo-deploy-ssh-key; rotate its restricted public half in sites/deploy-authorized-key.nix at the same time.

Format the repository with the official Nix formatter:

nix fmt

To evaluate and build the AArch64 checks, run the driver where an AArch64 builder or emulation is available:

nix run .#fleet-check -- --system aarch64-linux

Thor also runs fleet-check-aarch64.timer weekly, using its existing QEMU emulation. It pins the local checkout's committed HEAD for the entire run and builds all ARM checks, including build-magni-image and magni-dtb. Uncommitted edits are excluded. Results remain GC-rooted under /var/lib/fleet-check-aarch64/results; inspect failures with journalctl -u fleet-check-aarch64.

Show what a host composes by reading its composition.nix, or ask the evaluated configuration what it actually runs:

cat hosts/thor/composition.nix

nix eval --json '.#nixosConfigurations' --apply \
  'cs: builtins.mapAttrs (_: c: c.config.services.jellyfin.enable) cs'

The second form reports what each host really runs rather than what was declared, and stays correct as services gain their own options.

Local Host Resolution

The flake output schema stays pure and always exposes only named hosts. The rebuild-current app reads /proc/sys/kernel/hostname at runtime, verifies that the name exists in nixosConfigurations, and invokes nh for that host.

Switch the current host, or select another supported nh os action:

nix run .#rebuild-current
nix run .#rebuild-current -- boot

Adding a New Host

1. Create hosts/<hostname>/default.nix.
2. Import ./hardware.nix and any host-local modules from that file.
3. Pin nd.meta.stateVersion.system and nd.meta.stateVersion.home in that host module to the versions used for its initial installation and Home Manager setup. Do not bump them during ordinary input updates.
4. Optionally import ../../sites/nestoris.nix, ../../sites/cacodaemon.nix, or a new site module if the machine should inherit an existing operator/tenant trust boundary.
5. Generate hardware config on the target machine:

nixos-generate-config --show-hardware-config > hosts/<hostname>/hardware.nix

6. Add the host to hosts/inventory.nix under its hostname, with its mesh keys.
7. Create hosts/<hostname>/composition.nix listing the roles and services it should run.
8. Set deployment.enable = true in the inventory if Colmena and the guarded deploy command should manage the host.
9. If needed, add disks.nix and a small storage.nix wrapper for Disko.
10. Install with:

nixos-install --flake .#<hostname>

If you use Disko:

nix run github:nix-community/disko -- --mode disko ./hosts/<hostname>/disks.nix

Bootstrap and Recovery

Two separate mechanisms cover first-boot and recovery:

Installer ISO

The flake exposes a minimal installer target:

• installer (under packages.<system>)

Build it with:

nix build .#installer

The installer ISO is a standalone bootable image with SSH, iwd, disko, and git — just enough to partition disks and run nixos-install from this flake. It is not a NixOS host configuration and is not managed via inventory.nix.

When built with the bootstrap mesh key available, the installer also joins the AmneziaWG mesh as the bootstrap peer (hostname bootstrap, 10.90.100.8), so installs and recovery sessions can be run over SSH from anywhere the mesh reaches. With the office Wi-Fi password available it also provisions an iwd profile for RUTUBE and connects on its own. Both secrets are decrypted on thor into fixed paths the installer module expects before building (the --impure is required — the build reads paths outside the flake):

sops decrypt --extract '["mesh-private-key"]' secrets/hosts/bootstrap.yaml > /tmp/nix-bootstrap-mesh.key
sops decrypt --extract '["awg-header-protection-key-mesh"]' secrets/mesh.yaml > /tmp/nix-bootstrap-mesh-header.key
sops decrypt --extract '["eap-password"]' secrets/work.yaml > /tmp/nix-bootstrap-wifi-eap.key
nix build .#installer --impure

Without both mesh key files the same target builds a plain, mesh-less installer with manual Wi-Fi. Because the private key and the Wi-Fi password are embedded in the ISO, treat the image like any other secret-bearing artifact: delete it after the install is done.

Bootstrap mode for managed hosts

Any managed host can opt into nd.meta.bootstrap = true to disable secret-backed features (SOPS, mesh, media and other services, etc.) until the machine has its normal SOPS material available again.

Typical bootstrap flow:

1. Install or boot the target in bootstrap mode.
2. Reach the machine using SSH or manual Wi-Fi setup.
3. Restore the machine's existing /etc/ssh/ssh_host_ed25519_key, or replace its recipient in .sops.yaml with one derived from the new public host key.
4. If the recipient changed, run sops updatekeys on every encrypted file the host consumes.
5. Return the host to normal mode and rebuild.

Laptop Wi-Fi can be brought up manually with iwctl during bootstrap:

iwctl
device list
station wlan0 scan
station wlan0 get-networks
station wlan0 connect "<ssid>"

Secrets

This repo uses sops-nix with encrypted files split by trust boundary. Normal hosts decrypt with /etc/ssh/ssh_host_ed25519_key; they do not share an age private key. Every file also includes the administrator recipient so it can be edited with sops.

File	Host recipients	Purpose
secrets/common.yaml	all	user password hash and fleet DoH path token
secrets/desktop.yaml	thor, loki	Wi-Fi
secrets/work.yaml	thor, loki	shared work/mail credential plus GitLab, Forgejo, and Jira tokens
secrets/cluster.yaml	heimdall, vidar	backup and TURN credentials shared by the servers
secrets/hosts/<host>.yaml	that host only	host VPN/mesh keys and host-local service credentials

vpn-password is a runtime alias for the eap-password key in secrets/work.yaml. To edit a file, use sops, for example:

sops secrets/hosts/thor.yaml

Recipient policy lives in .sops.yaml. After replacing a host SSH key, convert its new public key with ssh-to-age, update the matching recipient, and run sops updatekeys on each file that host can read.

The former shared key at /var/lib/sops-nix/key.txt is no longer referenced by new configurations, but old installed generations may still require it during manual recovery. Keep that file until those generations have been removed from the host. The configuration preserves an existing copy and enforces mode 0600; it never creates or deletes the key.

DNS

Normal DNS has one client path across the fleet:

systemd-resolved -> local dnsproxy -> Caddy DoH on vidar -> Blocky -> Knot Resolver

Vidar uses its local Blocky directly so recovery of the DNS server does not depend on its own public Caddy endpoint. Blocky and Knot listen only on loopback; Caddy is the only network-facing DNS service. DHCP and AmneziaWG do not install DNS servers or catch-all routing domains.

The work VPN remains independent: snx-rs installs Check Point DNS servers with specific routing domains on its physical link. Those more-specific routes beat the global ~. DoH route for office names without changing normal DNS.

Notes

• Home Manager manual pages and programs.man integration are disabled in the shared baseline to avoid man-cache build noise.
• nix-ld is enabled from its own core system module, not from a role.
• Put packages in the narrowest applicable cli, workstation, development, operations, ai, work, or host-local module. The assertions in tests/software-stack.nix protect the important boundaries.