tappaas-cicd — Design notes
The TAPPaaS "mothership" control plane. This document describes the internal component contract and the three-level dispatch that drives every manager and controller uniformly (ADR-007 P10 deliverable). For the catalog entry see README.md; for installation see INSTALL.md; for test coverage see TEST.md. How the mothership pulls its updates and the repository patterns behind it (basic / community / downstream / private, and the developer workflows) are a separate design note: DESIGN-GIT.md.
For the full rationale see
docs/design/ADR-007-implementation.md
(packages P4 — layout — and P10 — the template + dispatch).
Overview — internal layout
src/foundation/tappaas-cicd/
├── install.sh / update.sh / test.sh # top-level entry scripts (drive the dispatchers)
├── bootstrap.sh # first-boot: clone repo + nixos-rebuild the VM
├── pre-update.sh # pre-update pass of the tappaas-cicd module
├── scripts/refresh-control-plane.sh # self-refresh: pull + relink ~/bin + build components
├── scripts/tappaas-self-prepare.sh # ExecStartPre 2: the refresh, before the sweep (ADR-017 D3)
├── scripts/run-migrations.sh # config migrations, between the refresh and the rebuild (ADR-025)
├── migrations/ # NNNN-<slug>.sh — one-time config rewrites (empty in this release)
├── scripts/tappaas-self-rebuild.sh # ExecStartPre 3 (root): nixos-rebuild of the mothership
├── scripts/update-tappaas-schedule.sh # renders update-tappaas.timer from site.json (D2)
├── scripts/notify-update-failure.sh # mails the site owner when a run fails (#651)
├── scripts/site-notify.sh # the mail itself: site.json email via a node's mailer (ADR-007e)
├── scripts/tappaas-support-bundle.sh # the support bundle an alert carries
├── scripts/support-anonymize.py # …its anonymizer: the config in, personal data and secrets out
├── scripts/tappaas-cluster-state.sh # nodes, guests and HA moves as JSON: the node-down alert
├── tappaas-cicd.nix / flake.nix # the cicd VM's NixOS configuration
├── manager/ # domain-object lifecycle (CONFIG state)
│ ├── install.sh / update.sh / test.sh # dispatcher: loop child components
│ └── <name>-manager/ # one dir per manager component
├── controller/ # infrastructure control (RUNTIME state)
│ ├── install.sh / update.sh / test.sh # dispatcher: loop child components
│ └── <name>-controller/ # one dir per controller component
├── ui/ # the console (ADR-034): one component, driven like a dispatcher
├── lib/ # shared libraries, sourced (never copied per component)
└── scripts/ # foundation bring-up + unit tests (scripts/test/)
manager/— owns config state: JSON config files, schemas, and their validation. A manager may call controllers to realize that config. Managers ship avalidate.sh. Examples:identity-manager,site-manager,environment-manager,module-manager,network-manager,health-manager.controller/— owns runtime state: APIs, network devices, VMs. A controller does not ship avalidate.sh. Examples:opnsense-controller,proxmox-controller,switch-manager,ap-manager,authentik-controller.lib/— shared code sourced by components (e.g.common-install-routines.sh). Shared logic lives here once; it is never copied into individual components.- top-level entry scripts —
install.sh/update.sh/test.shat the cicd root.
Component contract
Each manager/controller lives in its own subdirectory and exposes a fixed set of files:
<component>/
├── <name>.{ts,py,sh} # main entry: domain verbs (CRUD for a manager, reconcile for a controller)
├── install.sh # idempotent: build artifact, place bin/ symlink, one-time setup
├── update.sh # idempotent: rebuild, re-link, migrate on-disk state if schema changed
├── test.sh # self-contained tests; exit non-zero on failure
├── validate.sh # (managers only) schema/reference validation for the domain
└── README.md # what it owns; manager-vs-controller; which controllers it calls
install.sh,update.sh, andtest.share mandatory and must be idempotent in every component.validate.shis present for managers and absent for controllers.- The preferred entry-point language order is TypeScript → Python → Bash (see "Preferred language" below).
Runtime-state access rule (the F12 carve-out)
Controllers own runtime state — but managers sometimes need to read it (health gates, module list, drift tables). The rule (ADR-007 post-implementation refactor, F12 decision B):
- A manager MAY read cluster runtime state, but ONLY through the shared
lib/ts/src/cluster.tshelpers (one audited choke-point) — never via its own scatteredssh/pvesh/qmcalls. - Every runtime write goes through a controller. No exceptions for new code.
(Grandfathered: network-manager scp's
zones.jsonto the nodes — that distributes a config artifact, it does not mutate a device; and health-manager'supdate-osverb delegates toupdate-os.sh, to be revisited when that script is ported.) - health-manager is a read-mostly orchestrator under this rule — it stays a manager.
Config root resolution
Every component resolves the config root the same way:
$TAPPAAS_CONFIG, else $CONFIG_DIR, else /home/tappaas/config —
TAPPAAS_CONFIG is the TAPPaaS-specific override and wins; CONFIG_DIR is the
long-standing variable the bash layer exports. A component must never invent a different
precedence (managers used to disagree, which made two managers read different roots
under the same environment).
Three-level dispatch
The control plane is driven by three trivial levels — adding a component never changes anything above it:
- Top level —
tappaas-cicd/{install,update,test}.shcallmanager/<verb>.shandcontroller/<verb>.sh(andui/<verb>.sh, the console, which is a single component). Dispatcher level —
manager/<verb>.shandcontroller/<verb>.sheach loop their child component directories and run the matching verb script. There is no shared runner; each dispatcher is a few lines:bash for d in "${here}"/*/; do [ "$(basename "${d}")" = TEMPLATE ] && continue # defensive: scaffold/work dirs stay inert [ -x "${d}install.sh" ] || continue "${d}install.sh" "$@" doneComponent level — each component's own
install.sh/update.sh/test.shdoes the actual work.
Adding a component = drop a directory containing the standard verb scripts (copy the nearest real component and edit — see "Scaffolding a new component"). The dispatcher picks it up automatically; nothing above it is edited.
Top-level wiring is additive (option A). The cicd VM keeps its own
nixos-rebuild/testfor the VM itself and drives the manager/controller dispatchers — the dispatch is added alongside the existing VM lifecycle, it does not replace it.
Compiled components
A component that ships a built/packaged artifact must rebuild the package and
refresh its bin/ entry-point symlinks in install.sh/update.sh — not merely copy
source. This is what makes a code change get picked up on update.
- Python (e.g.
opnsense-controller,update-tappaas) — nix build /pip install -eof the component'spyproject.toml, then relink its entry points (the former whole-VMpre-update.shbehaviour, now per-component). - TypeScript (future) —
npm/pnpm install && build, then link the bin. - Bash — nothing to compile; just symlink the entry script onto
PATH.
The build step must be idempotent: a no-op when its inputs are unchanged.
Testing: fast vs deep slices
Every component test.sh runs a fast, non-disruptive slice by default and a deep
slice only when TAPPAAS_TEST_DEEP=1:
- Fast (default) — schema/CLI/validation + mocked-logic unit tests. No live services, no Authentik mutation, no cluster/VM ops. Quick and safe anytime.
- Deep (
TAPPAAS_TEST_DEEP=1) — adds the disruptive/slow tests: live Authentik reconcile, cluster ops, VM provisioning. These mutate real state, so they are scoped (e.g.zztest-names) and self-cleaning.
The cicd module gate honours the split:
module-manager module test tappaas-cicd(fast, ~seconds) runs the quick checks plus Test 11, a lightweight per-component smoke using the already-built bins (validate schemas, CLI loads, config reads) — confirms basic functionality without touching anything.module-manager module test tappaas-cicd --deepruns the VM/variant suites and themanager/+controller/dispatchers withTAPPAAS_TEST_DEEP=1, so every component's full suite (offline unit + live tiers) runs.
When adding a component: gate its disruptive tests behind
[[ "${TAPPAAS_TEST_DEEP:-0}" == "1" ]], and add a one-line smoke to Test 11.
A third axis: runtime vs source-tree
Fast/deep is about cost. --runtime-only (TAPPAAS_TEST_RUNTIME_ONLY=1) is about
subject, and it cuts across both. A check is runtime if it asks something about the
running system (a bin loads, a timer is active, a node answers SSH) and source-tree if
it asks something about the repository (an offline unit suite over temp fixtures, a
generated-doc drift check, a grep guard over a tracked file).
update-module.sh's Step 2 pre-update test is a gate on a mutation, so it passes
--runtime-only: a source-tree defect must never abort the update of a healthy module. A
stale generated README did exactly that on three consecutive nightly sweeps
(2026-09-07..09) — the mothership stopped updating itself and the fleet's shared manager
binaries went unrebuilt (#595). Every other caller — an operator, the deep sweep, CI —
runs everything.
Keep new checks on the right side of that line, and put source-tree checks inside the guarded region rather than among the runtime ones.
Self-update: the control plane refreshes before the sweep, not inside it
scripts/refresh-control-plane.sh is the mothership updating itself: pull the tracked
repositories, relink scripts/*.sh + lib/*.sh into ~/bin, and rebuild every compiled
component through the manager/ + controller/ dispatchers.
It runs as the unit's second ExecStartPre step (tappaas-self-prepare.sh), ahead of
every module and of the mothership's own nixos-rebuild (ADR-017 D3); pre-update.sh still
calls it so module modify tappaas-cicd stands alone. That placement is the point. The work used to live inline in pre-update.sh, which update-module.sh runs at
Step 3 — after the Step 2 pre-update test. So the git pull sat behind a test of the
code it would replace: a commit that broke a fast-mode check wedged the mothership, because
the pull that would carry the fix could no longer run (#595).
Two properties follow, and both matter:
- Idempotent, so
pre-update.shstill calls it (a standalonemodule modify tappaas-cicdmust be correct on its own) and the second run is a no-op. - Exit code 10 means STALE: the pull and relink succeeded, but a component failed to
build, so its bin is the previous one. Not fatal here — but since #719 the self-check
after the rebuild requires every flake component built at the path the new generation
implies and linked from
~/bin, so a stale one fails it, the generation rolls back and the sweep does not run (ADR-028 D10). It reaches the sweep summary andlast-update-result.jsonascontrol_plane. A stale control plane is the root cause that otherwise presents as N unrelated-looking module failures.
The sweep runs in the operator's context (#767)
update-tappaas.service gives each step the context of the tappaas console, because that is
where every change is tested: tappaas steps run in a login shell (bash -l -c), root steps
through sudo -n, and the unit has no sandbox (ADR-017 D3, amended). Three sweeps
failed on the difference before (#713 PATH, #742 sudo under NoNewPrivileges, #719 root's
HOME) — each worked by hand.
A systemd service still is not a terminal: no TTY, no SSH agent, a timer start. So the rule for
any change to code the sweep runs: verify it with scripts/tappaas-unit-run.sh, which runs
a command exactly as the unit runs its steps (read from the live unit) — not only from the
console, and never only through sudo by hand:
tappaas-unit-run.sh -- update-tappaas --dry-run # a tappaas step
tappaas-unit-run.sh --root -- tappaas-selfcheck.sh # a root step (sudo -n, as the unit)
The deep suite does the same for the self-check and the sweep's plan (Deep Test: the sweep's own context).
Config migrations run in that same step, before the rebuild
scripts/run-migrations.sh applies the pending migrations/NNNN-*.sh (ADR-025) from inside
tappaas-self-prepare.sh, immediately after the refresh above and before the hand-over. That
slot is the whole design:
- the migrations it runs are the ones the refresh just pulled;
- it is before
tappaas-self-rebuild.sh, so a migration may change config the new system generation will read —updateScheduleis the first one that must (ADR-017 D7); - it is before the sweep's first module,
clusterincluded, whichpre-update.shcannot be: that hook runs inside the sweep, after the modules it depends on.
The cost is a contract rather than a hazard: a migration runs before the rebuild and possibly
after a refresh that returned STALE, so it may use only bash, jq, coreutils and the files
under config/ — no manager binary, nothing from the not-yet-switched generation.
A failure stops the unit: no rebuild, no sweep, no module updated, and the #651 notice names
the stage migrate. The site keeps running the old code with a config/ that is untouched or
restorable from config/.migrations/backup/. The ledger (config/.migrations/applied) is
written after each success, so a migration interrupted mid-apply is simply re-run next sweep —
which is safe because every migration is idempotent.
migrations/README.md is the contract for writing one.
The console service (ADR-034)
ui/ is the console: a TypeScript backend built like a manager (lib/nix/ts-manager.nix, no
npm dependencies) and a React front end built from ui/web/package-lock.json, one package,
~/bin/console. tappaas-console.service (in tappaas-cicd.nix, started by a path unit once
the bin exists) runs it as tappaas with ProtectSystem=strict. It reads the site only
through the managers' --json and lib/ts, caches the answers on per-read timers, and
writes nothing but its state directory, /var/lib/tappaas/console/. It accepts the firewall's
address (Caddy and the Authentik gate) and loopback; before identity exists, mgmt and the
admin VPN with the setup token bootstrap.sh writes. See ui/README.md.
The console and the sweep take turns (ADR-034 D4)
Both change the site through the same managers, so they never overlap — in either direction:
- The console waits for the sweep. While
update-tappaas.serviceis active (starting, running or stopping) the console starts no job; a queued job whose turn comes during a sweep is refused, naming the stage fromconfig/.update-stage. - The sweep waits for the console. While the console's runner has a write job running it
holds
/var/lib/tappaas/console/console-job.lock(the job's id and argv, the console's pid).tappaas-self-prepare.sh, the unit's prepare step, waits for that lock to go before it pulls, relinks or builds anything — bounded at two hours, after which it proceeds with a warning in the journal; a lock whose console is gone (it stopped mid-job, which its Activity shows as cut off) is ignored. The wait cannot deadlock: the unit isactivatingmeanwhile, so the console admits no new job, and the running one ends.
The same reasoning keeps the console's own restart off a running job: ui/install.sh, having
linked a new build, leaves restart-pending in the console's state directory when a job runs
(it is reached from module update tappaas-cicd, itself a job) and the console exits into
the new build when its runner is idle. A NixOS switch that changes the console's unit does the
same: the unit is reloaded, not restarted (reloadIfChanged), and its reload leaves the marker.
Notices: one mailer, conditions, support bundles
Every notice the site sends goes through scripts/site-notify.sh to site.json email,
through a Proxmox node's own mail system (the mothership has no mailer; the nodes are
tried in turn). The senders — the update notices, the log alert receiver and its
watchdog, the security scan, the switch read — only say what to send.
- A condition is named. A notice about something that can clear carries a key
(
update-sweep,grafana:<rule>:<host>,alert-watchdog). Its first ALERT records the key as open in/var/lib/tappaas/alerts-open/, with the mail's Message-ID. A RESOLVED is sent only for an open key, as a reply to that mail, and closes it. So a RESOLVED never arrives for an alert nobody saw, and a mail client shows the two together. Something that only happened (an OOM kill, a lost quorum) has no key and no RESOLVED. - A support bundle goes with the alerts a supporter could act on: a failed or
degraded update, the log alerts about the cluster, backups and memory, log alerting
being down. Not disk errors (the mail names the disk), not security events (they stay
on the site), not the CVE scan, never a RESOLVED.
tappaas-support-bundle.shpacks the journal of the unit concerned, the module test logs of the window, the site's state (versions, failed units, modules, disk, cluster, open alerts) and its configuration;support-anonymize.pylets in only the files that describe how the site is built and replaces personal data and secrets in them and in every log alike, keeping domains, private addresses and module, environment and zone names — what a supporter debugs with. The map from originals to replacements stays next to the bundle, never in it. - Nothing goes to the maintainers by itself. The mail tells the owner to forward it
to
site.jsonsupportEmail, which is empty until the owner sets one.
Preferred language
In order: TypeScript → Python → Bash. Pick the highest applicable tier for a new
component: prefer TypeScript; use Python where an existing package/ecosystem fit makes
it cheaper (e.g. extending opnsense-controller); reserve Bash for thin glue,
install-time scripts, and small wrappers. Existing Bash/Python components are not
rewritten by this rule — they are migrated opportunistically.
Scaffolding a new component
Copy the nearest real component and edit in place (the scaffold TEMPLATE dirs were retired — they had drifted from what real components look like):
# a new TypeScript manager — copy a small real one and rename
cp -r manager/site-manager manager/my-manager
# a new bash controller — copy a small real one and rename
cp -r manager/ap-manager controller/my-controller
Then rename the entry point, rewrite src/ (or the entry script), fill in the verb
scripts, and (for a manager) the validate.sh. The dispatcher will run it on the next
install/update/test with no changes above the component directory.