TAPPaaS

site-manager

The Site manager. The Site is the umbrella over a whole TAPPaaS installation: site-wide identity, its locations, its nodes (each a pvehost instance with its storage pools), backup, update schedule, repositories, and the default environment and owning organization by name. (Domain / DNS / identity are per-environment, owned by environment-manager, not here. The organizations themselves are the files in config/identities/organizations/, owned by identity-manager; the site keeps no list of them.)

What it owns

config/site.json (default ${TAPPAAS_CONFIG:-/home/tappaas/config}/site.json), validated against site-fields.json. A site from before 2.1 may still list its repositories in the legacy config/configuration.json (.tappaas.repositories); the repository tools and the config migrations read them there while site.json has none.

site-manager

The site-manager bin is the front door. It owns the Site as a singleton and the location / node / repository sub-entities, following the same <entity> <verb> shape as network-manager. The heavy git/cluster I/O stays in the still-live bash tools, invoked as thin delegations: add → create-site.sh, repository add/delete → repository.sh, validate → validate-site.sh. The bin owns config CRUD (site modify, node CRUD, the site.json writes) + validate + reconcile.

Entities and verbs

site (singleton)   site show [--json]
                   site modify --set <field>=<value>... [--unset <field>] [--<shortcut> <value>]...
location           location list [--json]
                   location show <name> [--json]
                   location add <name> --country <CC> [--displayName <s>] [--kind site|remote]
                            [--city <c>] [--address <a>] [--timezone <tz>]
                            [--latitude <deg>] [--longitude <deg>] [--primary] [--check]
                   location modify <name> [--set <field>=<value>]... [--<shortcut> <value>]... [--unset <f,g>] [--check]
                   location delete <name> [--check]
node               node list [--json]
                   node add <N> [--adopt <ipv4>] [--boot-disk <d>] [--boot-mirror <d>] [--switch-port <sw>:<p>] [--mac <m>]
                            [--wan-port <if>|--no-wan] [--lan-ports <a,b>]
                            [--lan-mode <m>] [--lan-primary <if>] [--ttl <s>] [--no-reboot]
                            [--config-only [--storage-pools <a,b>]]
                   node storage <node> [--json]
                   node storage add <node> <pool> --disks <d,d> --layout single|mirror|raidz1|raidz2 [--wipe]
                   node usb <node> [--json]
                   node usb write <node> <device> --yes
                   node delete <name> [--apply] [--shutdown|--erase|--lost] [--asset keep|remove]
                               [--archive] [--yes] [--config-only]
                   node reconcile [--apply]
                   node qdevice add <host> [--user <u>] [--force] | remove | status
                   node network show <node> [--json]
                   node network set <node> [--lan-ports <a,b>] [--lan-mode <m>]
                            [--lan-primary <if>] [--mtu <n>] [--no-wan] [--switch-wait <s>]
                            [--config-only] [--apply] [--yes]
                   node network test <node> [--apply] [--yes]
repository         repository list [--json]
                   repository add <url> [--branch <b>] [--managed full|tracked] [--catalog <p>]
                            [--signer SHA256:<fp>]... [--unsigned]
                   repository introduce <name> [--signer SHA256:<fp>]...
                   repository delete <name> [--force]
                   repository reconcile [--apply]
                   repository sync <name> [--dry-run]                (#814)
                   repository hold <name> --reason <text> [--until <30m|12h|7d|date>]
                   repository release <name>
                   repository validate-catalog [<name>] [--strict]   (#463)
                   repository stash list [<name>]                    (#681)
                   repository stash show|restore|discard <name> <sha> [--force]
support            support list [--json]
                   support bundle --reason <text> [--unit <unit>]... [--since <time> [--until <time>]] [--no-cluster]
                            [--mail [--to <addr>] [--note <text>]] [--dry-run]
                   support mail <bundle>|latest [--to <addr>] [--note <text>] [--dry-run]
                   support test [--to <addr>] [--dry-run]
top-level          add --name <site-code> [--organization <org>] [create-site options]  (= create-site.sh)
                   validate [FILE] [--schema-dir PATH]     (= validate-site.sh)
                   (validate also checks updateSchedule and prints the OnCalendar it renders)
                   reconcile [--apply] [--deep]
                   update [--dry-run] [--force] [--no-git-pull]   (starts update-tappaas.service)
                   update --stop                                  (the running update stops after the module in progress)
                   test [--deep] [--verbose]                      (= module-manager test each; one line per module, output in ~/logs/site-test-<time>/)

update runs the whole-site update now by starting update-tappaas.service — the unit the timer starts, so the operator exercises the scheduled path (ADR-017 D4). It follows the run's journal; Ctrl-C only detaches. The options travel in a one-shot request, config/.update-request.json, which the unit claims. --dry-run starts nothing: it reports each repository against its origin, then the sweep's plan. --force updates every module even when its pre-update test failed fatally, and nothing more — it reboots nothing. --allow-disruption opens the downtime window for that run: a module with rebootOk: true may be rebooted or migrated offline, every other module keeps its disruptive changes deferred, and rebootOk: false is never overridden fleet-wide. For one module that is module-manager module update <module> --allow-disruption (ADR-020 v0.10 D8, #633). --no-git-pull (TAPPAAS_NO_GIT_PULL) updates whatever is checked out — refresh-control-plane.sh skips the per-repo pull — so local, not-yet-pushed changes can be tested. --dry-run previews the plan.

update --stop ends a running update gracefully: the module in progress finishes, every later module and the phases after them (suspended modules, node reboots, collation) are skipped, and the run is recorded as interrupted with who asked and what was not attempted. The next update takes the rest. The request is config/.update-stop, naming the run's InvocationID; the sweep reads it between modules, so a stop asked during a long step (a rebuild, a big module) takes effect when that step ends, and one left behind never stops a later run. To stop at once instead, sudo systemctl stop update-tappaas.service — the module in progress is then cut off, not rolled back: run module-manager reconcile <module> --apply on it afterwards.

test runs every deployed module's tests: it iterates module-manager list (foundation + apps, from every registered repository) and runs module-manager test <m>, forwarding --deep. Continue-on-failure; exits non-zero if any module test failed.

Common options: --config-dir DIR, --json (machine output for list/show), --apply (reconcile commits; default is preview), --deep (reconcile cascade), --force (repository delete → repository.sh remove --force).

When a run fails, the site owner hears about it: update-tappaas-failure.service mails site.json email through a Proxmox node's mail system, naming the step that failed (#651; ADR-007e v1.3 owns the target), with a support bundle of the run attached to forward to site.json supportEmail. The next clean run says so: "RESOLVED: update sweep OK again", as a reply to the failure. Every run is also followed by a security scan of what it left running (health-manager report security, ADR-011). site show prints the schedule as it is meant — no weekday under daily/none (#447) — and the TAPPaaS version the site runs, read from its checkout (2.2, or 2.2+14 between boundaries; ADR-028 D13), beside the one it was installed from.

repository hold stops the scheduled sweep from pulling one repository, so a test site can run changes that are not pushed yet (#653). The marker is config/.repo-hold/<repo>.json; every hold expires (default 24h), and an expired hold is removed by the next sweep, which then pulls again. repository list and update show active holds.

repository sync <name> brings one repository to the tip of its declared branch without the sweep (#814). It keeps the sweep's guards (the hold, the auto-stash, no commits that exist only in the checkout) and decides on the fetched tip before anything moves. It refuses, and proposes site-manager update, when the tip carries a config migration the ledger does not hold or changes the mothership's own NixOS configuration: both need steps only the sweep runs. It names every installed module whose source directory changes, and deploys none of them; a change under tappaas-cicd/lib/ is said to reach every module. When the repository is the one the control plane is linked from, refresh-control-plane.sh runs afterwards without a pull. repository reconcile now says how far behind origin each checkout is, rather than calling a stale checkout "Converged".

repository stash is where the local changes a sync had to set aside go to be found (#681). A dirty managed checkout is stashed so the pull can move and the entry is put back afterwards; one that no longer applies is kept, and used to be reported only as a number — git stash list, run by hand in a checkout the operator is told not to edit, was the only way to see what the number meant. One estate carried 12 such entries. stash list names every entry with its repository, its age and the files it holds; stash show prints its diff; stash restore puts it back when it still applies; stash discard --force drops it. Entries an operator stashed by hand carry no repo-sync tag: they are never listed, restored or dropped.

repository validate-catalog checks that a repository's module catalogue says true things (#463): its shape and fields, a moduleJson that exists, stack and vmid agreeing with the module's own file, one name and one VMID per module, and no module in the tree left unlisted. With no argument it checks every managed: full repository. It reports and exits 0 — every catalogue predates the check, and a stale one is no reason to refuse a repository; --strict exits non-zero and is what CI and a contributor run before opening a pull request. repository add runs it too, as a report.

First trust for a clone (ADR-033 D5). A release tag is verified against the signers of the release the site last verified — but a fresh clone has verified nothing yet. On a staging or production branch repository add therefore takes the fingerprints of the keys the repository's owners sign with — --signer SHA256:<fp>, once per key, as ssh-keygen -l prints it, from the owners, out of band. The branch's current release must be signed by one of them (and listed by its own allowed_signers, with a matching tree hash); no release number is given. They are written as the entry's trust.signers, and the release verified goes to config/.trust/<name>.json, where every later update starts. Without them the add is refused, unless --unsigned is given: for a repository that signs nothing, or one taken on trust from the revision cloned now. On an unstable branch none are needed; given ones must be keys the checkout lists. repository introduce <name> runs the same check on an existing checkout — against its trust.signers, or against --signer values, written to site.json once they hold, to confirm a repository's signers or start from new ones; repository reconcile --apply runs it after cloning a missing repository, and removes the clone again if it fails. The TAPPaaS repository's fingerprints are written by the installer, from the ones published in the root INSTALL.md.

site modify --set <field>=<value> sets a field by its name in site-fields.json, dotted (health.assessInterval=15 sets how often health-manager assesses, ADR-041 D2), inside an object (--set supportEmail=help@example.org, --set backup.target=pbs2, --set updateSchedule.hour=3); it repeats, and --unset <field> removes an optional one. The options below are shortcuts, one field each (ADR-020 D11): --email a@b.org is --set email=a@b.org, and both run the same checks. supportEmail — where an alert's support bundle is to be forwarded — is one e-mail address (the schema refuses anything else), empty by default and set as a day-two step. The shortcuts (scalar, site-wide): --channel, --displayName, --owner, --email, --supportEmail ('' clears it), --automaticReboot, --size, --snapshotRetention, --wanIpv6Mode, --wanIpv6Address, --wanIpv6Gateway, --wanIpv6Prefix, --wanIpv6PrefixLength (the WAN's IPv6, network.wan.ipv6; see network-manager's IPV6.md), --backupTarget, --backupOffsite, --primaryLocation, --locale, --keyboard, --networkIsp, --networkPublicIp, --ipv6, --requireSigned, --dnssec, --alertsIncludeLines, --brandingLogo, --brandingLogoDark, --dnsBlocklistsLists|Zones|Allow (comma lists, '' empties one), --ipBlocklists (a comma list of http(s) URLs, '' blocks nothing), --diskAutoGrowEnabled|Threshold|GrowPercent|MinGrowGB|PoolCeiling, and the update schedule. That is every field site-fields.json classes in-place. name, version and defaultEnvironment are immutable: --name, --version and --defaultEnvironment are refused, with the reason the schema gives, and nothing is written. The locations (use location …) and the repositories list (own CRUD) are manual: not modifiable here. The nodes are not a site field: they are their pvehost instances (use node …). The older --locationCountry, --locationTimezone, --locationCity, --locationBuilding, --locationLatitude and --locationLongitude are still accepted for one release and change the primary location; --locationLocale and --locationKeyboard change --locale and --keyboard.

--size small|medium|large says how big the site is (absent: medium). Foundation modules, and the modules of every environment that has no size of its own, take their cores, memory and disk from it: a module's small value at small, its plain value at medium, its large value at large. Modules installed from then on get the new size at once; installed ones at their next update, for every sizing field not set by hand (disks only grow). install.sh --size and create-site.sh --size set it at install; site show prints it.

channel — which release the site takes (ADR-028 D9, D11)

A channel is a promise about risk; the branch is where the code sits. They are separate fields because they are separate statements, and an operator changing one before the other is a normal moment, not an error.

Channel Branch (TAPPaaS repo) What it means
unstable main development is proven here; the test site
staging staging soaks a revision under real use for one boundary
production stable what everyone else runs — the default for a new site
site-manager site modify --channel staging          # the promise
site-manager repository modify TAPPaaS --branch staging   # the code

Either order works. Whichever you do first, the other command warns that the two disagree and names the command that would settle it. It does not refuse: refusing would make the two-step move impossible.

Two moves need --force, because both take the site backwards:

It checks every repository, not just TAPPaaS (ADR-028 D11). Which branch realizes which channel is declared by each repository in channels.json at its root, so setting a channel compares each registered repository's branch against what that repository says, warns where they disagree — or where no channels.json exists, in which case every branch in it reads as unstable — and prints the site-manager repository modify <name> --branch <b> that would settle each. It never switches a branch itself: that is a code change to a live site. site modify --channel refuses without --force and says why.

Both gates are passable on purpose. A release tool that cannot be overridden gets worked around with a text editor, which is worse than an override that announces itself — and each one prints what it is doing when --force lifts it.

Only the TAPPaaS repository is held to the train's branches: a community repository has no staging or stable to track, so its branch is whatever its publisher uses.

location — the site's places

A location is a place with an address: the site's own building, a provider's data centre that runs one of its machines, a backup buddy's site. site.json keeps them in locations, by name, each with a country (required), and optionally a displayName, city, address (the street address, or the provider and data centre), timezone, latitude and longitude. kind says whose place it is: site (the default) for this site's own equipment, remote for another site's.

site-manager location list
site-manager location add hetzner-fsn1 --country DE --city Falkenstein --address "Hetzner FSN1-DC14"
site-manager location modify rodovre --timezone Europe/Copenhagen
site-manager location show hareskov          # its place, and what names it

The primary location (primaryLocation) is where the site is. Its timezone is the site's: every node and guest is set to it at the next update, and the health check compares them with it. It is also what backup-manager validate compares an off-site copy with. The site's first own location becomes primary; location add --primary or site modify --primaryLocation <name> moves it, to a location that records a timezone.

Anything else that is somewhere names a location — a backup peer today — and reads the address from here. A location's name is how they name it, so it never changes: add the new one, point them at it, delete the old one. location delete refuses the primary location and any location something names, and says what does. --check on add, modify and delete (and on site modify) says in one line what would change and writes nothing — [check] would modify location 'hareskov': displayName=Rasmus; site.json otherwise unchanged — or [refused] <why> with exit 1; --check --json prints the whole resulting site.json. The apply reports the same line, done ("modified location 'hareskov': …"). validate reports a primaryLocation that names no location or has no timezone, and anything naming a location the site does not have.

Time and locale are the site's master data

Country, keyboard and timezone are answered by the operator once, on the first Proxmox install, and that node is the only place the answers exist. site add reads them back from it — timedatectl, XKBLAYOUT, LANG — rather than from the mothership, which is a NixOS template whose clock says nothing about where the site is. The country is the exception: the installer asks for it and stores it nowhere, so it stays derived from the timezone. Country and timezone go to the primary location, which site add names after the site code; locale and keyboard are site-wide fields, not part of any location.

What is recorded here is what the rest of the estate is set from: the PXE answer file (node add), the USB installer's defaults (bootstrap-tappaas.sh), and the per-OS convergence of guests and hosts. A value already in site.json is the operator's and is never overwritten — a site add --force re-run fills in what is missing and warns when the recorded value and the master disagree, naming the command that would adopt the master.

latitude / longitude are operator-set: no node knows where it is. They exist for modules that need a position rather than a country.

The nodes are their pvehost instances

The cluster's nodes are not listed in site.json: each is a pvehost module instance, config/<node>.json, with its address, uplinks, location and storagePools (the tankXY zpools it carries). node list lists them (--json: name, address, storagePools, location), in node order — by name, numbers as numbers; the first is the primary node. node add registers one (and allocates its asset, below), node delete deletes it (and keeps or removes the asset, below), node reconcile registers a cluster member that has no instance yet and fills a record's empty pools from the node. Every manager and the bash layer list the nodes the same way (lib/ts/src/nodes.ts, lib/cluster-nodes.sh), and validate refuses a site.json that still carries hardware.

node add — standing up a follow-on node (design N3/N4)

node add <tappaasN> is the operator front door for growing the cluster; hardware-validated end-to-end on 2026-07-07. It stands the node up and nothing more: Proxmox on a ZFS boot pool, the join, the cluster baseline, a reboot into the kernel the baseline installed, and HA maintenance lifted. Its guests go to the boot pool (local-zfs) until node storage add gives it a data pool (below). Three modes:

The join pipeline (shared by install and --adopt): asks for the WAN NIC and the lan ports with the node's REAL ports listed (skip the questions with --wan-port <if>/--no-wan), serves the repo from the mothership on :8090 (committed branch state — no GitHub dependency), runs the node step (install.sh --join), corrects /etc/hosts, seeds node→tappaas1 ssh trust, pvecm add, and registers the node as its pvehost instance (module adopt) with the tankXY zpools already on it and its boot disks by serial (bootDisks), before it registers the node on those pools' cluster storage entries and its boot pool as local-zfs. Every wait is time-boxed and the boot window is closed on every exit path. Fully unattended example:

site-manager node add tappaas2 --switch-port "USW Pro XG 10 PoE:1" --no-wan
site-manager node add tappaas2 --mac aa:bb:cc:dd:ee:ff --no-wan

Where it is cabled comes first. --switch-port <switch>:<port> names the port on a switch TAPPaaS programs: node add makes it a node trunk on the switch, then watches it while the machine boots the installer (switch-manager port-neighbors) and pins the registration to the one MAC that appears there and is no cluster node's. On a switch TAPPaaS does not program, node add prints the port to configure — a trunk, the mgmt network untagged, every active zone's VLAN tagged — and takes --mac. With neither a switch port nor a MAC, the first machine that asks for an install answer while the window is open gets it; keep --ttl short then. A registration that names no MAC never answers a machine that is a cluster node. --mac takes a MAC only.

The asset behind the node (ADR-040 D8). A node is a role an asset plays for a while; the asset is the box, in asset-manager's register. --asset <a> names it when it was recorded before (bought, racked, cabled): node add checks it before anything is installed (a server, in stock or placed, in no role) and takes --switch-port and --mac from its recorded cable and the cabled port's MAC when they are not given. Without --asset, once the node has joined, its DMI vendor, model and serial and its NICs' MACs — what the installer posted when it asked for its answer, kept by node-provisioner on the consumed registration (config/provision/<node>.json.consumed); read over ssh for a node adopted by hand — are matched against the free server assets — serial first, then any port's MAC; two matches are a conflict and nothing is allocated — and when none matches a new asset is recorded for it (srv-NN, the node's name as its display name, vendor, model and serial from the machine, and a port per NIC with its MAC, so the box is matched next time even when its BIOS gives no serial). Either way the asset is allocated to the node, and a placement the operator made under the node's name is the asset's from then on. A known asset (named with --asset, or matched) gets what its record lacks from the machine: its serial, vendor and model, and its NICs' MACs on the ports named as the NICs (a port per NIC on an asset with none); a record whose serial is another machine's is warned about and left as it is. A failure here is a warning naming the asset-manager command to run, never a failed join. node delete releases the asset and keeps it or removes it (below).

Its disks. With the asset settled, node add records the machine's whole disks in it as disk assets (asset-manager disk add <asset> <by-id name> --serial …, each the next dsk-NN with its model, size, kind, bus and WWN), from the same identity: the disks the installer posted, kept on the registration, else lsblk and /dev/disk/by-id over the same ssh. A disk whose serial a disk asset records already is placed in this server instead (asset-manager place <disk> --in <asset>: it moved here), one the asset holds is left alone, and one without a serial is a warning naming the command. Disks that could not be read are left to asset-manager disk adopt after the next report --discover.

Boot disks: two fields, blank is the default. --boot-disk and --boot-mirror each take a size (2TB, 1.92T, 500GB: within 10 %), a model ("Samsung 990") or a serial; the installer matches them against the machine's disks in its own boot, so no Linux disk name is needed. Blank, the boot disk is the smallest SSD or NVMe disk not on USB, and when the smallest two are the same model and size both are taken and the boot pool is mirrored; --boot-mirror none keeps such a pair to one disk. A field that matches nothing, or more disks than it may, makes the node's console ask. A kernel name (nvme0n1) is accepted with a warning: it is the name the installer's boot gives the disk, which NVMe drives do not keep between boots (#729 §3).

node delete — removing a node

node delete <node> previews: the cluster module's remove-node.sh --dry-run lists what still holds the node — a guest on it (with the module and how to move it), a replication job touching it, an HA rule naming it, the site's PBS — and the plan; then what becomes of its record and its asset. --apply carries it out and is refused while anything holds it: the node leaves Proxmox (pvecm delnode, its /etc/pve/nodes directory, its storage entries, known_hosts, and its root key in the cluster's authorized_keys, which its join put there), its DNS entry and reservation and its install registration go, the controller's host keys for it go, and the switch ports recorded for it are parked. One of three ways: --shutdown powers it off first (Proxmox removes a node only once it is off), --erase wipes its disks first, --lost is for a node that is down for good.

Then its record, config/<node>.json, is deleted: the node is gone from the modules list. The CMDB keeps the machine's history — its object is ended, not erased — so nothing is lost for a replacement. --archive keeps the record archived instead, for a script that reads it afterwards.

Its asset is released from the node, and then, by --asset:

A --lost node's asset is removed unless --asset keep is given. --config-only deletes (or archives) the record and keeps or removes the asset without touching the cluster. --yes takes the place of the typed confirmations, for the console (which has the node's name typed) and scripts.

node storage — a node's data pools (ADR-037 D4)

node storage <node> lists the node's pools and its disks in no pool — serial, size, type, model, an old pool's label and SMART health (from storage-manager inventory) — and proposes a pool on the healthy ones. node storage add <node> <pool> --disks <serial,...> --layout <l> builds it on the node (cluster/config-storage.sh, fed from this checkout), registers it as the node's storage and records it in storagePools. It asks nothing: a pool name that is not tank<…> or that the node has, a disk count the layout does not take, a kernel name, the boot disk, a disk in a live pool, or a disk that carries data without --wipe is a refusal that says why, and nothing is changed.

site-manager node storage tappaas2
site-manager node storage add tappaas2 tanka1 --disks S7HENJ0X100001,S7HENJ0X100002 --layout mirror

node media — the boot medium

Every new node installs from one medium on the mothership, /var/lib/tappaas-pxe: the kernel, initrd and boot script a machine network-boots (PXE), and beside them the installer stick image, tappaas-usb.iso, for a machine that cannot. Both come from one ISO, the Proxmox installer prepared to fetch its answer from the mothership (scripts/prepare-netboot.sh makes it on a cluster node and keeps it at /home/tappaas/pve-tappaas-netboot.iso). The cicd install stages it once.

node media says what is there: whether the assets are staged and carry this version's installer hook, the stick image (size, date), the ISO, and what prepare would run. --json is what the console's Add a node, stage 0, reads (ready, problems, prepare). node media prepare builds what is missing or stale: from the ISO already on the mothership when there is one (sudo node-provisioner prepare --iso …, a few minutes), else the whole of prepare-netboot.sh (the ISO fetched and prepared on a node, copied here and staged: ten minutes or more); --force stages it all again, for a new Proxmox version. The stick image needs xorriso on the mothership (in the cicd closure); without it node media says so and prepare stages the netboot assets only.

site-manager node media
site-manager node media prepare

node usb — the installer stick

A machine that cannot network-boot is installed from a USB stick that carries the installer stick image, /var/lib/tappaas-pxe/tappaas-usb.iso (built on the mothership by node media prepare, the same image for every node). node usb <node> lists the sticks plugged into a cluster node — the whole disks on the USB bus, as lsblk reports them; never a SATA/NVMe disk — one line each: device, size, vendor, model and serial, with (in use: mounted) (or zfs_member, LVM2_member, swap, linux_raid_member) when something holds it, and (key stick) when a partition is labelled TAPPAASKEYS (the backup key stick). --json is what the console reads:

{ "node": "tappaas1", "reachable": true,
  "sticks": [ { "device": "/dev/sdb", "size": 30752000000, "vendor": "SanDisk", "model": "Ultra",
                "serial": "4C530001", "inUse": false, "why": null, "keyStick": false } ] }

why is null, "mounted", the file system type that claims the stick, or "ejected: unplug it and plug it in again" for a stick with no medium (size 0): the eject at the end of a write leaves it so until it is pulled and plugged in again, and node usb write refuses it. A node that cannot be reached is an error in the human form; with --json it is {"node": …, "reachable": false, "sticks": []} and exit 0, so the console can say so.

node usb write <node> <device> --yes streams the image over ssh onto the stick (dd, then flush and eject) and erases it, printing its progress every few seconds (percent, MB written, rate, time left). It asks nothing and refuses, writing nothing, without --yes, when the image is not staged, when the node cannot be reached, when <device> is not a stick node usb <node> lists, or when the stick is in use.

site-manager node usb tappaas1
site-manager node usb write tappaas1 /dev/sdb --yes

After the install node add puts the installed disk first in the UEFI boot order, and if the answer's ssh key did not survive the first boot it restores it with the node's generated root password (/home/tappaas/.node-secrets/<name>.pw, the operator's, 0600).

Afterwards run site-manager update to fold HA + replication.

support — support bundles

A support bundle is one file holding what a supporter needs to look at a problem: the logs of the event, the site's state and its configuration, with personal data and secrets removed. Some alert mails carry one; site-manager support makes one when you need it, mails it, and lists the ones kept. The site never sends a bundle to the maintainers: it goes to the site's own email, and you decide whether to forward it.

In the console: Site → Support has the same: Make support bundle, Test alert mail, the bundles kept with a Mail button each, and the two addresses.

Make one:

site-manager support bundle --reason "console shows errors" --unit tappaas-console.service --since -3h
Option What it does
--reason <text> required: why it was made; written into the bundle
--unit <unit> a systemd unit on the mothership whose journal goes in — update-tappaas.service, tappaas-console.service, …; repeatable
--since <time> / --until <time> the time window, in journalctl syntax (-3h, "2026-11-14 08:00"): the journal lines and the module test logs written in it. Without it: the first unit's last run, or the last two hours when no unit is given
--no-cluster do not ask a node for the cluster's state (when the nodes are the problem)
--mail mail it as soon as it is made (see below); --to and --note as for support mail
--dry-run say what would go in and where it would be mailed; make and send nothing

It prints the bundle's path, /var/lib/tappaas/support/tappaas-support-<site>-<time>.tar.gz. Bundles are mode 0600 and the newest 20 are kept.

Mail one:

site-manager support mail latest
site-manager support mail tappaas-support-rossen-20261114-091502.tar.gz --to me@example.org --note "the console errors from this morning"

The bundle is named as support list shows it, or latest. It goes to the site's email unless --to gives one address, through a Proxmox node's mail system like every alert. The mail says to forward it to the site's support address, supportEmail — empty by default; set it with site-manager site modify --supportEmail <addr>. A bundle over 5 MB is not attached: the mail says where it is on the mothership. --dry-run says what would go to whom.

List them: site-manager support list [--json] — newest first, with why each was made and whether it is small enough to attach.

Test the whole path — on a good day, to prove that alert mail and its attachment arrive:

site-manager support test [--to <addr>]

It makes a bundle of the last update run and sends it with a test ALERT, then its RESOLVED as a reply, so the two thread in your mail client. No action is needed on either. Nothing arrives? Check the spam folder and the site's email.

What is in a bundle

Never in it: people and identities, VPN and backup peers, provisioning records, firewall state, the SBOM, and backup copies of config files.

Replaced, in the configuration and every log alike:

What Becomes
secrets (fields named like one, password=… in logs) REDACTED
the owner's and users' names, peers' names person-N
addresses, cities, coordinates place-N
e-mail addresses user-N@<their domain>
public IP addresses 203.0.113.N / 2001:db8::N

Domains, private addresses, MAC addresses and module, environment and zone names are kept, so the bundle still shows how the site fits together. If anonymizing fails, no bundle is written.

Reading a supporter's question. Next to each bundle in /var/lib/tappaas/support/ lies its map, <bundle>.map.json (original → replacement), also mode 0600. It is never in the bundle: when a supporter asks about person-2 or place-1, look it up there.

Alerts that carry a bundle: a failed, degraded or stopped update run; quorum lost, HA fencing, a silent host, a stuck HA service, a lost network leg, failed replication or backup, an out-of-memory kill; and "log alerting is down". Not the disk-error and security alerts, the vulnerability scan, the switch read, or any RESOLVED mail.

reconcile and the --deep cascade

reconcile is shallow by default — it converges the site's own concern: validate site.json, then bring each repositories[] entry to a live clone (clone if missing, checkout if the branch drifts). Default output is a preview; --apply commits. repository reconcile is the repo-scoped subset of the same engine.

reconcile --deep then cascades to the dependent managers in dependency order:

site reconcile --deep
  → identity-manager  reconcile          (people → Authentik)
  → network-manager reconcile          (the 4 network planes — ONE system-wide pass)
  → for each environment in config/environments/*.json:
       environment-manager reconcile <env> --deep --skip-network

people/network are single bins; environments fan out — one deep reconcile per registered environment. The network pass runs once for the whole site: network-manager reconcile has no zone or environment filter, so letting each environment run its own would repeat the identical whole-platform operation once per environment. Every leg is idempotent, so re-running is safe; this is the natural whole-platform converge after update-tappaas.

A cascade that exits non-zero is reported by name and makes site reconcile exit 1 — the remaining legs still run, so one bad environment does not strand the rest.

Build

Built by default.nix into result/bin/site-manager — mirroring identity-manager / network-manager. install.sh is not yet wired to build it (the bash tools below remain the installed entry points for now).

Reports to the CMDB

site-manager is a source of the CMDB (ADR-040 D4; cmdbSource in components.json, so cmdb rescan runs its report).

site-manager report [--json] interrogates the cluster — one ssh to the first node that answers (/cluster/status, /cluster/resources, the HA resources, every guest's config from /etc/pve) and one per node (os, addresses, pve version, tank pools; its disk list, lsblk and its ZFS pools) — and sends one observation; --json prints it instead.

Class Name (key) What
site the site code code, displayName, version (the running TAPPaaS release: the control plane's build, volatile, so a refresh's new build is history, not drift), primaryLocation (at), desired site.json
location its name kind, country, city, address, timezone, latitude, longitude
cluster the site code (pvecm create <site-code>) quorum
machine the node name its pvehost instance: desired, release pvehost + version, hostname, os, osVersion, addresses, clusterMember, storagePools, management, environment mgmt; member-of the cluster, at its location (else the primary)
app pve on its machine version
vm, lxc the module instance, else the guest's name (key: vmid) vmid, cores, memory, disks / rootfs, unprivileged, template, haPolicy, onNode; a module's guest also desired and release
storage-pool its name on its node zfsState, size, used, vdevs[] (each member by device, serial and state), onNode
disk the disk asset's tag, part-of the node's server a disk asset-manager records in the server the node is allocated to, matched by serial: health (Proxmox's word from its disk list), holds, partitions

Each pool is built-on each of its members that is a recorded disk: the members of a mirror or raidz vdev carry redundancyGroup <node>:<pool>:<vdev> (one failed disk degrades the pool, all of them take it down), a log, cache or spare member required: false, a single-disk vdev neither; a recorded disk of the server that is in no pool loses the built-on it had. A disk no asset records is only in the server's disks[] (asset-manager disk adopt records it).

Each guest is stored-on each ZFS pool of its node that one of its disks, its rootfs or a mount point is on (a Proxmox zfspool storage carries its pool's name; local-lvm and other storages are no pool), and its stored-on any other pool of any node ends, so a moved disk or a migrated guest leaves none behind. A pool the report does not describe is named by a stub, by its name on its node.

A guest's fields come from its /etc/pve config through the parse in lib/ts guest-conf.ts, the one module-manager reports an update or modify with.

A cluster that does not answer is a warning: the site, locations, cluster and nodes are reported from config/, without guests. An archived module is not reported.

Each leg of a node's lan uplink (node network show) is a connects of its own report, from the switch's port (LLDP's answer, else the switch inventory's record of the cable) to the NIC on the node's server asset, with redundancyGroup <node>:<bond> (null on a lan of one port), so health degrades the NIC while one leg is down rather than failing it. A leg on a port of an MLAG LAG is switch-manager's to group; a leg the CMDB refuses (its port not reported yet) is a warning and is sent again next time. With such legs --json prints an array: the site's report, then one per leg.

Changes: node add, node delete (the machine ends), site modify and the location verbs (a deleted location ends). node reconcile and reconcile report the membership and pools they read, as a side effect.

report --discover (ADR-040 D15, what cmdb sweep asks for) goes to the full depth, one more ssh per node and nothing polled; without it report is as above:

A verb's change report carries the verb, the console's job (TAPPAAS_JOB_ID) and the actor (TAPPAAS_ACTOR, else the OS user; lib/ts job.ts). A report the CMDB cannot take is spooled in config/.cmdb-spool/ (lib/ts cmdb.ts) and never fails the verb; TAPPAAS_CMDB=off turns reporting off (the tests do). The tests check every payload against cmdb-controller's metamodel (lib/ts/test/cmdb-check.ts).

Commands (legacy bash tools — kept live until cutover)

All scripts are bash, linked onto PATH by install.sh. repository.sh and validate-site.sh remain live and are the tools the repository add/delete and validate verbs delegate to; create-site.sh backs the add verb.

repository.sh — manage module repositories

The tool behind site-manager repository …: it registers the module repositories TAPPaaS pulls modules from (add / remove / modify / list) in site.json repositories[], cloning each under /home/tappaas/<name>/ over HTTPS. A repository is always cloned by add, never adopted from an existing checkout elsewhere. A private repository is reached with a deploy key — see Private repositories.

repository.sh add <url> [--branch <b>] [--managed full|tracked] [--catalog <path>]
repository.sh remove <name> [--force]
repository.sh modify <name> [--url <new>] [--branch <new>]
repository.sh list

validate-site.sh — validate site.json

This is the manager's validate operation, named validate-site.sh per the script-manager validate-<manager>.sh convention; runnable directly.

validate-site.sh [FILE] [--schema-dir PATH] [--quiet]
validate-site.sh

Legacy tools (kept until the flag-day cutover)

These predate site.json and operate on the legacy configuration.json:

create-configuration.sh

Create/update configuration.json by discovering the running Proxmox cluster. Accepts named flags (--upstream-git, --branch, --domain, --email, --schedule monthly|weekly|daily|none, --weekday, --hour, --primary-node, --update) or legacy positionals (<upstreamGit> <branch> <domain> <email> <schedule> [weekday] [hour]). Idempotent.

validate-configuration.sh

Validate configuration.json. Flags: --config <path>, --check-connectivity (ping nodes), --check-cluster (SSH the first node, compare cluster membership), --check-repos (git ls-remote each repo URL), --quiet.

convert-json-to-config.sh

Convert a flat module JSON into the canonical config-block form. CLI: convert-json-to-config.sh [--in-place|--dry-run] <module-json>; or source it and call regroup_to_pattern_a.