TAPPaaS

Appendix H — The install from the command line

The commands behind the install script and the console, for an operator who wants to run them by hand, script them, or repair a step. The install guides and Appendix E describe the same install as most people run it: the script bootstrap-tappaas.sh on a Mac or PC, then the console's Day 1 page. Everything here can be done from those too.

Defaults (subnet, hostnames, sizing, passwords, branch, …) work out of the box. Anywhere a default is mentioned, it can be changed — see Appendix: install options.

Before you start: hardware, network, domain, credentials and branch are in Appendix D — Preparation.


The six steps

Steps 3, 4, and 6 are optional — a single-node, internal-only site with no managed switches needs only steps 1, 2, and 5.

Step What happens Done when
1 First node: install Proxmox VE (preconfigured media, or stock ISO) Proxmox VE running on tappaas1
2 First node: one command bootstraps minimal TAPPaaS — firewall, network cut-over and the CICD mothership tappaas1 runs behind the firewall; tappaas-cicd owns the platform
3 Additional nodes join over the network, unattended (optional) all nodes in the cluster
4 Certificates for the default domain (optional) — wildcard, or per-service cert strategy configured (or deliberately skipped)
5 Remaining foundation modules + your organisation the 🎉 foundation summary prints
6 Managed switches and Wi-Fi APs (optional) — register inventory, then reconcile switch/AP VLANs + SSIDs in sync with zones.json

Step 1 — Install Proxmox VE on the first node

Install Proxmox VE 9.2 on the first machine. Two ways — pick one option, then continue with Step 2.

Option A — preconfigured install media (recommended)

One script builds a USB stick that installs Proxmox unattended, writes it, and starts the TAPPaaS install when the node is up. The boot disk is chosen on the target itself from its real disks — the smallest SSD or NVMe disk, or both when the smallest two are the same model and size (a mirrored ZFS boot pool) — and everything else is answered at build time, with TAPPaaS defaults for the rest. It runs on your laptop, Mac or Linux — no Proxmox and no repo checkout needed: on Linux it fetches the Proxmox assistant itself, on macOS the build runs in a Debian container, so Docker Desktop must be installed and running.

Fetch the script and run it in an empty directory. It downloads the stock Proxmox VE ISO itself, after the questions: the newest one Proxmox lists, checked against Proxmox's published checksum. To use another version, put that ISO from https://www.proxmox.com/en/downloads/proxmox-virtual-environment in the directory first, or give --iso <iso>.

curl -fsSL "https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/main/src/foundation/cluster/bootstrap-tappaas.sh" \
     -o bootstrap-tappaas.sh && chmod +x bootstrap-tappaas.sh
./bootstrap-tappaas.sh

It asks everything first and then runs without questions. The first question is what to do:

  1. Make the install image and write it to a USB stick (install later, with 3).
  2. Make the install image, write it to a USB stick, and install TAPPaaS on the machine booted from it (the default).
  3. The stick is made already: install TAPPaaS on the machine booted from it. It takes the image made earlier in this directory (or --use <image>) and asks nothing about the image.
  4. Make the install image, boot the machine over the network from this computer (see Option C), and install TAPPaaS.
  5. Only make the install image file.

Then only what that choice needs:

Every question has an option (./bootstrap-tappaas.sh --help; --boot instead of the first question), and only a mandatory value that no option gave is asked. The image is proxmox-ve_9.2-1-tappaas-auto.iso.

When it installs TAPPaaS, it waits for the node, finds it on your network and runs Step 2 for you, unattended, so skip that step's commands. The install keeps running on the node if this machine loses the connection, and the script picks it up again. It takes the network ports the node detects, makes no data pools and generates the firewall's root password, which stays on the node in /root/.opnsense-root-password. To find it the node must be on the same network as this machine; otherwise give its address with --node <address>. To follow an install again, run it again and choose 3. Everything it shows, the install on the node included, also goes to bootstrap-tappaas-<date>.log in this directory. Only you can read it; keep it safe, as the install's output can carry credentials. The passwords you type are not in it.

(If you already have — or prefer — a full checkout: git clone https://codeberg.org/TAPPaaS/TAPPaaS.git and run src/foundation/cluster/bootstrap-tappaas.sh from it instead.)

Boot the target from the stick with the NIC to your upstream router connected — network comes from DHCP, exactly like the PXE flow (the management addressing is applied later by the TAPPaaS install). Connect only that NIC for the install: on a machine with several, the installer can take its address from another network, and keeps that address as the node's fixed one. The install is hands-off and WIPES the boot disk(s) it picks. To pick another, pass --disk at build time: a size (--disk 500GB), a model (--disk "Samsung 990") or a serial; a second --disk mirrors it, --disk ask asks on the target's console — which it also does when the field matches no disk. ⚠ The stick embeds the root password — treat the media like a credential and rewrite it after use. proxmox-ve_9.2-1-tappaas-auto.key beside the image logs in to the node as root: keep it as safe.

Disk standard: the PVE system goes on a ZFS boot pool (one disk, or a mirrored pair), which also carries the guests until the node has a data pool (ADR-037 D1). The tankXY data pools live on other disks and are created later (Step 2, [1/5] / config-storage.sh, or site-manager node storage add), never by the installer.

Option B — stock ISO, manual screens

Download the ISO and create a bootable USB per the official guide: https://pve.proxmox.com/wiki/Installation.

On the installer screens:

Option C — PXE / network-boot the first node (advanced; no USB stick)

When the target can't boot from USB, or you want to reinstall it repeatedly, netboot it from your laptop. bootstrap-tappaas.sh does all of this for you: pick Network-boot the target from this machine after the build, then pick Network / PXE (IPv4) in the target's one-time UEFI boot menu. It needs dnsmasq (brew install dnsmasq, or apt install dnsmasq), and the laptop wired to the target's network. It stops the service once the target has fetched the installer. By hand, on a Mac, the steps are below.

This stands up a temporary PXE service on the Mac — dnsmasq in proxy-DHCP + TFTP mode (it does not hand out IP leases; your existing router keeps doing DHCP), which chainloads iPXE, which then pulls the same preconfigured installer as Option A over HTTP. It reuses the exact netboot repack TAPPaaS uses for follow-on nodes (validated against the PVE 9.2 ISO).

Prerequisites: Docker Desktop + Homebrew on the Mac; the Mac and the target on the same wired L2 network (no VLAN/managed-switch separation between them); the target set to UEFI network boot.

1. Build the preconfigured ISO as in Option A — the install picks the boot disk on the target, so the netboot install is hands-off:

./bootstrap-tappaas.sh --iso proxmox-ve_9.2-1.iso --fqdn tappaas1.mgmt.internal
# → proxmox-ve_9.2-1-tappaas-auto.iso  (answer embedded)
# --disk 500GB / --disk "Samsung 990" / --disk <serial> picks another disk; NOT a
# kernel name like nvme0n1 — NVMe names are not stable across boots.

2. Stage the netboot assets — extract the installer kernel + initrd and append the whole ISO to the initrd (a Linux operation; run it in a container):

mkdir -p pxe
docker run --rm -v "$PWD:/work" -w /work debian:trixie bash -c '
  apt-get update -qq && apt-get install -y -qq xorriso cpio >/dev/null
  iso=proxmox-ve_9.2-1-tappaas-auto.iso
  xorriso -osirrox on -indev "$iso" \
    -extract /boot/linux26    pxe/linux26 \
    -extract /boot/initrd.img pxe/initrd >/dev/null 2>&1
  cp "$iso" pxe/proxmox.iso
  sz=$(stat -c %s pxe/initrd); pad=$(( (4 - sz % 4) % 4 ))
  [ "$pad" -gt 0 ] && head -c "$pad" /dev/zero >> pxe/initrd
  ( cd pxe && echo proxmox.iso | cpio -L -H newc -o >> initrd )
  chmod 644 pxe/*'

3. Fetch iPXE and write its boot script (point it at your Mac's LAN IP):

cd pxe
curl -fSLO https://boot.ipxe.org/x86_64-efi/ipxe.efi   # UEFI targets (most machines)
curl -fSLO https://boot.ipxe.org/undionly.kpxe        # legacy-BIOS targets (optional)
MAC_IP=$(ipconfig getifaddr en0)                # your Mac's LAN interface → IP
cat > boot.ipxe <<EOF
#!ipxe
echo TAPPaaS: booting the Proxmox VE auto-installer
kernel http://${MAC_IP}:8080/linux26 ro ramdisk_size=16777216 rw splash=silent proxmox-start-auto-installer
initrd http://${MAC_IP}:8080/initrd
boot
EOF
cd ..

4. Serve the assets over HTTP (the initrd is ~1.5 GB — HTTP, not TFTP):

( cd pxe && python3 -m http.server 8080 )        # leave running; Ctrl-C when done

5. Start proxy-DHCP + TFTP in another terminal — fill in your LAN subnet, the absolute pxe/ path, your interface and Mac IP:

brew install dnsmasq
cp pxe/boot.ipxe pxe/autoexec.ipxe      # iPXE also looks for this beside itself
cat > pxe/dnsmasq-pxe.conf <<'EOF'
port=0                                  # DNS off — PXE only
interface=en0                           # your Mac's LAN interface
bind-interfaces
log-dhcp
dhcp-range=192.168.2.0,proxy            # YOUR lan subnet + 'proxy' (hands out NO leases)
enable-tftp
tftp-root=/absolute/path/to/pxe
# Proxy mode answers only with pxe-service (dhcp-boot alone sends nothing):
# the firmware gets iPXE, iPXE (user class "iPXE") gets the script over TFTP.
dhcp-userclass=set:ipxe,iPXE
pxe-service=tag:!ipxe,x86-64_EFI,"TAPPaaS: iPXE",ipxe.efi
pxe-service=tag:!ipxe,BC_EFI,"TAPPaaS: iPXE",ipxe.efi
pxe-service=tag:ipxe,x86-64_EFI,"TAPPaaS: installer",boot.ipxe
pxe-service=tag:ipxe,BC_EFI,"TAPPaaS: installer",boot.ipxe
EOF
sudo dnsmasq --no-daemon --conf-file=pxe/dnsmasq-pxe.conf   # Ctrl-C when done

macOS firewall: allow dnsmasq (UDP 67/69 and 4011) and the HTTP port, or turn the firewall off on a trusted lab network. port=0 stops dnsmasq clashing with anything already on port 53.

6. Boot the target from the network — enter its UEFI boot menu (F12/F11/F8, board-dependent) and pick Network / PXE (IPv4). It chainloads iPXE → boot.ipxe → the kernel + initrd over HTTP, then runs the unattended installer, which WIPES the boot disk(s) it picks and reboots into Proxmox.

7. Tear down — once PVE is up (it takes a normal DHCP lease from your router), Ctrl-C both dnsmasq and the HTTP server, then continue with Step 2.

Same version caveat as Option A: this uses the PVE netboot repack (kernel/initrd + ISO-in-initramfs) that follow-on nodes also use. The extract step fails loudly if /boot/linux26 or /boot/initrd.img ever move in a future PVE release.

Step 2 — Bootstrap minimal TAPPaaS on the first node

One command turns the bare Proxmox node from Step 1 into a minimal, running TAPPaaS — the firewall, the network cut-over, and the CICD mothership that owns the platform. Whichever install option you used in Step 1, continue here: run the bootstrap from the Proxmox node console/shell.

Notes:

REPO="https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/"; BRANCH="main"
curl -fsSL ${REPO}${BRANCH}/src/foundation/install.sh >install.sh
chmod +x install.sh && ./install.sh "$REPO" "$BRANCH" --name <site-code> --organization <org> --domain "yourdomain.com"

The install re-launches itself inside a tmux session (tappaas-install) so a dropped SSH connection won't abort it — the firewall step reconfigures networking and can briefly cut your session. If you get disconnected, reconnect to the node and run tmux attach -t tappaas-install. (Opt out with --no-tmux.)

Pass three things up front:

Size Meant for
small A few users. One machine, or up to three small ones. A test or lab system.
medium The default. Up to about 50 people using it at once, many modules in one or two environments.
large 100+ users. Three or more large machines, possibly many environments.

A module states its medium size and, where it scales, a smaller and a bigger one: a small site gets the smaller, a large site the bigger. Each environment can have a size of its own (Configure Environments). Change the site's later with site-manager site modify --size <size> or on the console's Site page: installed modules follow at their next update, and their disks only ever grow. On one machine with 32 GB of memory, choose small: the foundation plus the home and office modules then fit.

On the first node this runs the whole foundation bring-up end-to-end as a five-phase chain — you run it once and watch:

  1. [1/5] Node — Proxmox post-install, the lan/wan bridges (auto-detected: the install NIC with internet becomes WAN; the other, to your downstream switch, becomes LAN) and the Proxmox cluster named <site-code>. Data pools come later, see below.
  2. [2/5] Firewall — downloads and boots the prebuilt OPNsense image at 10.0.0.1, self-configured with unique credentials (no GUI, no installer).
  3. [3/5] Gateway cutover — adds this node's management IP 10.0.0.10 and points its default route + DNS at the firewall. Additive and non-disruptive: the upstream IP is kept, no cables move, your session is not dropped.
  4. [4/5] Sanity check — gateway, DNS, internet.
  5. [5/5] Platform — imports the prebuilt NixOS template and builds the tappaas-cicd mothership (bootstrap.sh clone+nixos-rebuild → reboot → install.sh). The cicd's install.sh then writes the system's configuration, automatically:
    • site.json — the site singleton (email, locations, repos), via create-site.sh --name <site-code> --organization <org> (the site code is the neutral cluster name; the org names the default env/zone)
    • zones.json — the network zones, via network-manager init
    • the mgmt + default <org> environments, via environment-manager add (the minimal-set bootstrap)
    • the foundation modules (cluster, templates, network, tappaas-cicd) and the Caddy reverse proxy

Two names: <site-code> = the Proxmox cluster name = site.json .name; <org> = the default environment name = your organisation name (site.json .defaultEnvironment). For a single-site install they are often the same — just pass --name. The organisation itself isn't created until Step 5 (rest-of-foundation.sh, after the identity provider is up).

The domain you passed configures the reverse proxy. To stop earlier, pass --skip-firewall or --skip-platform; to drive the in-VM cicd install by hand, see Appendix: install options.

When it finishes, tappaas1 is at 10.0.0.10 behind the firewall and tappaas-cicd owns the platform. The rest of the install can continue in the console (http://<tappaas-cicd>:9097/, unlocked with the setup token in ~/config/console-setup-token on the mothership). The bootstrap's end note says how to reach it from where you are:

No VPN peer is made for you: you enrol your own device in Day 1's admin VPN step, or move your computer onto the site's LAN. The VPN stays the way in from outside.

Optionally move your admin laptop onto the downstream switch (you'll get a 10.0.0.x lease; Proxmox at https://10.0.0.10:8006, firewall GUI at https://10.0.0.1) — not required, since the node also keeps its upstream IP until you harden it later.

A data pool on the first node. The firewall and the mothership start on the boot pool. To build a pool on the node's other disks and move them onto it, on the mothership:

site-manager node storage tappaas1      # the node's disks, by serial
site-manager node storage add tappaas1 tanka1 --disks <serial>,<serial> --layout mirror
module-manager module modify network --set storage=tanka1
module-manager module modify tappaas-cicd --set storage=tanka1

The disks move live, without a restart. A disk that carries data needs --wipe.

Step 3 — Add additional nodes (optional; skip for single-node)

Do this after the first node's bootstrap has finished (cicd is up). Follow-on nodes install over the network, fully unattended — no USB stick, no installer screens. The mothership runs a TTL-limited network-boot trap (node-provisioner): a machine you have registered gets installed when it PXE-boots on the management LAN; every other machine is refused.

Wire the new node's LAN NIC to the downstream switch (the 10.0.0.0/24 management network) and find out which disk is its boot disk — that disk is wiped. (Managed inter-node switch? Configure its VLAN trunks first; see the network section.)

The netboot assets are staged automatically by the first-node bootstrap (prepare-netboot.sh, run at the end of the cicd install) — adding nodes is a latent capability of every TAPPaaS system. After a PVE version upgrade, re-stage once with site-manager node media prepare --force on the mothership.

Per node — one command:

site-manager node add tappaas2

The command walks the whole flow and asks only what it cannot know:

  1. Registers the node's install answer and opens a TTL-limited boot window (closes on its own; a cluster node is never answered).
  2. Tells you to boot the machine from the network (UEFI PXE, e.g. F11) or from the USB stick image (below). The installer picks the boot disk itself — the smallest SSD or NVMe disk, and both of an identical smallest pair, mirrored (ZFS) — or the one --boot-disk describes (a size such as 2TB, a model or a serial). Only when nothing matches does the node's console ask, listing the machine's real disks; the chosen disk is WIPED. Everything else is hands-off: the installer fetches its per-node answer from the mothership and reboots into PVE. The generated root password lands under ~/.node-secrets/ (0600) — the mothership's ssh key is already authorized.
  3. Asks the join-time questions with the node's real ports on screen: the WAN NIC (only if this node will host firewall HA) and the lan port(s).
  4. Runs the node step (served from the mothership — no GitHub needed), moves the node to its standard mgmt IP (tappaasN → 10.0.0.<9+N>), joins the Proxmox cluster, brings it to the cluster baseline, reboots it into its new kernel, and registers it as its pvehost instance. Its guests go to the boot pool until it has a data pool.
  5. Data pools are a separate step, against the node's real disks: site-manager node storage tappaas2 lists them, and site-manager node storage add tappaas2 tanka1 --disks <serial>,<serial> --layout mirror builds one.

Every question can be pre-answered for a fully unattended run:

site-manager node add tappaas2 --boot-disk 2TB --mac aa:bb:cc:dd:ee:ff --no-wan
# --mac also pre-reserves the standard IP, so the node installs onto it directly;
# with neither --mac nor --switch-port, the first machine that asks gets the answer.
# --boot-disk: a size, a model or a serial (blank: the automatic choice).
site-manager node storage add tappaas2 tanka1 --disks 50026B73842AD86F --layout single

Afterwards, fold the new topology into HA + replication:

site-manager update

Machine can't PXE-boot? Write the installer stick image to a stick — dd if=/var/lib/tappaas-pxe/tappaas-usb.iso of=/dev/<usb> bs=4M (it is also served at http://tappaas-cicd.mgmt.internal:8090/tappaas-usb.iso while the boot window is open) — and boot from USB instead of pressing F11. The flow (and the node add <name> command) is otherwise identical: the answer still comes from the mothership, which matches --boot-disk against the disks the stick reports. The stick cannot ask on the console: a boot-disk field that matches no disk stops the install with the list of the machine's disks.

Manual install instead? Install from a stock ISO or the Step 1 Option A stick, authorize the cicd key for root, then run site-manager node add tappaas2 --adopt <its IPv4> — it finds the Proxmox there and runs the same join + capture pipeline, which moves it to the node's standard mgmt IP. The console's Add node wizard always installs. The underlying tools remain available for surgery: node-provisioner register/enable/disable/status and dhcp-manager pxe/host.

Step 4 — Set up certificates for the default domain (optional)

This step is optional — it provisions the HTTPS certificate that lets your TAPPaaS services be reached over TLS on <service>.yourdomain.com. Skip it if:

Your domain is already configured (you passed --domain in Step 2). Pick one of the two certificate strategies below.

(The public domain lives per-environment. To change the default environment's domain later: environment-manager modify <system-name> --domain <yourdomain>, then set up certificates.)

Option A — one wildcard certificate via DNS-01 (recommended)

The default TLS strategy (proxyTls: dns01) issues one wildcard certificate per TAPPaaS domain via ACME DNS-01, then binds it to every module's reverse-proxy entry through Caddy's CustomCertificate. DNS-01 needs no inbound :80 traffic, so internal-only services get a public cert too.

Why not Caddy's own DNS-01? Since os-caddy 2.0.0 the OPNsense build only ships the Cloudflare DNS provider. TAPPaaS uses os-acme-client (which wraps acme.sh) so you can pick any of the 120 DNS APIs it supports — Cloudflare, deSEC, Hetzner, OVH, Route 53, Namecheap, etc.

Run the operator-driven setup on the mothership:

ssh tappaas@tappaas-cicd      # or its 10.0.0.x address
acme-setup.sh                 # interactive: picks up domain/email, prompts for token

The script:

  1. Reads the domain from the default environment file (config/environments/<system-name>.json, domains.primary) and the admin email from site.json (.email).
  2. Prompts for your DNS provider (default cloudflare) and its API token (offers to save them to ~/.acme-dns-credentials.txt, mode 600, for re-runs).
  3. Provisions an ACME account, a DNS-01 validation, a caddy-reload automation action, and a wildcard certificate (*.<domain> + bare apex) on the firewall — then signs it and waits for issuance (~10–30 s).
  4. Stores the issued cert's OPNsense Trust refid in the runtime config/cert-refids.json, keyed by environment name (ADR-007c). Every later proxyTls: dns01 module install reads this refid and binds the wildcard via CustomCertificate automatically.

Using Cloudflare? Its DNS-01 token needs specific scopes — see Appendix: Cloudflare API token scope.

# Non-Cloudflare? Pass the provider name; the script asks for that provider's fields:
acme-setup.sh --provider hetzner          # deSEC, hetzner, ovh, route53, ...
# Or seed creds non-interactively (chmod 600):
cat >~/.acme-dns-credentials.txt <<EOF
provider=cloudflare
dns_cf_token=YOUR-TOKEN-HERE
EOF
chmod 600 ~/.acme-dns-credentials.txt
acme-setup.sh

To test before going live, add --staging once (Let's Encrypt staging = untrusted certs, no rate limits); then re-run without --staging for the trusted prod cert (the script swaps the cert in place — Caddy picks up the new one automatically via the registered caddy-reload action).

Option B — per-service certificates via HTTP-01 (no DNS API needed)

Choose this if you don't have ACME / DNS-01 API access to your DNS provider — i.e. you can't (or don't want to) issue an API token that lets TAPPaaS edit DNS records, which Option A's DNS-01 requires. HTTP-01 validates each service over inbound port 80 instead of via DNS, so it needs no DNS API token — but each publicly-reachable service gets its own per-domain certificate (no shared wildcard), and that service must be reachable from the internet on :80 for issuance and renewal.

There is no wildcard to provision, so you don't run acme-setup.sh. Instead set the TLS mode to http01 on each module you want publicly reachable, in its <module>.json:

{ "proxyTls": "http01" }

Caddy then obtains and renews a per-domain certificate for that service via the ACME HTTP-01 :80 challenge. The two strategies coexist per module, so you can keep the wildcard (Option A) for most services and use HTTP-01 for a specific public one.


Step 5 — Complete the foundation

From here on you work on the cicd mothership. Your laptop's key is not on it yet, and its name resolves only behind the firewall, so go through the node's Shell (Datacenter → tappaas1 → Shell):

ssh -i /root/tappaas/tappaas-cicd.key tappaas@tappaas-cicd.mgmt.internal

One command does two things:

rest-of-foundation.sh
  1. Installs the remaining foundation modules in order — backup → identity → logging — then runs a final system update + tests.
  2. Bootstraps your people domain — once the identity provider (Authentik) is up and config/identities/ is still empty (first install), it creates the organisation <org> (the --organization from Step 2), the users group and your installer user (from site.json's email), and pushes them into Authentik. So this is where your organisation is actually created — the earlier --organization only reserved the name; the org entity is materialised here.

It's idempotent — safe to re-run if a module needs attention (the people bootstrap is skipped once config/identities/ exists, so it never disturbs people you've added). When it finishes you'll see a "🎉 your TAPPaaS foundation is installed" summary (nodes, firewall, mothership, domain/TLS, modules, organisation).

Prefer to do it by hand? module-manager module add reads ./<module>.json from the current directory, so cd in first: bash cd ~/TAPPaaS/src/foundation/backup && module-manager module add backup # Proxmox Backup Server cd ~/TAPPaaS/src/foundation/identity && module-manager module add identity # Identity provider cd ~/TAPPaaS/src/foundation/logging && module-manager module add logging # Loki / Grafana / Alloy (Module sizing/zones are defaults — see appendix.)


Step 6 — Managed switches and Wi-Fi APs (optional)

Skip this entirely if you have unmanaged switches and no TAPPaaS-driven APs — an unmanaged switch passes tagged frames transparently, so nothing to configure (see Network — cutting over to the firewall for the inter-node trunk story). Do this step if you have managed switches or Wi-Fi access points whose per-zone VLANs/SSIDs you want TAPPaaS to keep in sync.

TAPPaaS treats switches and APs as two more reconcile providers alongside OPNsense and Proxmox — all driven from zones.json (the VLAN tags and the per-zone SSID field). Unlike the firewall/Proxmox planes, these are not applied automatically by rest-of-foundation.sh or update-tappaas: they need an inventory you register once, and (for hardware without a vendor plugin) a manual "tag these VLANs" step. Run everything below from the mothership (ssh tappaas@tappaas-cicd).

6a — Switches

Fastest path is the interactive bootstrap, which registers each brand/switch/uplink port and then applies:

setup-switches.sh          # interactive; ends by running `switch-manager reconcile --apply`

Or register by hand with the switch-manager CLI, then reconcile:

# A manual switch with two node-uplink (trunk) ports:
switch-manager add-switch core --vendor tplink --managed manual
switch-manager add-port  core 9  --type node --target tappaas1 --target-port eth0
switch-manager add-port  core 10 --type node --target tappaas2 --target-port eth0
switch-manager reconcile --apply     # manual switch → prints the VLANs to tag (and records them)
#   …tag those VLANs on ports 9 & 10 in the switch UI…

6b — Wi-Fi access points

A WiFi network is one site-wide SSID bridged to one zone and broadcast by every access point. Register the AP(s), declare each SSID on its zone (this replaces the zone's <…_SSID> placeholder), give each WPA SSID its passphrase (stdin or a file — it goes to a 0600 file, never into a config), and reconcile:

ap-manager add livingroom --vendor unifi --ip 10.0.0.21
ap-manager link livingroom --switch core --port 5      # the AP's uplink port
ap-manager ssid add HomeWiFi  --zone home  --security wpa2-personal
ap-manager ssid add GuestWiFi --zone guest --security wpa2-personal
ap-manager ssid set-passphrase HomeWiFi                 # asked twice, hidden
ap-manager ssid set-passphrase GuestWiFi
ap-manager reconcile --apply

setup-wlan-secrets.sh walks the zones still on a placeholder and asks the same questions; --list shows each zone's SSID and whether its passphrase is set. A changed passphrase (ssid set-passphrase again) reaches every access point at the next reconcile --apply.

Re-running

All of the above is idempotent. After the initial setup, a full network-manager reconcile --apply re-converges every plane (OPNsense, Proxmox, switch, ap) against zones.json; add --only switch / --only ap to scope it. Manual switches always report as "needs-manual" on apply (they can't self-program) — that's expected, not an error. Full reference: network/scripts/README.md (ADR-008).


Next steps

Do this one first: take your backup encryption key off the machine.

Put a USB stick into tappaas1 and run, on the mothership:

backup-manager key export --usb

It writes every backup key to the stick, encrypted with a password you choose, checks it by reading it back, and ejects it. Make two — run it once per stick — and keep them at two different places, both off-site, with the password elsewhere (RESTORE.md §0). Rewriting a stick later replaces only its keys and keeps anything else you store on it, such as a release signer's emergency key.

Backups are encrypted client-side, and the keys are escrowed on the mothership — which is inside the system a full rebuild would be recreating. Without a copy that lives somewhere else, a site-loss restore has nothing to decrypt with, and no amount of surviving backup data helps (ADR-012 §2.5.1). Store the media off-site. Everything else on this list can wait; this cannot, because the day you need it is the day you cannot make it.

If your first node had no tankc pool, the backup module installed as a shim — dependencies are satisfied but nothing is being backed up yet. Add the pool and run module-manager module update backup; it promotes in place. Check with backup-manager placement.

Check that alert mails reach you. On the mothership:

site-manager support test

Or in the console: Site → Support → Test alert mail.

It sends a test ALERT to the site's email, with a support bundle attached, and then its RESOLVED, which arrives as a reply in the same thread. If neither arrives, check the spam folder and the address (site-manager site show), then the mail log of the first node (journalctl -t postfix/smtp there). Run it again any time, for instance after changing the address.

Later, as a day-two step: a support address. An alert that carries a support bundle says to forward the mail to the site's support address. It is empty after the install — the mail then says to forward it to your support contact. Set it when you have one:

site-manager site modify --set supportEmail=<address>

(or on the console's Site page). Which address that is will come with the TAPPaaS support plans; until then, leave it empty or use your own support contact.

The rest of the foundation is complete. Everything from here is covered on the documentation site: configure environments, add a satellite, add workloads (each stack's install guide), and day-to-day operation — start at the install overview.

Network — cutting over to the firewall

Putting the firewall (a VM on tappaas1, at 10.0.0.1) inline as the gateway is done for you by the bootstrap (Step 2, [3/5]) — config-network.sh --swap-gateway. You normally never run it by hand; this section explains what it does. Each node has two NICs, wired at install time and left in place:

The cutover is additive — nothing is removed, nothing is unplugged:

   internet / upstream router
        │
   [tappaas1 WAN NIC] ─────────────► OPNsense WAN  (DHCP from upstream)
   │                                 node also keeps its install IP here
   ┌───────────────────────────────────────────────┐
   │  tappaas1 (Proxmox)                             │
   │    OPNsense VM:   lan → 10.0.0.1  (the gateway) │
   │    node mgmt:     lan → 10.0.0.10               │   ← default route now via 10.0.0.1
   └──[tappaas1 LAN NIC]──────────────────────────────┘
        │
   [ downstream switch ] ── admin laptop  (optional; 10.0.0.x by DHCP)

Later hardening: once you're managing via the mgmt net (or netbird), run config-network.sh --drop-upstream on a node to remove its upstream IP, so Proxmox is reachable only behind the firewall.

The switch (3-node clusters): the lan bridge is VLAN-aware — it carries the management network untagged (10.0.0.0/24) plus every TAPPaaS VLAN tagged as a trunk. For VMs on different nodes to reach each other on a VLAN, the switch between the nodes must pass those tagged frames. How that works depends on the switch type:

(Single node: no switch needed for VLANs — all VLANs live inside the one node; the lan NIC can go to your existing LAN or any switch.)

Upstream options: the firewall WAN can sit behind your ISP router (port-forward), in bridge mode, or replace the ISP router directly.


Appendix: Cloudflare API token scope

(Extra detail — only relevant if you chose Cloudflare* as the DNS provider for the wildcard certificate in Step 4, Option A.)*

Create a custom token at https://dash.cloudflare.com/profile/api-tokens with permissions Zone → Zone → Read and Zone → DNS → Edit, restricted to your domain. Optionally allow only the firewall WAN IPv4 and IPv6 (or skip the IP filter — the token is already zone-scoped).


Appendix: install options

Defaults are chosen so the commands above "just work". Override as needed:

Default How to change
Branch stable Pass a different branch as the 2nd arg to install.sh (e.g. main).
Hostnames tappaas1/2/3 Set during the Proxmox install; the first node must be tappaas1 (it creates the cluster).
Management subnet 10.0.0.0/24, gateway/firewall 10.0.0.1 config-network.sh --mgmt-ip <CIDR> --gateway <ip>; firewall LAN lives in src/foundation/network/firewall-config.xml.template.
Site code (cluster name) install.sh --name <site-code> (Proxmox cluster + site.json .name; lowercase, ≤15 chars; prompted if omitted).
Organisation (default environment) install.sh --organization <org> (default environment/zone + organisation; lowercase; defaults to the site code, no prompt).
Auto cluster create/join install.sh --cluster / --join / --no-cluster.
Chained first-node install (node→firewall→cutover→sanity→platform) The entry point foundation/install.sh runs all 5 phases on the first node by default; stop earlier with --skip-firewall or --skip-platform. The node step alone is cluster/install.sh.
Gateway cutover (route via firewall) Done automatically by the bootstrap; manual: config-network.sh --swap-gateway (additive — keeps the upstream IP).
Take Proxmox off the upstream net (hardening) config-network.sh --drop-upstream (run later, once you manage via the mgmt net / netbird).
Platform install branch / domain install-platform.sh --branch <name> --domain <domain>.
Automated vs. manual cicd install install-platform.sh automates the in-VM install over SSH; pass --manual-cicd to run bootstrap.sh then install.sh by hand inside the cicd VM instead.
ZFS pools (tankXY, topology) install.sh --pool name=topology:disks (none by default), or site-manager node storage add later.
Firewall root password config-firewall.sh --root-pw <pw> (otherwise generated, in /root/.opnsense-root-password on the node; the API key is always unique per deploy).
VM sizing, storage, network zone per module edit the module's <name>.json (cores, memory, diskSize, storage, zone0/bridge0).
Domain / TLS provided to the firewall/app modules; TLS issuance is DNS-01 by default.
Unattended runs most scripts accept --non-interactive (supply the values via flags).

Field definitions for module JSON are in src/foundation/schemas/module-fields.json. Network zones/VLANs come from the git template src/foundation/tappaas-cicd/manager/network-manager/zones.json, which network-manager init transforms into the live, per-installation config/zones.json at install time.


Appendix: Installing via SSH

The main install guide recommends the Proxmox console, but SSH works fine with proper preparation. The key is using tmux so the install continues if your connection drops during network reconfiguration.

1. Generate and install an SSH key (from your workstation)

If you don't already have an SSH key pair:

# Generate a new key (accept defaults, optionally set a passphrase)
ssh-keygen -t ed25519 -C "your-email@example.com"

Reinstalling? If you previously installed TAPPaaS on the same IP, SSH will refuse to connect due to a changed host key (WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!). Remove the old entry first:

ssh-keygen -R <proxmox-ip>
# Copy the public key to the Proxmox node (use the node's current IP)
ssh-copy-id root@<proxmox-ip>

Now you can SSH without a password:

ssh root@<proxmox-ip>

Do this step — it is how you will reach the nodes from now on. Once the install has finished, every node is key-only: no password logins over SSH, for any user (#19). Root keeps key login, which Proxmox itself and the mothership depend on. The PVE web GUI (its node Shell included) and the physical console do not use SSH and keep working with the root password — that is the way in when a key is missing. From that node Shell you can go on to the mothership, and from the mothership to every VM: the nodes present at install carry the mothership's key for exactly this, as /root/tappaas/tappaas-cicd.key:

ssh -i /root/tappaas/tappaas-cicd.key tappaas@tappaas-cicd.mgmt.internal

To add a key later (another laptop, a colleague), ssh-copy-id no longer works, because it needs a password. Open Datacenter → → Shell in the web GUI and append the public key to the cluster-wide file, which every node reads:

echo 'ssh-ed25519 AAAA… you@laptop' >> /etc/pve/priv/authorized_keys

2. Fix locale warnings

Fresh Proxmox installs may show Perl locale warnings. Fix them:

# Quick fix for current session
export LC_ALL=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

# Permanent fix (then start a new shell)
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && locale-gen && \
  update-locale LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8

3. (Optional) Use tmux for session persistence

tmux keeps your session alive if SSH disconnects during network reconfiguration. This is optional but recommended:

apt update && apt install -y tmux
tmux new -s install

If your SSH connection drops, reconnect and reattach:

ssh root@<proxmox-ip>
tmux attach -t install

Useful tmux commands:

4. Run the install

Run the bootstrap command from Step 2:

REPO="https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/"; BRANCH="main"
curl -fsSL ${REPO}${BRANCH}/src/foundation/install.sh >install.sh
chmod +x install.sh && ./install.sh "$REPO" "$BRANCH" --name <site-code> --organization <org> --domain "yourdomain.com"

Even if your SSH drops during network cutover, the install continues inside tmux. Reconnect with tmux attach -t tappaas-install to see progress.

5. After install

Once complete, you can access: