Appendix H — The install from the command line
The commands behind the install script and the console, for an operator who wants to run them
by hand, script them, or repair a step. The install guides and
Appendix E describe the same install as most people run it: the
script bootstrap-tappaas.sh on a Mac or PC, then the console's Day 1 page. Everything here can
be done from those too.
Defaults (subnet, hostnames, sizing, passwords, branch, …) work out of the box. Anywhere a default is mentioned, it can be changed — see Appendix: install options.
Before you start: hardware, network, domain, credentials and branch are in Appendix D — Preparation.
The six steps
Steps 3, 4, and 6 are optional — a single-node, internal-only site with no managed switches needs only steps 1, 2, and 5.
| Step | What happens | Done when |
|---|---|---|
| 1 | First node: install Proxmox VE (preconfigured media, or stock ISO) | Proxmox VE running on tappaas1 |
| 2 | First node: one command bootstraps minimal TAPPaaS — firewall, network cut-over and the CICD mothership | tappaas1 runs behind the firewall; tappaas-cicd owns the platform |
| 3 | Additional nodes join over the network, unattended (optional) | all nodes in the cluster |
| 4 | Certificates for the default domain (optional) — wildcard, or per-service | cert strategy configured (or deliberately skipped) |
| 5 | Remaining foundation modules + your organisation | the 🎉 foundation summary prints |
| 6 | Managed switches and Wi-Fi APs (optional) — register inventory, then reconcile | switch/AP VLANs + SSIDs in sync with zones.json |
Step 1 — Install Proxmox VE on the first node
Install Proxmox VE 9.2 on the first machine. Two ways — pick one option, then continue with Step 2.
Option A — preconfigured install media (recommended)
One script builds a USB stick that installs Proxmox unattended, writes it, and starts the TAPPaaS install when the node is up. The boot disk is chosen on the target itself from its real disks — the smallest SSD or NVMe disk, or both when the smallest two are the same model and size (a mirrored ZFS boot pool) — and everything else is answered at build time, with TAPPaaS defaults for the rest. It runs on your laptop, Mac or Linux — no Proxmox and no repo checkout needed: on Linux it fetches the Proxmox assistant itself, on macOS the build runs in a Debian container, so Docker Desktop must be installed and running.
Fetch the script and run it in an empty directory. It downloads the stock Proxmox VE ISO
itself, after the questions: the newest one Proxmox lists, checked against Proxmox's
published checksum. To use another version, put that ISO from
https://www.proxmox.com/en/downloads/proxmox-virtual-environment in the directory first,
or give --iso <iso>.
curl -fsSL "https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/main/src/foundation/cluster/bootstrap-tappaas.sh" \
-o bootstrap-tappaas.sh && chmod +x bootstrap-tappaas.sh
./bootstrap-tappaas.sh
It asks everything first and then runs without questions. The first question is what to do:
- Make the install image and write it to a USB stick (install later, with 3).
- Make the install image, write it to a USB stick, and install TAPPaaS on the machine booted from it (the default).
- The stick is made already: install TAPPaaS on the machine booted from it. It takes the
image made earlier in this directory (or
--use <image>) and asks nothing about the image. - Make the install image, boot the machine over the network from this computer (see Option C), and install TAPPaaS.
- Only make the install image file.
Then only what that choice needs:
- for the image: your admin email (for email alerting) and a root password for the
node. Country, keyboard, timezone and hostname are not asked: it takes this machine's
settings and shows them;
--country,--keyboard,--timezoneand--fqdnchange them. - for the stick: it lists the USB sticks in this machine and erases the one you pick, after you confirm it, then reads the stick back. For the network boot: the interface on this machine wired to the target's network.
- for the install: the branch, the Site name (Step 2's site code), the organisation controlling the Site, the domain, and the site's size: small (a few users, one node or a test system), medium (up to ~50 users; the default) or large (100+ users, 3+ large nodes). The size sets every module's cores, memory and disk (see Step 2).
- your login password on this computer (
sudo), once, for the stick or the network boot.
Every question has an option (./bootstrap-tappaas.sh --help; --boot instead of the
first question), and only a mandatory value that no option gave is asked. The image is
proxmox-ve_9.2-1-tappaas-auto.iso.
When it installs TAPPaaS, it waits for the node, finds it on your network and runs Step 2 for
you, unattended, so skip that step's commands. The install keeps running on the node if
this machine loses the connection, and the script picks it up again. It takes the network
ports the node detects, makes no data pools and generates the firewall's root password,
which stays on the node in /root/.opnsense-root-password. To find it the node must be on the same network as
this machine; otherwise give its address with --node <address>. To follow an install
again, run it again and choose 3. Everything it shows, the
install on the node included, also goes to bootstrap-tappaas-<date>.log in this
directory. Only you can read it; keep it safe, as the install's output can carry
credentials. The passwords you type are not in it.
(If you already have — or prefer — a full checkout:
git clone https://codeberg.org/TAPPaaS/TAPPaaS.git and run
src/foundation/cluster/bootstrap-tappaas.sh from it instead.)
Boot the target from the stick with the NIC to your upstream router
connected — network comes from DHCP, exactly like the PXE flow (the
management addressing is applied later by the TAPPaaS install). Connect only
that NIC for the install: on a machine with several, the installer can take its address
from another network, and keeps that address as the node's fixed one. The install
is hands-off and WIPES the boot disk(s) it picks. To pick another, pass
--disk at build time: a size (--disk 500GB), a model (--disk "Samsung 990")
or a serial; a second --disk mirrors it, --disk ask asks on the target's
console — which it also does when the field matches no disk.
⚠ The stick embeds the root password — treat the media like a credential
and rewrite it after use. proxmox-ve_9.2-1-tappaas-auto.key beside the image logs in to
the node as root: keep it as safe.
Disk standard: the PVE system goes on a ZFS boot pool (one disk, or a mirrored pair), which also carries the guests until the node has a data pool (ADR-037 D1). The
tankXYdata pools live on other disks and are created later (Step 2, [1/5] /config-storage.sh, orsite-manager node storage add), never by the installer.
Option B — stock ISO, manual screens
Download the ISO and create a bootable USB per the official guide: https://pve.proxmox.com/wiki/Installation.
On the installer screens:
- Network Nic select the one you have connected to the upstream router. Initially this is the Lan port but it will eventually become the "wan" port of the TAPPaaS firewall. For secondary TAPPaaS nodes this is will stay lan port, as these nodes wil connect directly via the switch to the lan side of the firewall we create in the first node.
- Hostname (FQDN):
tappaas1.mgmt.internal— TAPPaaS uses the internal management domainmgmt.internal, not your public domain. (Your public domain is supplied with--domainin Step 2.) - Email: a working address you actually monitor — Proxmox sends system/health notifications here, and TAPPaaS reuses it as the admin email: its alert mails (a failed update, a lost quorum, a silent host …) go here too.
- Management interface / IP / netmask / gateway / DNS: must be valid for your existing network — use a free IP (the proxmox installer is not using DHCP), and the real gateway and DNS server of the network this node currently sits on (so it has internet for the bootstrap).
Option C — PXE / network-boot the first node (advanced; no USB stick)
When the target can't boot from USB, or you want to reinstall it repeatedly,
netboot it from your laptop. bootstrap-tappaas.sh does all of this for you: pick
Network-boot the target from this machine after the build, then pick Network / PXE
(IPv4) in the target's one-time UEFI boot menu. It needs dnsmasq
(brew install dnsmasq, or apt install dnsmasq), and the laptop wired to the target's
network. It stops the service once the target has fetched the installer. By hand, on a
Mac, the steps are below.
This stands up a temporary PXE service on the Mac —
dnsmasq in proxy-DHCP + TFTP mode (it does not hand out IP leases; your
existing router keeps doing DHCP), which chainloads iPXE, which then pulls the
same preconfigured installer as Option A over HTTP. It reuses the exact netboot
repack TAPPaaS uses for follow-on nodes (validated against the PVE 9.2 ISO).
Prerequisites: Docker Desktop + Homebrew on the Mac; the Mac and the target on the same wired L2 network (no VLAN/managed-switch separation between them); the target set to UEFI network boot.
1. Build the preconfigured ISO as in Option A — the install picks the boot disk on the target, so the netboot install is hands-off:
./bootstrap-tappaas.sh --iso proxmox-ve_9.2-1.iso --fqdn tappaas1.mgmt.internal
# → proxmox-ve_9.2-1-tappaas-auto.iso (answer embedded)
# --disk 500GB / --disk "Samsung 990" / --disk <serial> picks another disk; NOT a
# kernel name like nvme0n1 — NVMe names are not stable across boots.
2. Stage the netboot assets — extract the installer kernel + initrd and append the whole ISO to the initrd (a Linux operation; run it in a container):
mkdir -p pxe
docker run --rm -v "$PWD:/work" -w /work debian:trixie bash -c '
apt-get update -qq && apt-get install -y -qq xorriso cpio >/dev/null
iso=proxmox-ve_9.2-1-tappaas-auto.iso
xorriso -osirrox on -indev "$iso" \
-extract /boot/linux26 pxe/linux26 \
-extract /boot/initrd.img pxe/initrd >/dev/null 2>&1
cp "$iso" pxe/proxmox.iso
sz=$(stat -c %s pxe/initrd); pad=$(( (4 - sz % 4) % 4 ))
[ "$pad" -gt 0 ] && head -c "$pad" /dev/zero >> pxe/initrd
( cd pxe && echo proxmox.iso | cpio -L -H newc -o >> initrd )
chmod 644 pxe/*'
3. Fetch iPXE and write its boot script (point it at your Mac's LAN IP):
cd pxe
curl -fSLO https://boot.ipxe.org/x86_64-efi/ipxe.efi # UEFI targets (most machines)
curl -fSLO https://boot.ipxe.org/undionly.kpxe # legacy-BIOS targets (optional)
MAC_IP=$(ipconfig getifaddr en0) # your Mac's LAN interface → IP
cat > boot.ipxe <<EOF
#!ipxe
echo TAPPaaS: booting the Proxmox VE auto-installer
kernel http://${MAC_IP}:8080/linux26 ro ramdisk_size=16777216 rw splash=silent proxmox-start-auto-installer
initrd http://${MAC_IP}:8080/initrd
boot
EOF
cd ..
4. Serve the assets over HTTP (the initrd is ~1.5 GB — HTTP, not TFTP):
( cd pxe && python3 -m http.server 8080 ) # leave running; Ctrl-C when done
5. Start proxy-DHCP + TFTP in another terminal — fill in your LAN subnet, the
absolute pxe/ path, your interface and Mac IP:
brew install dnsmasq
cp pxe/boot.ipxe pxe/autoexec.ipxe # iPXE also looks for this beside itself
cat > pxe/dnsmasq-pxe.conf <<'EOF'
port=0 # DNS off — PXE only
interface=en0 # your Mac's LAN interface
bind-interfaces
log-dhcp
dhcp-range=192.168.2.0,proxy # YOUR lan subnet + 'proxy' (hands out NO leases)
enable-tftp
tftp-root=/absolute/path/to/pxe
# Proxy mode answers only with pxe-service (dhcp-boot alone sends nothing):
# the firmware gets iPXE, iPXE (user class "iPXE") gets the script over TFTP.
dhcp-userclass=set:ipxe,iPXE
pxe-service=tag:!ipxe,x86-64_EFI,"TAPPaaS: iPXE",ipxe.efi
pxe-service=tag:!ipxe,BC_EFI,"TAPPaaS: iPXE",ipxe.efi
pxe-service=tag:ipxe,x86-64_EFI,"TAPPaaS: installer",boot.ipxe
pxe-service=tag:ipxe,BC_EFI,"TAPPaaS: installer",boot.ipxe
EOF
sudo dnsmasq --no-daemon --conf-file=pxe/dnsmasq-pxe.conf # Ctrl-C when done
macOS firewall: allow
dnsmasq(UDP 67/69 and 4011) and the HTTP port, or turn the firewall off on a trusted lab network.port=0stops dnsmasq clashing with anything already on port 53.
6. Boot the target from the network — enter its UEFI boot menu (F12/F11/F8,
board-dependent) and pick Network / PXE (IPv4). It chainloads iPXE →
boot.ipxe → the kernel + initrd over HTTP, then runs the unattended
installer, which WIPES the boot disk(s) it picks and reboots into Proxmox.
7. Tear down — once PVE is up (it takes a normal DHCP lease from your router),
Ctrl-C both dnsmasq and the HTTP server, then continue with
Step 2.
Same version caveat as Option A: this uses the PVE netboot repack (kernel/initrd + ISO-in-initramfs) that follow-on nodes also use. The extract step fails loudly if
/boot/linux26or/boot/initrd.imgever move in a future PVE release.
Step 2 — Bootstrap minimal TAPPaaS on the first node
One command turns the bare Proxmox node from Step 1 into a minimal, running TAPPaaS — the firewall, the network cut-over, and the CICD mothership that owns the platform. Whichever install option you used in Step 1, continue here: run the bootstrap from the Proxmox node console/shell.
Notes:
- Prefer the console — use the xterm.js shell option in the tappaas1 menu; it gives more scrollback and persistence on the install output.
- SSH works too — see Appendix: Installing via SSH for setup steps (known_hosts, locale, tmux).
- ⚠ Run from a client that is NOT on
10.0.0.0/24. The cutover puts the management network (10.0.0.0/24) on the node, so a browser/SSH client that sits in that subnet loses its return path to the node and freezes. Drive the install from a client on your install/upstream network, or from a true out-of-band console (IPMI/iKVM) — not from a10.0.0.xlaptop. (The node keeps its install IP throughout, so such a client never loses it.)
REPO="https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/"; BRANCH="main"
curl -fsSL ${REPO}${BRANCH}/src/foundation/install.sh >install.sh
chmod +x install.sh && ./install.sh "$REPO" "$BRANCH" --name <site-code> --organization <org> --domain "yourdomain.com"
The install re-launches itself inside a
tmuxsession (tappaas-install) so a dropped SSH connection won't abort it — the firewall step reconfigures networking and can briefly cut your session. If you get disconnected, reconnect to the node and runtmux attach -t tappaas-install. (Opt out with--no-tmux.)
Pass three things up front:
--name <site-code>— the site code (lowercase, ≤15 chars). It names the Proxmox cluster andsite.json.name, and is fixed once the cluster exists. If omitted you'll be prompted.--organization <org>— your organisation name (lowercase). It names the default environment / zone and (later) the organisation in the identity provider. Optional: if omitted it defaults to the site code, without prompting.--domain— your public domain; the reverse proxy is configured for<service>.yourdomain.com. If omitted you'll be prompted. You don't need the domain's DNS-01 API token yet — that comes in Step 4.--size small|medium|large— how big the site is. It sets the cores, memory and disk every module is installed with, the firewall and the mothership included. Optional: medium when omitted.
| Size | Meant for |
|---|---|
small |
A few users. One machine, or up to three small ones. A test or lab system. |
medium |
The default. Up to about 50 people using it at once, many modules in one or two environments. |
large |
100+ users. Three or more large machines, possibly many environments. |
A module states its medium size and, where it scales, a smaller and a bigger one: a
small site gets the smaller, a large site the bigger. Each environment can have a size
of its own (Configure Environments). Change the site's later with
site-manager site modify --size <size> or on the console's Site page: installed modules
follow at their next update, and their disks only ever grow. On one machine with 32 GB of
memory, choose small: the foundation plus the home and office modules then fit.
On the first node this runs the whole foundation bring-up end-to-end as a five-phase chain — you run it once and watch:
- [1/5] Node — Proxmox post-install, the
lan/wanbridges (auto-detected: the install NIC with internet becomes WAN; the other, to your downstream switch, becomes LAN) and the Proxmox cluster named<site-code>. Data pools come later, see below. - [2/5] Firewall — downloads and boots the prebuilt OPNsense image at
10.0.0.1, self-configured with unique credentials (no GUI, no installer). - [3/5] Gateway cutover — adds this node's management IP
10.0.0.10and points its default route + DNS at the firewall. Additive and non-disruptive: the upstream IP is kept, no cables move, your session is not dropped. - [4/5] Sanity check — gateway, DNS, internet.
- [5/5] Platform — imports the prebuilt NixOS template and builds the
tappaas-cicdmothership (bootstrap.shclone+nixos-rebuild → reboot →install.sh). The cicd'sinstall.shthen writes the system's configuration, automatically:site.json— the site singleton (email, locations, repos), viacreate-site.sh --name <site-code> --organization <org>(the site code is the neutral cluster name; the org names the default env/zone)zones.json— the network zones, vianetwork-manager init- the
mgmt+ default<org>environments, viaenvironment-manager add(the minimal-set bootstrap) - the foundation modules (cluster, templates, network, tappaas-cicd) and the Caddy reverse proxy
Two names:
<site-code>= the Proxmox cluster name =site.json.name;<org>= the default environment name = your organisation name (site.json.defaultEnvironment). For a single-site install they are often the same — just pass--name. The organisation itself isn't created until Step 5 (rest-of-foundation.sh, after the identity provider is up).
The domain you passed configures the reverse proxy. To stop earlier, pass
--skip-firewall or --skip-platform; to drive the in-VM cicd install by hand,
see Appendix: install options.
When it finishes, tappaas1 is at 10.0.0.10 behind the firewall and
tappaas-cicd owns the platform. The rest of the install can continue in the
console (http://<tappaas-cicd>:9097/, unlocked with the setup token in
~/config/console-setup-token on the mothership). The bootstrap's end note says how to
reach it from where you are:
- the install LAN, when the firewall's WAN is a private network (an install at home):
http://<the firewall's WAN address>:9097/from that network only. It closes when the console module is installed behind the identity gate; - an ssh tunnel through the node, when the WAN is public (a hosted server):
ssh -N -L 9097:<tappaas-cicd>:9097 root@<node>, thenhttp://localhost:9097/.
No VPN peer is made for you: you enrol your own device in Day 1's admin VPN step, or move your computer onto the site's LAN. The VPN stays the way in from outside.
Optionally move your admin laptop onto the downstream switch (you'll get a
10.0.0.x lease; Proxmox at https://10.0.0.10:8006, firewall GUI at
https://10.0.0.1) — not required, since the node also keeps its upstream IP until
you harden it later.
A data pool on the first node. The firewall and the mothership start on the boot pool. To build a pool on the node's other disks and move them onto it, on the mothership:
site-manager node storage tappaas1 # the node's disks, by serial
site-manager node storage add tappaas1 tanka1 --disks <serial>,<serial> --layout mirror
module-manager module modify network --set storage=tanka1
module-manager module modify tappaas-cicd --set storage=tanka1
The disks move live, without a restart. A disk that carries data needs --wipe.
Step 3 — Add additional nodes (optional; skip for single-node)
Do this after the first node's bootstrap has finished (cicd is up).
Follow-on nodes install over the network, fully unattended — no USB stick,
no installer screens. The mothership runs a TTL-limited network-boot trap
(node-provisioner): a machine you have registered gets installed when it
PXE-boots on the management LAN; every other machine is refused.
Wire the new node's LAN NIC to the downstream switch (the 10.0.0.0/24
management network) and find out which disk is its boot disk — that disk
is wiped. (Managed inter-node switch? Configure its VLAN trunks first; see
the network section.)
The netboot assets are staged automatically by the first-node bootstrap
(prepare-netboot.sh, run at the end of the cicd install) — adding nodes is
a latent capability of every TAPPaaS system. After a PVE version upgrade,
re-stage once with site-manager node media prepare --force on the mothership.
Per node — one command:
site-manager node add tappaas2
The command walks the whole flow and asks only what it cannot know:
- Registers the node's install answer and opens a TTL-limited boot window (closes on its own; a cluster node is never answered).
- Tells you to boot the machine from the network (UEFI PXE, e.g. F11)
or from the USB stick image (below). The
installer picks the boot disk itself — the smallest SSD or NVMe disk, and
both of an identical smallest pair, mirrored (ZFS) — or the one
--boot-diskdescribes (a size such as2TB, a model or a serial). Only when nothing matches does the node's console ask, listing the machine's real disks; the chosen disk is WIPED. Everything else is hands-off: the installer fetches its per-node answer from the mothership and reboots into PVE. The generated root password lands under~/.node-secrets/(0600) — the mothership's ssh key is already authorized. - Asks the join-time questions with the node's real ports on screen: the WAN NIC (only if this node will host firewall HA) and the lan port(s).
- Runs the node step (served from the mothership — no GitHub needed),
moves the node to its standard mgmt IP (
tappaasN→10.0.0.<9+N>), joins the Proxmox cluster, brings it to the cluster baseline, reboots it into its new kernel, and registers it as itspvehostinstance. Its guests go to the boot pool until it has a data pool. - Data pools are a separate step, against the node's real disks:
site-manager node storage tappaas2lists them, andsite-manager node storage add tappaas2 tanka1 --disks <serial>,<serial> --layout mirrorbuilds one.
Every question can be pre-answered for a fully unattended run:
site-manager node add tappaas2 --boot-disk 2TB --mac aa:bb:cc:dd:ee:ff --no-wan
# --mac also pre-reserves the standard IP, so the node installs onto it directly;
# with neither --mac nor --switch-port, the first machine that asks gets the answer.
# --boot-disk: a size, a model or a serial (blank: the automatic choice).
site-manager node storage add tappaas2 tanka1 --disks 50026B73842AD86F --layout single
Afterwards, fold the new topology into HA + replication:
site-manager update
Machine can't PXE-boot? Write the installer stick image to a stick —
dd if=/var/lib/tappaas-pxe/tappaas-usb.iso of=/dev/<usb> bs=4M(it is also served athttp://tappaas-cicd.mgmt.internal:8090/tappaas-usb.isowhile the boot window is open) — and boot from USB instead of pressing F11. The flow (and thenode add <name>command) is otherwise identical: the answer still comes from the mothership, which matches--boot-diskagainst the disks the stick reports. The stick cannot ask on the console: a boot-disk field that matches no disk stops the install with the list of the machine's disks.Manual install instead? Install from a stock ISO or the Step 1 Option A stick, authorize the cicd key for root, then run
site-manager node add tappaas2 --adopt <its IPv4>— it finds the Proxmox there and runs the same join + capture pipeline, which moves it to the node's standard mgmt IP. The console's Add node wizard always installs. The underlying tools remain available for surgery:node-provisioner register/enable/disable/statusanddhcp-manager pxe/host.
Step 4 — Set up certificates for the default domain (optional)
This step is optional — it provisions the HTTPS certificate that lets your
TAPPaaS services be reached over TLS on <service>.yourdomain.com. Skip it if:
- you don't want any external access to the TAPPaaS system — internal / LAN-only
use is fully supported; every service stays reachable on the LAN, only the
public HTTPS endpoint of
dns01modules lacks a certificate; or - you want to set it up later — it can be run any time after the foundation is up.
Your domain is already configured (you passed --domain in Step 2). Pick one of the
two certificate strategies below.
(The public domain lives per-environment. To change the default environment's domain later:
environment-manager modify <system-name> --domain <yourdomain>, then set up certificates.)
Option A — one wildcard certificate via DNS-01 (recommended)
The default TLS strategy (proxyTls: dns01) issues one wildcard certificate per
TAPPaaS domain via ACME DNS-01, then binds it to every module's reverse-proxy
entry through Caddy's CustomCertificate. DNS-01 needs no
inbound :80 traffic, so internal-only services get a public cert too.
Why not Caddy's own DNS-01? Since os-caddy 2.0.0 the OPNsense build only ships the Cloudflare DNS provider. TAPPaaS uses os-acme-client (which wraps acme.sh) so you can pick any of the 120 DNS APIs it supports — Cloudflare, deSEC, Hetzner, OVH, Route 53, Namecheap, etc.
Run the operator-driven setup on the mothership:
ssh tappaas@tappaas-cicd # or its 10.0.0.x address
acme-setup.sh # interactive: picks up domain/email, prompts for token
The script:
- Reads the domain from the default environment file
(
config/environments/<system-name>.json,domains.primary) and the admin email fromsite.json(.email). - Prompts for your DNS provider (default
cloudflare) and its API token (offers to save them to~/.acme-dns-credentials.txt, mode 600, for re-runs). - Provisions an ACME account, a DNS-01 validation, a
caddy-reloadautomation action, and a wildcard certificate (*.<domain>+ bare apex) on the firewall — then signs it and waits for issuance (~10–30 s). - Stores the issued cert's OPNsense Trust refid in the runtime
config/cert-refids.json, keyed by environment name (ADR-007c). Every laterproxyTls: dns01module install reads this refid and binds the wildcard viaCustomCertificateautomatically.
Using Cloudflare? Its DNS-01 token needs specific scopes — see Appendix: Cloudflare API token scope.
# Non-Cloudflare? Pass the provider name; the script asks for that provider's fields:
acme-setup.sh --provider hetzner # deSEC, hetzner, ovh, route53, ...
# Or seed creds non-interactively (chmod 600):
cat >~/.acme-dns-credentials.txt <<EOF
provider=cloudflare
dns_cf_token=YOUR-TOKEN-HERE
EOF
chmod 600 ~/.acme-dns-credentials.txt
acme-setup.sh
To test before going live, add --staging once (Let's Encrypt staging =
untrusted certs, no rate limits); then re-run without --staging for the
trusted prod cert (the script swaps the cert in place — Caddy picks up the new
one automatically via the registered caddy-reload action).
Option B — per-service certificates via HTTP-01 (no DNS API needed)
Choose this if you don't have ACME / DNS-01 API access to your DNS provider — i.e. you can't (or don't want to) issue an API token that lets TAPPaaS edit DNS records, which Option A's DNS-01 requires. HTTP-01 validates each service over inbound port 80 instead of via DNS, so it needs no DNS API token — but each publicly-reachable service gets its own per-domain certificate (no shared wildcard), and that service must be reachable from the internet on :80 for issuance and renewal.
There is no wildcard to provision, so you don't run acme-setup.sh. Instead set
the TLS mode to http01 on each module you want publicly reachable, in its
<module>.json:
{ "proxyTls": "http01" }
Caddy then obtains and renews a per-domain certificate for that service via the ACME
HTTP-01 :80 challenge. The two strategies coexist per module, so you can keep
the wildcard (Option A) for most services and use HTTP-01 for a specific public one.
Step 5 — Complete the foundation
From here on you work on the cicd mothership. Your laptop's key is not on it yet, and its name resolves only behind the firewall, so go through the node's Shell (Datacenter → tappaas1 → Shell):
ssh -i /root/tappaas/tappaas-cicd.key tappaas@tappaas-cicd.mgmt.internal
One command does two things:
rest-of-foundation.sh
- Installs the remaining foundation modules in order — backup → identity → logging — then runs a final system update + tests.
- Bootstraps your people domain — once the identity provider (Authentik) is
up and
config/identities/is still empty (first install), it creates the organisation<org>(the--organizationfrom Step 2), theusersgroup and your installer user (fromsite.json's email), and pushes them into Authentik. So this is where your organisation is actually created — the earlier--organizationonly reserved the name; the org entity is materialised here.
It's idempotent — safe to re-run if a module needs attention (the people bootstrap
is skipped once config/identities/ exists, so it never disturbs people you've added).
When it finishes you'll see a "🎉 your TAPPaaS foundation is installed" summary
(nodes, firewall, mothership, domain/TLS, modules, organisation).
Prefer to do it by hand?
module-manager module addreads./<module>.jsonfrom the current directory, socdin first:bash cd ~/TAPPaaS/src/foundation/backup && module-manager module add backup # Proxmox Backup Server cd ~/TAPPaaS/src/foundation/identity && module-manager module add identity # Identity provider cd ~/TAPPaaS/src/foundation/logging && module-manager module add logging # Loki / Grafana / Alloy(Module sizing/zones are defaults — see appendix.)
Step 6 — Managed switches and Wi-Fi APs (optional)
Skip this entirely if you have unmanaged switches and no TAPPaaS-driven APs — an unmanaged switch passes tagged frames transparently, so nothing to configure (see Network — cutting over to the firewall for the inter-node trunk story). Do this step if you have managed switches or Wi-Fi access points whose per-zone VLANs/SSIDs you want TAPPaaS to keep in sync.
TAPPaaS treats switches and APs as two more reconcile providers alongside OPNsense
and Proxmox — all driven from zones.json (the VLAN tags and the per-zone SSID
field). Unlike the firewall/Proxmox planes, these are not applied automatically
by rest-of-foundation.sh or update-tappaas: they need an inventory you register
once, and (for hardware without a vendor plugin) a manual "tag these VLANs" step. Run
everything below from the mothership (ssh tappaas@tappaas-cicd).
6a — Switches
Fastest path is the interactive bootstrap, which registers each brand/switch/uplink port and then applies:
setup-switches.sh # interactive; ends by running `switch-manager reconcile --apply`
Or register by hand with the switch-manager CLI, then reconcile:
# A manual switch with two node-uplink (trunk) ports:
switch-manager add-switch core --vendor tplink --managed manual
switch-manager add-port core 9 --type node --target tappaas1 --target-port eth0
switch-manager add-port core 10 --type node --target tappaas2 --target-port eth0
switch-manager reconcile --apply # manual switch → prints the VLANs to tag (and records them)
# …tag those VLANs on ports 9 & 10 in the switch UI…
--managed manual→ TAPPaaS prints the VLANs to tag; you apply them on the device (no separateconfirmneeded —reconcile --applyalready records intent).--managed auto→ a vendor plugin (currently UniFi OS) programs the switch over its API onreconcile --apply. Auto/controller brands read~/.unifi-os-credentials.txt(url/username/password, mode 0600).- Check state any time:
switch-manager list-ports(per-port config + drift vszones.json), ornetwork-manager reconcile --only switchfor a dry-run.
6b — Wi-Fi access points
A WiFi network is one site-wide SSID bridged to one zone and broadcast by every access
point. Register the AP(s), declare each SSID on its zone (this replaces the zone's
<…_SSID> placeholder), give each WPA SSID its passphrase (stdin or a file — it goes to a
0600 file, never into a config), and reconcile:
ap-manager add livingroom --vendor unifi --ip 10.0.0.21
ap-manager link livingroom --switch core --port 5 # the AP's uplink port
ap-manager ssid add HomeWiFi --zone home --security wpa2-personal
ap-manager ssid add GuestWiFi --zone guest --security wpa2-personal
ap-manager ssid set-passphrase HomeWiFi # asked twice, hidden
ap-manager ssid set-passphrase GuestWiFi
ap-manager reconcile --apply
setup-wlan-secrets.sh walks the zones still on a placeholder and asks the same
questions; --list shows each zone's SSID and whether its passphrase is set. A changed
passphrase (ssid set-passphrase again) reaches every access point at the next
reconcile --apply.
- With a UniFi controller,
switch-manager interrogateauto-populates the AP inventory and marks the AP's uplink port as a WiFi-VLAN trunk — so adopting a controller wires most of this up for you. security:open | wpa2-personal | wpa3-personal | wpa2-enterprise | wpa3-enterprise(enterprise needs a RADIUS profile — currently a manual step).
Re-running
All of the above is idempotent. After the initial setup, a full
network-manager reconcile --apply re-converges every plane (OPNsense, Proxmox,
switch, ap) against zones.json; add --only switch / --only ap to scope it.
Manual switches always report as "needs-manual" on apply (they can't self-program)
— that's expected, not an error. Full reference:
network/scripts/README.md (ADR-008).
Next steps
Do this one first: take your backup encryption key off the machine.
Put a USB stick into tappaas1 and run, on the mothership:
backup-manager key export --usb
It writes every backup key to the stick, encrypted with a password you choose, checks it by reading it back, and ejects it. Make two — run it once per stick — and keep them at two different places, both off-site, with the password elsewhere (RESTORE.md §0). Rewriting a stick later replaces only its keys and keeps anything else you store on it, such as a release signer's emergency key.
Backups are encrypted client-side, and the keys are escrowed on the mothership — which is inside the system a full rebuild would be recreating. Without a copy that lives somewhere else, a site-loss restore has nothing to decrypt with, and no amount of surviving backup data helps (ADR-012 §2.5.1). Store the media off-site. Everything else on this list can wait; this cannot, because the day you need it is the day you cannot make it.
If your first node had no tankc pool, the backup module installed as a shim
— dependencies are satisfied but nothing is being backed up yet. Add the pool and
run module-manager module update backup; it promotes in place. Check with
backup-manager placement.
Check that alert mails reach you. On the mothership:
site-manager support test
Or in the console: Site → Support → Test alert mail.
It sends a test ALERT to the site's email, with a support bundle attached, and then its
RESOLVED, which arrives as a reply in the same thread. If neither arrives, check the
spam folder and the address (site-manager site show), then the mail log of the first
node (journalctl -t postfix/smtp there). Run it again any time, for instance after
changing the address.
Later, as a day-two step: a support address. An alert that carries a support bundle says to forward the mail to the site's support address. It is empty after the install — the mail then says to forward it to your support contact. Set it when you have one:
site-manager site modify --set supportEmail=<address>
(or on the console's Site page). Which address that is will come with the TAPPaaS support plans; until then, leave it empty or use your own support contact.
The rest of the foundation is complete. Everything from here is covered on the documentation site: configure environments, add a satellite, add workloads (each stack's install guide), and day-to-day operation — start at the install overview.
Network — cutting over to the firewall
Putting the firewall (a VM on tappaas1, at 10.0.0.1) inline as the gateway is
done for you by the bootstrap (Step 2, [3/5]) — config-network.sh
--swap-gateway. You normally never run it by hand; this section explains what it
does. Each node has two NICs, wired at install time and left in place:
- WAN — the NIC connected to your upstream router (the one with internet at install time). It's the firewall VM's uplink. On the first node the node also keeps its original install IP here.
- LAN — the NIC connected to your downstream switch (the
10.0.0.0/24management LAN). VLAN-aware bridge; holds the node's mgmt IP10.0.0.10.
The cutover is additive — nothing is removed, nothing is unplugged:
--swap-gatewaymakes sure10.0.0.10is on the LAN side, points the node's default route + DNS at the firewall (10.0.0.1), and restarts corosync so it binds to the mgmt IP. The upstream IP is kept, so the node (and your existing session) never lose connectivity.- Because both IPs remain, you can keep managing the node where you are, or
move your admin laptop onto the downstream switch (you'll get a
10.0.0.xlease; firewall GUI athttps://10.0.0.1). Moving is optional until you harden.
internet / upstream router
│
[tappaas1 WAN NIC] ─────────────► OPNsense WAN (DHCP from upstream)
│ node also keeps its install IP here
┌───────────────────────────────────────────────┐
│ tappaas1 (Proxmox) │
│ OPNsense VM: lan → 10.0.0.1 (the gateway) │
│ node mgmt: lan → 10.0.0.10 │ ← default route now via 10.0.0.1
└──[tappaas1 LAN NIC]──────────────────────────────┘
│
[ downstream switch ] ── admin laptop (optional; 10.0.0.x by DHCP)
Later hardening: once you're managing via the mgmt net (or netbird), run
config-network.sh --drop-upstream on a node to remove its upstream IP, so
Proxmox is reachable only behind the firewall.
The switch (3-node clusters): the lan bridge is VLAN-aware — it carries
the management network untagged (10.0.0.0/24) plus every TAPPaaS VLAN
tagged as a trunk. For VMs on different nodes to reach each other on a VLAN, the
switch between the nodes must pass those tagged frames. How that works depends
on the switch type:
- Unmanaged switch — passes all frames transparently, tagged or untagged, so the TAPPaaS VLANs cross between nodes out of the box, with no configuration. This is the simplest choice for a 3-node cluster.
- Managed switch — by default forwards only untagged traffic; it drops or ignores VLAN-tagged frames until you configure (manage) it. You must set the inter-node ports as trunks carrying the TAPPaaS VLAN tags (plus the untagged management network) before adding nodes — otherwise cross-node VLAN traffic won't pass even though everything else looks fine.
(Single node: no switch needed for VLANs — all VLANs live inside the one node;
the lan NIC can go to your existing LAN or any switch.)
Upstream options: the firewall WAN can sit behind your ISP router (port-forward), in bridge mode, or replace the ISP router directly.
Appendix: Cloudflare API token scope
(Extra detail — only relevant if you chose Cloudflare* as the DNS provider for the wildcard certificate in Step 4, Option A.)*
Create a custom token at https://dash.cloudflare.com/profile/api-tokens with
permissions Zone → Zone → Read and Zone → DNS → Edit, restricted to your domain.
Optionally allow only the firewall WAN IPv4 and IPv6 (or skip the IP filter — the
token is already zone-scoped).
Appendix: install options
Defaults are chosen so the commands above "just work". Override as needed:
| Default | How to change |
|---|---|
Branch stable |
Pass a different branch as the 2nd arg to install.sh (e.g. main). |
Hostnames tappaas1/2/3 |
Set during the Proxmox install; the first node must be tappaas1 (it creates the cluster). |
Management subnet 10.0.0.0/24, gateway/firewall 10.0.0.1 |
config-network.sh --mgmt-ip <CIDR> --gateway <ip>; firewall LAN lives in src/foundation/network/firewall-config.xml.template. |
| Site code (cluster name) | install.sh --name <site-code> (Proxmox cluster + site.json .name; lowercase, ≤15 chars; prompted if omitted). |
| Organisation (default environment) | install.sh --organization <org> (default environment/zone + organisation; lowercase; defaults to the site code, no prompt). |
| Auto cluster create/join | install.sh --cluster / --join / --no-cluster. |
| Chained first-node install (node→firewall→cutover→sanity→platform) | The entry point foundation/install.sh runs all 5 phases on the first node by default; stop earlier with --skip-firewall or --skip-platform. The node step alone is cluster/install.sh. |
| Gateway cutover (route via firewall) | Done automatically by the bootstrap; manual: config-network.sh --swap-gateway (additive — keeps the upstream IP). |
| Take Proxmox off the upstream net (hardening) | config-network.sh --drop-upstream (run later, once you manage via the mgmt net / netbird). |
| Platform install branch / domain | install-platform.sh --branch <name> --domain <domain>. |
| Automated vs. manual cicd install | install-platform.sh automates the in-VM install over SSH; pass --manual-cicd to run bootstrap.sh then install.sh by hand inside the cicd VM instead. |
ZFS pools (tankXY, topology) |
install.sh --pool name=topology:disks (none by default), or site-manager node storage add later. |
| Firewall root password | config-firewall.sh --root-pw <pw> (otherwise generated, in /root/.opnsense-root-password on the node; the API key is always unique per deploy). |
| VM sizing, storage, network zone per module | edit the module's <name>.json (cores, memory, diskSize, storage, zone0/bridge0). |
| Domain / TLS | provided to the firewall/app modules; TLS issuance is DNS-01 by default. |
| Unattended runs | most scripts accept --non-interactive (supply the values via flags). |
Field definitions for module JSON are in src/foundation/schemas/module-fields.json.
Network zones/VLANs come from the git template
src/foundation/tappaas-cicd/manager/network-manager/zones.json, which
network-manager init transforms into the live, per-installation
config/zones.json at install time.
Appendix: Installing via SSH
The main install guide recommends the Proxmox console, but SSH works fine with
proper preparation. The key is using tmux so the install continues if your
connection drops during network reconfiguration.
1. Generate and install an SSH key (from your workstation)
If you don't already have an SSH key pair:
# Generate a new key (accept defaults, optionally set a passphrase)
ssh-keygen -t ed25519 -C "your-email@example.com"
Reinstalling? If you previously installed TAPPaaS on the same IP, SSH will
refuse to connect due to a changed host key (WARNING: REMOTE HOST IDENTIFICATION
HAS CHANGED!). Remove the old entry first:
ssh-keygen -R <proxmox-ip>
# Copy the public key to the Proxmox node (use the node's current IP)
ssh-copy-id root@<proxmox-ip>
Now you can SSH without a password:
ssh root@<proxmox-ip>
Do this step — it is how you will reach the nodes from now on. Once the
install has finished, every node is key-only: no password logins over SSH,
for any user (#19). Root keeps key login, which Proxmox itself and the
mothership depend on. The PVE web GUI (its node Shell included) and the
physical console do not use SSH and keep working with the root password —
that is the way in when a key is missing. From that node Shell you can go on
to the mothership, and from the mothership to every VM: the nodes present at
install carry the mothership's key for exactly this, as
/root/tappaas/tappaas-cicd.key:
ssh -i /root/tappaas/tappaas-cicd.key tappaas@tappaas-cicd.mgmt.internal
To add a key later (another laptop, a colleague), ssh-copy-id no longer works,
because it needs a password. Open Datacenter →
echo 'ssh-ed25519 AAAA… you@laptop' >> /etc/pve/priv/authorized_keys
2. Fix locale warnings
Fresh Proxmox installs may show Perl locale warnings. Fix them:
# Quick fix for current session
export LC_ALL=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
# Permanent fix (then start a new shell)
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && locale-gen && \
update-locale LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8
3. (Optional) Use tmux for session persistence
tmux keeps your session alive if SSH disconnects during network
reconfiguration. This is optional but recommended:
apt update && apt install -y tmux
tmux new -s install
If your SSH connection drops, reconnect and reattach:
ssh root@<proxmox-ip>
tmux attach -t install
Useful tmux commands:
Ctrl+b d— Detach (leave session running in background)Ctrl+b [— Scroll mode (arrow keys to scroll,qto exit)
4. Run the install
Run the bootstrap command from Step 2:
REPO="https://codeberg.org/TAPPaaS/TAPPaaS/raw/branch/"; BRANCH="main"
curl -fsSL ${REPO}${BRANCH}/src/foundation/install.sh >install.sh
chmod +x install.sh && ./install.sh "$REPO" "$BRANCH" --name <site-code> --organization <org> --domain "yourdomain.com"
Even if your SSH drops during network cutover, the install continues inside
tmux. Reconnect with tmux attach -t tappaas-install to see progress.
5. After install
Once complete, you can access:
- Proxmox UI:
https://10.0.0.10:8006(or the node's upstream IP) - Firewall GUI:
https://10.0.0.1 - CICD Mothership:
ssh tappaas@tappaas-cicd.mgmt.internal, from behind the firewall