TAPPaaS

Appendix F — Environments

An environment is a separate world on the same platform: its own network zone, its own domain, its own copy of each module installed into it, and a firewall between it and the others. Use one for test next to production, for guests next to employees, for a client, or to keep a home business apart from the household. Decide which environments you need before installing the application stacks: a module is installed into one environment.

From the console

On the Day 2 page, Add an environment:

  1. The environment's record: its name, display name, owning organisation, domain (e.g. client1.example.org), its size (the site's when empty), how its names get certificates (per service, or one wildcard), and create its zone, a new VLAN that inherits the service zone's reachability.
  2. Make it live: the preview lists what the reconcile does (the zone on the firewall, the nodes and the switches, DNS, the certificate); apply does it.

Then install stacks into it: Add a stack asks which environment. The environment's page on the console (Environments) shows its zone and modules, and modifies, reconciles or deletes it. Public DNS records for its names are added at your DNS provider (step 2 below).

The rest of this appendix walks the same through on the command line, with one worked example.

The always-present environments

Every TAPPaaS system ships two environments, created at install time by the environment-manager add minimal-set bootstrap:

This guide walks one concrete additional environment end to end.

Scenario

You want to host Nextcloud and EURO Office for a client, isolated from your own services and from other clients, reachable at the client's domain client1.<your-tappaas-domain> (written client1.tappaas.org below — replace tappaas.org with your configured domain).

You will create an environment client1 in a dedicated client1 zone, and install both modules into it. euro-office depends on nextcloud:vm, so the environment install of euro-office automatically wires to the environment's Nextcloud (nextcloud-client1), not your default one.

Result (the effective module name gains a -<env> suffix for non-default environments):

Module VM name Zone Public URL
nextcloud nextcloud-client1 client1 nextcloud.client1.tappaas.org
euro-office euro-office-client1 client1 euro-office.client1.tappaas.org

Prerequisites


Step 1 — Create the environment and its dedicated zone

Pick one DNS/cert mode now (see the Appendix: Which certificate should I pick?):

First author the dedicated zone (network-manager owns zones), then create the environment that references it:

# A dedicated zone for the tenant, inheriting srv's reachability profile.
network-manager zone add client1 --from-zone srv

# Create the environment file referencing that zone.
environment-manager add client1 \
    --display "Client One" \
    --owner client1-company \
    --domain client1.tappaas.org \
    --zone client1 \
    --dns-mode wildcard

--dns-mode selects domains.dnsMode: per-service (default — Caddy issues a per-host HTTP-01 cert) or wildcard (one *.<primary> ACME cert for the whole environment, this guide's scenario). Change it later with environment-manager modify client1 --dns-mode <mode>.

network-manager zone add authors the zone in zones.json and reconciles all four planes (OPNsense interface + DHCP + rules, the Proxmox nodes, the switch, the WiFi APs) so the new VLAN actually reaches everything. --from-zone srv makes the new client1 zone inherit srv's type, bridge, access-to and pinhole-allowed-from (so it can reach the internet and be reached by the DMZ reverse proxy). VLAN is auto-allocated in the type band unless you pass --vlan.

What this changes

ownerOrg must reference an existing People Organization. Use environment-manager list / environment-manager show client1 to inspect.

How big the environment is. An environment takes the site's size unless you give it one: --size small|medium|large on add, or later environment-manager modify client1 --size large (also on the console's Environments page). The modules installed into it get the cores, memory and disk of that size: the module's smaller values at small, its default at medium, its bigger values at large. A test environment on a medium site can be small; a client with a hundred users can be large. Installed modules follow a change at their next update; disks only grow.

Size Meant for
small A few users. One machine, or up to three small ones. A test or lab system.
medium The default. Up to about 50 people using it at once, many modules in one or two environments.
large 100+ users. Three or more large machines, possibly many environments.

Step 2 — Public DNS

Internal split-horizon DNS (so cluster clients reach Caddy on an internal firewall interface rather than hair-pinning out the WAN) is handled automatically in Step 3/4 — you only need to set up the public records here. Point them at your firewall's WAN IP.

Option A — wildcard record (simplest; works with either cert mode):

*.client1.tappaas.org   A   <your-WAN-IP>

Option B — dedicated per-service records:

nextcloud.client1.tappaas.org     A   <your-WAN-IP>
euro-office.client1.tappaas.org   A   <your-WAN-IP>

For the wildcard cert in Step 3 you also need a DNS provider with an ACME API (e.g. Cloudflare) so acme-setup.sh can solve the DNS-01 challenge. The per-service cert path needs the names public on :80 for HTTP-01 instead.

What this changes: nothing in TAPPaaS — these records live at your DNS provider.


Step 3 — TLS certificate (pick the mode you chose in Step 1)

Not sure which? See the Appendix: Which certificate should I pick?.

Option A — Wildcard certificate (dnsMode=wildcard)

One *.client1.tappaas.org certificate, shared by every service in the environment, issued via DNS-01:

acme-setup.sh --environment client1        # never use --staging

This issues the wildcard cert on the firewall, and also registers the split-horizon override *.client1.tappaas.org → <environment service-zone gateway> in Unbound so internal clients resolve the environment's services to the reverse proxy. (A wildcard is a single Unbound "redirect" zone with one target, so — unlike per-service, Option B — it cannot be self-traffic for every client zone at once; as an interim it points at the environment's own service-zone gateway rather than the DMZ gateway, and a fully subnet-aware answer awaits Unbound access-control-view. It falls back to the DMZ gateway if the environment declares no resolvable zone.)

What this changes: the wildcard cert lands in OPNsense Trust and an Unbound host override for client1.tappaas.org → <environment service-zone gateway> is created. The cert's OPNsense refid is reconciler-populated runtime state owned by the network/cert layer — it is not written into environment.json.

Option B — Per-service certificates (dnsMode=per-service, the default)

Do nothing here. With dnsMode=per-service, each module's firewall:proxy install (Step 4) makes Caddy issue that domain's own certificate via Let's Encrypt HTTP-01, and registers a per-service Unbound split-horizon override pointing at the service's authorized client-zone gateway — the firewall interface on that zone's own subnet, so the request is self-traffic and crosses no inter-zone rule (nextcloud.client1.tappaas.org → <client-zone gateway>; ADR-005 §6). It is deliberately not the DMZ gateway, which client zones such as home/work cannot reach. No acme-setup.sh, no DNS API — but the names must be publicly reachable on :80.


Step 4 — Install the two modules into the environment

These Community modules live under ~/Community. module-manager module add reads the module JSON from the current directory, so cd into each module first.

# Nextcloud first (euro-office depends on it)
cd ~/Community/src/AndreasJe/nextcloud-hub/nextcloud
module-manager module add nextcloud --environment client1

# Then EURO Office — its nextcloud:vm dependency resolves to the ENVIRONMENT one
cd ~/Community/src/AndreasJe/nextcloud-hub/euro-office
module-manager module add euro-office --environment client1

--variant is a deprecated alias for --environment (kept only until the single production site is cut over). New work should use --environment.

module-manager module add validates the environment exists, then for each module:

What this changes


Step 5 — Verify

environment-manager show client1      # domain, zone, dnsMode, owner
environment-manager list              # all environments at a glance

# External (from anywhere): the client's public URL serves the app over HTTPS
curl -fsSI https://nextcloud.client1.tappaas.org/ | head -1

# Internal split-horizon: clients resolve to an internal firewall interface, not
# the WAN — per-service resolves to each service's client-zone gateway; a wildcard
# environment resolves to the environment's service-zone gateway.
getent hosts nextcloud.client1.tappaas.org      # -> an internal gateway IP, not the WAN

Both nextcloud-client1 and euro-office-client1 VMs run in the client1 zone, isolated from your default services and from any other client environment.


Removing the environment (clean teardown)

# Delete the deployed modules first (consumer before provider)
module-manager module delete euro-office-client1
module-manager module delete nextcloud-client1

# Remove the environment file
environment-manager delete client1

# network-manager owns the zone lifecycle: deleting the zone disables it on
# OPNsense (tears down its VLAN/DHCP/rules across all planes) and drops the key.
network-manager zone delete client1

module-manager module delete removes the VMs, their Caddy entries and (per-service) DNS overrides. environment-manager delete removes config/environments/client1.json (it refuses while modules are still deployed unless you force it). network-manager zone delete deactivates the zone across every plane and then removes the key from the runtime zones.json — the git-tracked source template is never touched.


Appendix: Which certificate should I pick?

First, two terms that sound alike but are different:

They are independent — you can use a wildcard DNS record with per-service certificates, or per-service DNS records with… per-service certificates. The choice below is about the certificate strategy (Step 3A vs 3B).

Wildcard Cert (3A) per-Service Cert (3B)
DNS records (Step 2) wildcard DNS A record wildcard or per-service DNS A records
DNS provider API required (DNS-01) not required
Inbound :80 reachable not required required (HTTP-01)
Certs one shared *.client1.tappaas.org one per service
Best when you have services that need public DNS names but are not reachable from the internet the client's domain has no ACME DNS API

The environment file (config/environments/client1.json) and zone (zones.json) are identical in both cases — only the cert/DNS mechanics differ (domains.dnsMode).


Provider notes & troubleshooting

DNS provider — deSEC

acme-setup.sh --environment client1 --provider desec   # reads the token from ~/.acme-dns-credentials.txt

Cert fails with statusCode 400

The controller only surfaces the status code, and /var/log/acme.sh.log stays empty — the real error is elsewhere:

  1. os-acme-client logs to syslog tag AcmeClient (not acme.sh.log).
  2. For the actual Let's Encrypt / DNS error, run acme.sh directly with --debug 2 using the exact args from the syslog AcmeClient: The shell command ... line. It prints e.g. "Cannot find DNS API hook", "No TXT record found" (dns_sleep), or the LE problem document.

Guest in the environment zone gets no IP

A new environment VLAN N must be carried by every L2 layer, not just OPNsense L3. network-manager zone add / reconcile drives all four planes (OPNsense, Proxmox, switch, AP), but if a guest's DHCP DISCOVER never reaches OPNsense (check netstat -I vlan0.N Ipkts on the firewall) the VLAN is being dropped at an L2 layer — most often the firewall VM's Proxmox net trunks= (verify with qm config <firewall-vmid> | grep ^net) or a switch/AP trunk profile. Re-run network-manager reconcile --apply to converge the planes.