TAPPaaS

logging

Primary audience: TAPPaaS admin.

Centralized log aggregation for the whole TAPPaaS system — a Loki + Grafana + Alloy VM that collects logs from every TAPPaaS VM, the Proxmox nodes and the OPNsense firewall.

What you get

Capability Access from How
Grafana UI (dashboards, LogQL Explore) mgmt zone (via Caddy) https://logging.<tappaas.domain>
Loki log store (30-day retention) mgmt zone logging.mgmt.internal:3100, admin credential (~tappaas/.loki-admin-password, user tappaas)
Journal shipping from TAPPaaS VMs any zone, per VM logging:logs — the baseline's Alloy client pushing to the push-only receiver :3101
OPNsense firewall logs firewall syslog RFC 5424 → tcp/1514 (source=opnsense)
Proxmox node logs nodes rsyslog forwarder → tcp/1515 (source=proxmox)
CLI queries on the VM mgmt admins (SSH) logcli installed on the VM
Performance dashboards (ADR-046) Grafana, linked from the console's Performance pages one per class (node, guest, storage pool, backup datastore, published service) and per declaring module, each for the CI in its variable, with the CMDB's changes on that CI (updates, modifies, reboots, migrations, level changes) as annotations, written by the mothership every 5 minutes
Alert mails from the logs site owner (site.json email) Grafana rules over every host's logs — quorum lost, fencing, a silent host, a stuck HA service, a lost network leg, failed replication or backup, an out-of-memory kill, disk errors, a changed root key or firewall certificate — mailed by the mothership's tappaas-alert-receiver as ALERT: …. One that clears is mailed again as RESOLVED: …, a reply to the alert. A node down is one mail, not one per silent host: it names the guests down with the node and the HA-protected ones moved to another node, and one RESOLVED follows when the node is back. The ones about the cluster, backups and memory carry a support bundle (the cluster's state and the anonymized configuration) to forward to site.json supportEmail. A watchdog on the mothership mails when the alerting itself stops

What is not included

Requirements

Dependencies

Depends on Purpose
cluster:vm Creates the VM (clone of the NixOS template)
templates:nixos NixOS base image + OS configuration hooks
backup:vm PBS snapshot backup of the VM (Loki state included)
network:proxy Caddy entry logging.<domain> → Grafana :3000, mgmt-only

Availability

Logging is not an HA module by default. When its node is down, the senders keep what they can and deliver it when it is back: NixOS guests from their journal, the firewall and the Proxmox nodes from their syslog buffers (in memory, so a long outage loses the oldest lines). Alert rules evaluated by Loki do not fire while it is down.

On a site with two or more nodes an operator may make it HA, like any module: add cluster:ha to its dependsOn with an HANode (see cluster:ha); its disk then replicates to that node.

For installation steps see INSTALL.md.

Design and implementation detail: DESIGN.md. Test coverage: TEST.md.