Supervision & Monitoring

Source: /opt/supervision/supervise.py — a 7-layer, 27-checkpoint self-healing monitor.
Runs every 5 min (vps-supervise.timer); daily digest at 08:00 SGT
(vps-supervise-digest.timer).

The 7 layers

Layer	Scope	Checks
L1 host	disk %, RAM, swap (2 GB), load	+ `storage_5gb` (any of `/opt/sportstlm`, `/opt/racing`, `/opt/lottery`, `/opt/backups`, `/var/lib/docker`, `/var/log` over 5 GB)
L2 runtime	docker daemon, caddy + docker active & boot-enabled, ssh.socket, Caddyfile validates, NTP
L3 container	each site running with a restart policy
L4 endpoint	local ports, public HTTPS, `latency_p95` (opt-in, 2000 ms)
L5 tls	live cert expiry per domain
L6 logs	docker + caddy log ceilings
L7 backup	`inrun_backup` fails if newest archive > 108 h old (catches a silently dead Mon/Fri run)

Self-heal rules

Fires only after 2 consecutive failures.
Dedupes actions; re-checks before reporting.
An invalid Caddyfile is never auto-healed.

Alerting

Telegram @sportstlm_supervisor_bot → chat 8774908595 (WT).
Transition-only: OK→FAIL and FAIL→OK, plus the daily digest (a known failure
doesn't repeat every 5 min).
Token is encrypted at rest via systemd-creds (/etc/ops/tg-token.cred,
host-key bound). The 5 alerting units load it with LoadCredentialEncrypted=tg-token:…
and read $CREDENTIALS_DIRECTORY/tg-token. Manual shell runs fall back to
/opt/supervision/supervision.env.

Key files

Path	Purpose
`/opt/supervision/supervise.py`	the monitor (single script)
`/opt/supervision/README.md`	per-checkpoint detail
`/opt/supervision/targets.json`	site target list (domain/port/container)
`/opt/supervision/state.json`	current probe state
`/opt/supervision/alerts.json`	alert state
`/opt/supervision/supervision.env` (0600)	bot token fallback
`/opt/supervision/backup.sh` / `rollback.sh`	Caddy/supervision snapshot + rollback

Change discipline

/opt/supervision/backup.sh <label>       # snapshot caddyfile, script, env, units, logrotate, sysctl
/opt/supervision/rollback.sh --list
/opt/supervision/rollback.sh latest      # restore, validate caddy, reload, re-probe sites

Adopted "foreign" monitors (still separate, not in supervise.py)

vps-leak-scan — credential leak scan.
vps-module-monitor — module freshness (incl. Sports Mole previews.created_at).
vps-inrun-signal — in-run signal monitor.

These three read /opt/supervision/supervision.env at runtime (token no longer
hardcoded), are chmod 700.