VPS Runbook (authoritative import)

Verbatim import of /opt/RUNBOOK.md (symlink → /opt/supervision/RUNBOOK.md),
captured 2026-11-01. This is the on-box operations index — the single source of
truth. Structured notes elsewhere in this vault are derived from it.

srv2047208 — operations index

One page for everything running on this VPS (sportstlm.com, lottery.sportstlm.com,
racing.sportstlm.com). Deeper detail lives in the per-system READMEs referenced below.

What is running

Timers: vps-supervise.timer, vps-supervise-digest.timer, inrun-db-backup.timer.
Check with systemctl list-timers 'vps-*' 'inrun*' --no-pager.

Alerting

Telegram @sportstlm_supervisor_bot → chat 8774908595 (WT). Credentials in
/opt/supervision/supervision.env (0600). Transition-only: OK→FAIL and FAIL→OK,
plus the daily digest — a known failure does not repeat every 5 minutes.

Supervision checkpoints

• L1 host — disk %, available RAM, swap (2 GB), load, storage_5gb (any of
  /opt/sportstlm, /opt/racing, /opt/lottery, /opt/backups, /var/lib/docker,
  /var/log over 5 GB alerts)
• L2 runtime — docker daemon, caddy + docker active and boot-enabled, ssh.socket,
  Caddyfile validates, NTP
• L3 container — each site running with a restart policy
• L4 endpoint — local ports, public HTTPS, latency_p95 (opt-in, 2000 ms)
• L5 tls — live cert expiry per domain
• L6 logs — docker + caddy log ceilings
• L7 backup — inrun_backup fails if the newest archive is over 108 h old
  (covers a silently dead Mon/Fri run)

Self-heal fires only after 2 consecutive failures, dedupes actions, and re-checks
before reporting. An invalid Caddyfile is deliberately never auto-healed.

Changing anything

/opt/supervision/backup.sh <label>     # snapshot caddyfile, script, env, units, logrotate, sysctl
# apply + validate + reload
/opt/supervision/rollback.sh --list
/opt/supervision/rollback.sh latest    # restore, validate caddy, reload, re-probe all 3 sites

Edge block list lives in supervision/staged/Caddyfile.probe-block; the live
Caddyfile answers 403 for wp-*, xmlrpc.php, phpmyadmin*, .env*, .git*,
backup*, install.php and friends before they reach the app.

In-running DB backup

• Source: /opt/sportstlm/data/sportstlm.db (opened read-only).
• Output per run: inrun-*.sqlite.gz (the three inrun tables, ~1.65 MB) and
  fulldb-*.sqlite.gz (whole DB, ~6.9 MB, 37 tables). Row counts + integrity
  verified before push.
• Local: /opt/backups/inrun/<stamp>/, last 12 runs kept (~100 MB).
• GitHub: private repo investconfig-spec/sportstlm-db-backups, pushed from
  /opt/backups/github/inrun-db with deploy key /root/.ssh/id_ed25519_vps_backup
  (wired through GIT_SSH_COMMAND).
• PAT alternative: /etc/inrun-backup/github.token (0700 dir / 0600 file), installed
  via /opt/backups/accept_credential.py --from-file|--from-url.
• Resource cost: ~4 s wall, ~150 MB peak RSS per run.

Code snapshots

/opt/backups/code_backup.py, hourly (code-backup.timer), mirror at
/opt/backups/github/code-backups/{sportstlm,racing,lottery} — 218 files / 20 MB.
Excludes data/, *.db, *.sqlite*, .env*, vps-credentials.txt, media,
node_modules/, .next/, __pycache__/, browser_profile/, {src.
Every run credential-scans the tree (OpenAI/Stripe/GitHub/AWS/Slack/Google keys,
private-key blocks, Telegram tokens) and aborts the commit on a hit — tested with
a planted key. Push target: investconfig-spec/sportstlm-code-backups.

Two deploy keys, one per repo (deploy keys are repo-scoped, not account-wide):

code_backup.py and the supervisor probe both use the _code key for the code
repo. Probing the code repo with the _backup key returns
ERROR: Repository not found — that means wrong key, not a missing repo.

Ops dashboard (published)

https://sportstlm.com/ops/ — basic auth wt / password in /root/ops-dashboard.cred
(0600). Rotate with /opt/ops/rotate-password.sh.

2026-10-22: password rotated after the stored value was found not to verify against the
Caddy hash. After rotating, always verify with curl -u wt:<pw> https://sportstlm.com/ops/
→ 200 and anonymous → 401. Served by opsd.service on 127.0.0.1:9600, proxied by Caddy
handle_path /ops/* with basic_auth.

JSON for scripts: /ops/api/state, /ops/api/entry/<id>, POST /ops/api/refresh.

Ops control table

/opt/ops/registry.json lists every tool/process on the box. /opt/ops/collect.py
probes each one every 5 min (ops-collect.timer) and writes /opt/ops/state.json +
/opt/ops/index.html.

Status vocabulary: online / offline (Windows-only) / degraded / unknown.

Credential encryption (applied)

The Telegram bot token is now encrypted at rest with systemd-creds
(/etc/ops/tg-token.cred, host-key bound). The five units that alert load it via
LoadCredentialEncrypted=tg-token:… and read it from $CREDENTIALS_DIRECTORY/tg-token.

Sports Mole pipeline (online)

/opt/sportstlm/scripts/sportsmole_online.py, timer sportsmole-online.timer
(09:00 & 18:00 SGT). Flow: index pages → unseen preview URLs → preview pages →
parse → POST /api/admin/sportsmole/import-batch → paraphrase new rows. Cloudflare
handling: one request at a time, 2.5–3.5 s apart, cookie jar + browser headers,
exponential backoff, challenge detection aborts before import and alerts.

LLM provider order & cost tracking

Order (editorial._llm_complete): Groq → OpenRouter → DeepSeek → Ollama.
Usage appended to /opt/sportstlm/data/llm_usage.jsonl. Report:
/opt/ops/llm_cost_report.py.

Staging datasets — no-deletion policy

/opt/sportstlm/data, /opt/racing/data, /opt/lottery/data are fallback datasets —
observed only, never purged.

Known traps

• ssh -T git@github.com → Permission denied even with valid deploy key (test with push).
• write_file refuses /etc/**; use terminal heredoc + .bak.
• Timers need Asia/Singapore suffix in OnCalendar (host runs UTC).
• Alert chat id comes from TELEGRAM_ALLOWED_USERS, not telegram_update_receipts_<id>.json.
• Log files must never receive tokens.