Sports Mole Pipeline
Online preview scraper → paraphrase, replacing the old local scrape+push.
Entry point
/opt/sportstlm/scripts/sportsmole_online.py, timer sportsmole-online.timer
(09:00 & 18:00 SGT, RandomizedDelaySec 5 min).
Flow per run
- Index pages → unseen preview URLs (deduped against
sportsmole_previews). - Preview pages → parse.
POST /api/admin/sportsmole/import-batch.- Paraphrase new/backlogged rows via
paraphrase?id=N(async, poll for summary). - Log to
/var/log/sportsmole-online.log.
Cloudflare handling
- One request at a time, 2.5–3.5 s apart.
- Cookie jar + browser headers.
- Exponential backoff on 403/429/5xx.
- Challenge detection (
Just a moment,cf-chl) aborts before importing and
alerts the supervisor bot. - Only unseen URLs fetched; 60-page cap per run.
Modes
| Flag | Purpose |
|---|---|
| (default) | dry run |
--push |
write |
--paraphrase-only --paraphrase-max N |
drain rows missing a summary |
--limit N |
cap |
--refresh |
re-parse known URLs |
--backfill-comp |
re-label rows still on generic fallback |
Each writing run emits a RUN {...} JSON line the dashboard reads.
Competition mapping
resolve_competition() uses only two signals (nav/body text produced wrong leagues):
1. URL slug — authoritative (/football/france-ligue-1/... → Ligue 1).
2. form-line label — "
Anything else stays Football (counted as unlabelled competition), never guessed.
The scraper sends only columns it owns (match_title/home_team/away_team/competition/
date/source_url/raw_text/prediction/form_home/form_away) and deliberately does not
send summary/team_news (paraphrase owns), lineups_json/key_players_json (app owns),
or status/hidden/archived (server decides).
Rollback
systemctl disable --now sportsmole-online.timer — the local poller and
push_sportsmole.py still work; nothing in the DB is deleted. vps-module-monitor
reads sportsmole_previews.created_at (the online path writes rows, not scrape_runs).