Case study · Web change monitoring
ScoutFlow
Watches public web pages for price, stock and content changes, and alerts you on Discord or Telegram.
Every fetch is SSRF-guarded, and a failed check is recorded as a failure, never as a change.
- Type
- Personal open-source project
- Automated tests
- 46 (pytest)
- Built with
- Python · FastAPI · httpx
- Live demo
- Runs locally — not hosted

Overview
What it does
ScoutFlow checks public pages and APIs on a schedule with five monitor types — text, price, availability, whole-page content and a JSON field — and shows exactly what changed. It alerts Discord, Telegram or a signed webhook, and treats a failed check as a failure with a precise reason, never as a change.
Problem
People refresh pages to catch price drops, restocks and policy edits, and naive monitors cry wolf whenever a page times out or a layout changes.
Solution
A staged check runner (fetching → extracting → comparing) records one of four outcomes: baseline, no change, changed or failed. A failure carries a type — timeout, network, HTTP error, blocked, selector missing, robots.txt, too large — and the last good value is kept, so recovery compares against real data.
Screenshots
From the running application
Captured from the app running locally with its seeded demo data. Select an image to open it full size.
Architecture
How the pieces fit
- Scheduler → Safe fetcher
- Check now → Safe fetcher
- Safe fetcher → Extractors
- Extractors → Comparison
- Comparison → Alerts
- Comparison → Database
- Core service
- External system
- Storage / entry point
Core features
What's in it
Five monitor types
Text, price (locale-aware parsing), availability, page content with ignorable selectors, and a JSON field by path.
Readable changes
Word diffs that collapse unchanged runs, price deltas with percentages, and in-stock/out-of-stock transitions.
Live “Check now”
The UI follows each stage of a manual check as it happens, with a cooldown and one check per monitor at a time.
Alerts
Discord (mentions disabled, markdown escaped), Telegram (HTML-escaped) and HMAC-signed webhooks; one alert at the failure threshold.
Selector tester
Try a selector against the live page before saving and see exactly what would be recorded.
Responsible by design
robots.txt respected, per-host request spacing, an honest user agent and a page-size cap. No CAPTCHA or anti-bot evasion.
Technical decisions
Interesting problems, and how they're solved
- 01
A failure is never a change
Failures are a separate outcome and never overwrite the stored value, so a timeout can't trigger a false “price changed”.
- 02
SSRF checks at every hop
URLs are validated when saved, before each request and at every redirect, against resolved addresses including IPv4-mapped IPv6.
- 03
Output escaping per channel
Page content goes into alerts, so Discord mentions are disabled and Telegram HTML is escaped — no @everyone from a scraped headline.
- 04
A seed that really runs
The demo week is produced by real checks against built-in demo pages, then back-dated — the history on screen is genuine output.
started = time.perf_counter()
redirects: list[str] = []
current = url
try:
async with _client() as client:
while True:
await _space_requests(urlsplit(current).hostname or "")
response, body = await _raw_get(client, current, accept)
if response.is_redirect and "location" in response.headers:
if len(redirects) >= settings.max_redirects:
raise CheckFailure("http_error", "Too many redirects.", response.status_code)
nxt = urljoin(current, response.headers["location"])
try:
await validate_url(nxt)
except UnsafeURL as exc:
raise CheckFailure("invalid_config", f"The page redirected to a blocked address. {exc}") from exc
redirects.append(nxt)
current = nxt
continue
break
except httpx.TimeoutException as exc:
raise CheckFailure("timeout", f"No response within {settings.http_timeout_seconds:g} s.") from exc
except httpx.TransportError as exc:
raise CheckFailure("network", _network_message(exc)) from excStack & skills
Built with
Technology
- Python
- FastAPI
- httpx
- asyncio
- BeautifulSoup
- SQLAlchemy
- Next.js
- TypeScript
- shadcn/ui
What this project demonstrates
- Scraping done responsibly
- Async scheduling
- Change detection & diffs
- SSRF protection
- Alert integrations
Repository & demo
Run it yourself
Seeded demo account: demo@scoutflow.dev / scoutflow-demo-1234. Monitors watch built-in demo pages you can change from the Demo lab.
No public live demo is hosted. Everything below runs locally with Python 3.12+ and Node 20+.
$ git clone https://github.com/Moeijiro/scoutflow.git
$ cd scoutflow
$ make install
$ make seed
$ make api
$ make web
Have a project in mind?
I build APIs, automation systems, Discord integrations and full-stack applications. Tell me what it should do and I'll tell you how I'd build it.





