Skip to content
Moeijiro
← All projects

Case study · Web change monitoring

ScoutFlow

Watches public web pages for price, stock and content changes, and alerts you on Discord or Telegram.

Every fetch is SSRF-guarded, and a failed check is recorded as a failure, never as a change.

AutomationSecurity & Monitoring
Type
Personal open-source project
Automated tests
46 (pytest)
Built with
Python · FastAPI · httpx
Live demo
Runs locally — not hosted
Check now in progress: opening page, extracting, comparing, change detected

Overview

What it does

ScoutFlow checks public pages and APIs on a schedule with five monitor types — text, price, availability, whole-page content and a JSON field — and shows exactly what changed. It alerts Discord, Telegram or a signed webhook, and treats a failed check as a failure with a precise reason, never as a change.

Problem

People refresh pages to catch price drops, restocks and policy edits, and naive monitors cry wolf whenever a page times out or a layout changes.

Solution

A staged check runner (fetching → extracting → comparing) records one of four outcomes: baseline, no change, changed or failed. A failure carries a type — timeout, network, HTTP error, blocked, selector missing, robots.txt, too large — and the last good value is kept, so recovery compares against real data.

Screenshots

From the running application

  • Check now in progress: opening page, extracting, comparing, change detected
    Check now in progress: opening page, extracting, comparing, change detected
  • Price monitor with previous and current value and a price chart
    Price monitor with previous and current value and a price chart
  • Change feed with a word diff of a headline and a stock transition
    Change feed with a word diff of a headline and a stock transition
  • A failing monitor that keeps its last good value
    A failing monitor that keeps its last good value
  • New monitor with a live selector test
    New monitor with a live selector test
  • Monitor on a phone
    Monitor on a phone

Captured from the app running locally with its seeded demo data. Select an image to open it full size.

Architecture

How the pieces fit

Schedulerper-monitor intervalCheck nowcooldown · one at a timeSafe fetcherSSRF · robots.txtExtractorstext · price · stock · JSONComparisondiff · delta · outcomeAlertsDiscord · Telegram · webhookDatabasechecks · changes
  • Scheduler → Safe fetcher
  • Check now → Safe fetcher
  • Safe fetcher → Extractors
  • Extractors → Comparison
  • Comparison → Alerts
  • Comparison → Database
  • Core service
  • External system
  • Storage / entry point

Core features

What's in it

  • Five monitor types

    Text, price (locale-aware parsing), availability, page content with ignorable selectors, and a JSON field by path.

  • Readable changes

    Word diffs that collapse unchanged runs, price deltas with percentages, and in-stock/out-of-stock transitions.

  • Live “Check now”

    The UI follows each stage of a manual check as it happens, with a cooldown and one check per monitor at a time.

  • Alerts

    Discord (mentions disabled, markdown escaped), Telegram (HTML-escaped) and HMAC-signed webhooks; one alert at the failure threshold.

  • Selector tester

    Try a selector against the live page before saving and see exactly what would be recorded.

  • Responsible by design

    robots.txt respected, per-host request spacing, an honest user agent and a page-size cap. No CAPTCHA or anti-bot evasion.

Technical decisions

Interesting problems, and how they're solved

  1. 01

    A failure is never a change

    Failures are a separate outcome and never overwrite the stored value, so a timeout can't trigger a false “price changed”.

  2. 02

    SSRF checks at every hop

    URLs are validated when saved, before each request and at every redirect, against resolved addresses including IPv4-mapped IPv6.

  3. 03

    Output escaping per channel

    Page content goes into alerts, so Discord mentions are disabled and Telegram HTML is escaped — no @everyone from a scraped headline.

  4. 04

    A seed that really runs

    The demo week is produced by real checks against built-in demo pages, then back-dated — the history on screen is genuine output.

backend/app/fetch/http.py · L208–231View on GitHub ↗
started = time.perf_counter()
redirects: list[str] = []
current = url
try:
    async with _client() as client:
        while True:
            await _space_requests(urlsplit(current).hostname or "")
            response, body = await _raw_get(client, current, accept)
            if response.is_redirect and "location" in response.headers:
                if len(redirects) >= settings.max_redirects:
                    raise CheckFailure("http_error", "Too many redirects.", response.status_code)
                nxt = urljoin(current, response.headers["location"])
                try:
                    await validate_url(nxt)
                except UnsafeURL as exc:
                    raise CheckFailure("invalid_config", f"The page redirected to a blocked address. {exc}") from exc
                redirects.append(nxt)
                current = nxt
                continue
            break
except httpx.TimeoutException as exc:
    raise CheckFailure("timeout", f"No response within {settings.http_timeout_seconds:g} s.") from exc
except httpx.TransportError as exc:
    raise CheckFailure("network", _network_message(exc)) from exc

Stack & skills

Built with

Technology

  • Python
  • FastAPI
  • httpx
  • asyncio
  • BeautifulSoup
  • SQLAlchemy
  • Next.js
  • TypeScript
  • shadcn/ui

What this project demonstrates

  • Scraping done responsibly
  • Async scheduling
  • Change detection & diffs
  • SSRF protection
  • Alert integrations

Repository & demo

Run it yourself

Seeded demo account: demo@scoutflow.dev / scoutflow-demo-1234. Monitors watch built-in demo pages you can change from the Demo lab.

No public live demo is hosted. Everything below runs locally with Python 3.12+ and Node 20+.

$ git clone https://github.com/Moeijiro/scoutflow.git
$ cd scoutflow
$ make install
$ make seed
$ make api
$ make web

Have a project in mind?

I build APIs, automation systems, Discord integrations and full-stack applications. Tell me what it should do and I'll tell you how I'd build it.