Visualping

Promptfree, for everyone, and the only version there is

Copy the prompt for Claude CodeCopy the prompt for CodexCopy the prompt for Cursor

Copy

Build me page change monitoring that replaces Visualping: tell me when a page changes, and not when an advert rotates.

STACK
- Node 20+ with Fastify
- SQLite through better-sqlite3, WAL mode
- A plain HTTP fetch first, Playwright only for pages that genuinely need it
- A worker for the checks
- Caddy in front

THE DATA MODEL
- watches: id, name, url, mode, selector, ignore_selectors_json, ignore_patterns_json, render_js, interval_seconds, next_check_at, threshold_percent, is_active, consecutive_failures, last_error, notify_json
- snapshots: id, watch_id, kind, content_gzip, text_content, image_path, dom_hash, text_hash, image_hash, fetched_at — keep several, because a diff needs the previous one and a false positive needs the one before that
- changes: id, watch_id, from_snapshot_id, to_snapshot_id, kind, magnitude, diff_summary, detected_at, notified_at, acknowledged_at
- checks: id, watch_id, status, changed, ms, at — every check, whether or not anything changed, so 'the page was stable' and 'we never looked' are distinguishable
- Snapshots are the bulk of the storage. Compress them, keep a bounded number per watch, and state the retention

MODES
- Text: extract the visible text, normalise whitespace, and diff. The right default for almost everything
- Element: the same but scoped to a selector, which is what most people actually want — a price, a status, a date
- HTML: diff the markup, for detecting a structural change
- Visual: render and compare images, which catches what the others cannot and produces the most false positives
- Choose per watch, and default to element or text. Visual is the last resort, not the first

THE NOISE PROBLEM, WHICH IS THE ENTIRE DIFFICULTY
- Every real page changes constantly for reasons nobody cares about: rotating adverts, a timestamp, a session identifier, a view counter, a carousel, a randomised recommendation strip
- An unfiltered monitor fires every check and is switched off within two days. So the work is all in the filtering:
  - ignore selectors, configurable per watch, removed before any comparison
  - ignore patterns as regular expressions, applied to the extracted text: timestamps, numbers matching a shape, tokens
  - a change threshold, so a difference below a percentage is recorded but not notified
  - a learning period: on the first several checks, note which regions changed every time and offer them as suggested ignores. That single feature is what makes the tool usable without an afternoon of tuning
- For visual mode: a pixel difference with a tolerance, ignoring anti-aliasing, plus masked regions. And rendering must be deterministic — fixed viewport, animations disabled, fonts settled, lazy images loaded — or every check differs from the last

FETCHING POLITELY
- Conditional requests with ETag and If-Modified-Since; honour Retry-After and robots.txt
- An identifiable user agent with a URL explaining what it is, and rate limits per host
- The browser only where the plain fetch finds nothing, because it costs fifty times as much. Record which mode each watch uses and retry the cheap one periodically
- Adaptive intervals: a page that changes monthly does not need checking every five minutes, and checking it that often is rude
- Back off on failures and disable after enough consecutive ones, with a clear message rather than silence

WHAT A NOTIFICATION SAYS
- Not 'the page changed'. The actual diff: the text that was added and removed, in context, or the two images side by side with the changed region marked
- A notification you have to go and investigate is a notification that gets ignored
- Channels: email, webhook, chat, RSS. Digest option, so a busy watch sends once a day rather than eleven times
- A link to the stored snapshots, so the history is browsable

WHEN A PAGE STOPS BEING WATCHABLE
- A watch returning nothing after previously returning content is a broken selector, not an empty page. Detect it and alert differently
- A page that starts requiring a login or a challenge is reported honestly, and no attempt is made to get around it
- Keep the last snapshots so a broken watch can be repaired against what actually arrived, with a repair screen showing the stored HTML beside the selectors

THE LIMITS, WORTH STATING
- Pages behind a login can be watched with a stored session, but that session will expire and the watch will break. Say so
- Some sites forbid automated access in their terms. Watching a page for yourself is one thing; hammering it is another. Default the intervals to something respectful

OPERATIONS
- .env: DATABASE_PATH, STORAGE_PATH, BASE_URL, USER_AGENT_URL, SESSION_SECRET, RENDER_TIMEOUT_MS
- Migrations on boot, each once
- Snapshot retention with a sweeper, and a disk watchdog
- Nightly backup off the machine, restore script
- Health endpoint reporting the oldest overdue watch and the failure count

WHAT MATTERS MOST
The ignore rules and the learning period. Build the filtering before the notification is pretty, then watch twenty real pages for a week and count the false positives. That number is the only measure of whether this tool is worth having.

Give me the repository, the checker, the selector and ignore editor, migrations, .env.example, and a README with deploy steps behind Caddy.