Sitebulb

Promptfree, for everyone, and the only version there is

Copy the prompt for Claude CodeCopy the prompt for CodexCopy the prompt for Cursor

Copy

Build me the site auditor I actually need instead of Sitebulb.

The crawler is a weekend; **the check library and the prioritisation are the product.** Build ten checks that matter rather than three hundred nobody reads.

STACK
- Node 20+ as a CLI and a small server for the report
- SQLite through better-sqlite3, WAL mode, one database per crawl
- undici for fetching, a real HTML parser, and a headless browser only for the pages that need it
- Caddy in front of the report viewer

THE DATA MODEL
- crawls: id, root_url, started_at, finished_at, settings_json
- pages: id, crawl_id, url, status, redirect_to, content_type, bytes, title, meta_desc, h1, canonical, robots, indexable, depth, fetched_at, render_used
- links: id, from_page_id, to_url, rel, anchor, is_internal
- issues: id, crawl_id, page_id, code, severity, detail_json
- **One database per crawl, so comparing two crawls is a query across two files** and nothing is overwritten

CRAWLING POLITELY AND CORRECTLY
- Respect robots.txt, an identifiable user agent, a concurrency cap and a delay. **You are crawling somebody's site, possibly your own client's** — an impolite crawler causes an incident
- Normalise URLs properly: trailing slash, case, query parameter order, fragments dropped. Bad normalisation makes one page look like forty
- Follow redirects but record every hop; a chain of four is itself the finding
- Render with a browser only where the initial HTML has no content, and record which pages needed it. **A site that needs rendering to show its content is the finding**, not an inconvenience

THE CHECKS WORTH HAVING
- Broken internal links and redirect chains
- Pages with no title, a duplicate title, or a title far too long
- Canonical pointing somewhere unexpected, or a page canonicalising to a non-indexable one
- noindex on a page that is linked from navigation, which is almost always an accident
- Orphan pages: in the sitemap and not linked from anywhere
- Sitemap disagreeing with the crawl in either direction
- Duplicate content by shingle similarity, not by exact match
- Images with no alt text and images far larger than their displayed size
- **Rank the findings by how many pages they affect and how bad each is.** An unranked list of four thousand issues is why these reports go unread

COMPARING CRAWLS
- Diff two crawls: pages appeared, disappeared, changed status, changed title. **This is the most useful feature in the category and the one most tools bury** — an audit is a snapshot, a diff is a monitoring system

THE REPORT
- A static HTML report from the database, with every issue linked to the affected URLs and a plain sentence explaining why it matters
- CSV export per issue type, because that is what gets handed to a developer

WHAT MATTERS MOST
URL normalisation and the crawl diff. The first decides whether the data is true; the second turns an audit into something you run every week.