Screaming Frog

Promptfree, for everyone, and the only version there is

Copy the prompt for Claude CodeCopy the prompt for CodexCopy the prompt for Cursor

Copy

Build me the site crawler I actually need instead of Screaming Frog: walk the whole site, and tell me what is broken before somebody else finds it.

STACK
- Node 20+ as a command-line tool, with a small server-rendered report
- SQLite through better-sqlite3 for the crawl store, WAL mode
- undici for fetching, a real HTML parser, and Playwright only for the pages that need JavaScript
- Runs locally or in CI; there is no reason for this to be a hosted service

THE DATA MODEL
- crawls: id, start_url, started_at, finished_at, pages, config_json
- pages: id, crawl_id, url, status, redirect_to, content_type, bytes, depth, found_at_url, title, title_length, meta_description, description_length, h1_json, canonical, robots_meta, lang, word_count, text_hash, load_ms, is_indexable, indexability_reason
- links: id, crawl_id, from_page_id, to_url, to_page_id, anchor_text, rel, is_internal, position
- resources: id, crawl_id, page_id, kind, url, status, bytes — images, scripts, styles
- images: page_id, url, alt, width_attr, height_attr, bytes, natural_width, natural_height
- issues: id, crawl_id, page_id, rule, severity, detail_json
- Every crawl kept, so two can be compared. The comparison is what turns a crawler into a monitor

CRAWLING POLITELY AND CORRECTLY
- Obey robots.txt for your own site too, because that is what a search engine will do and you want to see what it sees
- An identifiable user agent, a concurrency limit, a delay per host, and a crawl budget so a calendar with infinite pages does not consume the afternoon
- Normalise URLs consistently: resolve relative links, strip the fragment, decide about trailing slashes and case once and apply it everywhere. Inconsistent normalisation is what produces phantom duplicates
- Follow redirects but record every hop, because the chain is the finding
- Respect nofollow when configured, and record it either way
- Render with a browser only where the page is empty without it; note which pages needed it, because that list is itself a finding

WHAT TO CHECK — THE LIST IS THE PRODUCT
- Broken internal links, with the pages that link to them. This is the single most valuable output and the reason to run it weekly
- Broken external links, checked more gently and with the flakiness accounted for — a single timeout is not a broken link, so re-check before reporting
- Redirect chains and loops, with the number of hops
- Pages returning an error, and pages linked from the site but blocked by robots
- Duplicate and missing titles, descriptions and h1s; lengths outside the range that gets truncated
- Duplicate content by text hash, and near-duplicates
- Canonical problems: missing, self-referencing when it should not be, pointing at a redirect, pointing off-site, or conflicting with the sitemap
- Pages in the sitemap that are not linked from anywhere, and linked pages missing from the sitemap
- Orphan pages, reachable only from the sitemap
- Depth: pages more than a few clicks from the home page, which is a structural problem rather than a page problem
- Images without alt text, without dimensions, or served far larger than they are displayed
- Mixed content, insecure links, and links to the wrong protocol or hostname
- hreflang consistency if the site has it, which is almost always wrong somewhere
- Structured data parsed and validated per page
- Response times, and the slowest pages ranked

WHAT THE HOSTED TOOL DOES THAT THIS SHOULD NOT PRETEND TO
- The value of the original is a decade of accumulated checks, each one a lesson from a real site. Your list will be shorter
- So make it easy to add a rule: a rule is a small function over a parsed page and its links, with a severity and a message. Add one every time you find something the hard way, and the tool improves in the direction of your own sites

THE REPORT
- A summary of issues by severity, then each rule with the affected pages and the specific detail
- Exports as CSV and JSON so it can go into a spreadsheet or a ticket
- A diff against the previous crawl: what broke, what was fixed, what is new. That is the output somebody will actually read every week
- Run in CI on a schedule and fail the build on new issues above a severity, which is how a site stops rotting silently

OPERATIONS
- No server needed. A local database file per crawl, and a static report
- .env or flags: start URL, concurrency, delay, budget, user agent, whether to render
- Sitemaps and a URL list accepted as seeds as well as a crawl

WHAT MATTERS MOST
URL normalisation and the crawl-to-crawl diff. Get normalisation right or every report will be full of duplicates that do not exist, and build the diff early, because a list of four hundred issues is ignored while a list of six new ones gets fixed.

Give me the repository, the crawler, the rule set with each rule as its own small module, the report, the CI workflow, and a README explaining how to add a rule.