Ahrefs

Promptfree, for everyone, and the only version there is

Copy the prompt for Claude CodeCopy the prompt for CodexCopy the prompt for Cursor

Copy

Do not build this. The product is a crawl of the entire web, and you cannot have one.

Backlinks pointing at your competitors, keyword volumes from clickstream data, rankings measured across millions of queries — every one of those comes from an index built over years at enormous cost. No amount of good code substitutes for it.

WHAT TO DO INSTEAD
- Keep the subscription if you genuinely use the competitive data. That is what you are paying for
- If you only ever look at your own site, cancel it. Search Console gives you your own queries, impressions, positions and backlinks for nothing, straight from the source, and it is more accurate about your own site than any third party

WHAT YOU CAN BUILD, AND SHOULD

A technical audit of your own site, which is the part that needs no index at all. Node 20+, undici, a real HTML parser, SQLite for the crawl store, run locally or in CI:

- Crawl your own site politely, obeying robots.txt, and record every page, link, redirect and resource
- Broken internal links with the pages that link to them — the most valuable output and worth running weekly
- Redirect chains and loops, orphan pages, pages more than a few clicks from the home page
- Duplicate or missing titles, descriptions and h1s, and lengths outside what gets truncated
- Canonical problems: missing, pointing at a redirect, pointing off-site, or disagreeing with the sitemap
- Pages in the sitemap that nothing links to, and linked pages missing from the sitemap
- Images without alt text or dimensions, and structured data that does not validate
- A diff against the previous crawl: what broke, what was fixed. That report is what somebody actually reads

Then pull your own performance from Search Console's API into the same database, so 'this page lost half its impressions after the redesign' is one query rather than two tabs.

THE ONE-LINE VERSION
Nobody can build the index. Everybody can build the audit, and the audit is where most of the fixable problems are.