VWO

Promptfree, for everyone, and the only version there is

Copy the prompt for Claude CodeCopy the prompt for CodexCopy the prompt for Cursor

Copy

Build the A/B testing I actually need instead of VWO — and know the two halves that make this ALMOST.

**The statistics**, which produce a confident wrong answer when done badly, and **the flicker**, which is the thing that quietly invalidates client-side tests and that nobody mentions.

STACK
- Node 20+ with Fastify for the control plane
- SQLite through better-sqlite3 for definitions and assignments
- Your existing warehouse or database for the metrics
- A tiny script, or no script at all — read the next section first
- Caddy in front

DECIDE WHERE THE TEST RUNS, BEFORE ANYTHING ELSE
- **Server-side is better in every way that matters.** The variant is chosen before the HTML is built, so there is nothing to flash, no layout shift, and no script blocking the render. If the page is server-rendered, do this and skip the rest of this section
- Client-side means the original paints first and is then rewritten. That flash is visible, it changes behaviour, and it biases the test towards the control — which is the version people saw first
- If you must do it client-side: an anti-flicker block hides the content until the variant applies, with a timeout that reveals it anyway. That timeout is a real cost paid by every visitor including the ones not in the test, and it is why this approach is worse

THE DATA MODEL
- experiments: id, key, hypothesis, variants_json, traffic_percent, unit_kind, started_at, stopped_at, decision, decision_note
- assignments: id, experiment_id, unit_id, variant, at — append-only, and the record everything is computed from. Record it from the first minute; it cannot be reconstructed
- exposures: id, assignment_id, at — assigned is not the same as saw it, and analysing on assignment dilutes the effect with people who never reached the page
- metrics: id, name, kind, query_sql, is_guardrail, direction, minimum_effect
- results: id, experiment_id, metric_id, computed_at, stats_json, method, verdict

ASSIGNMENT
- Deterministic: hash the unit identifier with the experiment key and a salt. The same person always gets the same variant, on the server and in the browser, with no coordination
- Stable when the traffic percentage changes: somebody in the first ten per cent stays in when it becomes twenty
- One experiment per surface, enforced by layers, or two tests change the same button and neither result means anything

THE STATISTICS, WHICH IS THE PART THAT GOES WRONG SILENTLY
- Use a published method valid under continuous monitoring — a sequential test, or a Bayesian approach with a stated prior
- **Never a fixed-horizon test looked at every day.** That combination calls a neutral change a winner roughly a third of the time, and it is the single most common error in this field
- Compute the required sample size before starting, from the baseline rate and the smallest effect worth acting on, and show progress towards it
- Report an interval, never a point estimate, with the method named beside it
- **A sample-ratio mismatch check every day.** If the split is significantly off, the test is broken and the numbers mean nothing. This catches more real bugs than any statistical refinement
- Cap extreme per-unit values at a stated percentile before analysis, or one enormous order decides your experiment

AND THE HONEST PART
- Most tests on most sites are underpowered: the traffic is not there to detect the effect being looked for, and running one anyway produces noise that gets shipped. Compute the sample size first and, if the answer is six months, do not run the test — change the thing on judgement and say so

WHAT MATTERS MOST
Server-side assignment and a valid sequential method. The first removes the flicker problem entirely; the second is the difference between an experiment programme and an expensive way to confirm what somebody already believed.